Armed the gate on / on the live server. Aborted after about twenty
seconds. The box was never at risk — Caddy stayed sub-millisecond and
load never rose — but the gate blocked reads of an AI agent's session
transcript, reporting it as Linux.Coinminer.XMRig.
It was not wrong about the bytes. That transcript contains
"stratum+tcp://", "donate-level" and "xmrig" because the miner rule was
being written in that session. The rule matched a document ABOUT
malware.
Three bugs, none of which the tmpfs stage could have shown:
1. The miner rule had no file-type condition, so any text mentioning
mining tripped it: threat-intelligence reports, security blog posts,
support tickets, an antivirus's own logs. It now requires ELF magic,
as the rootkit rule always did. Two regression tests: a transcript
discussing the rule is clean, and an ELF carrying the same strings
still matches — the fix must not cost the detection it exists for.
2. The gate requested FAN_OPEN_PERM, so it held every OPEN, not every
execve. A matching file could not be read by anything. That is a
different product from the one advertised, and on a multi-tenant box
it is a denial of service against the operator rather than a defence.
Read events are no longer requested at all. FAN_OPEN_EXEC_PERM and
FAN_CLOSE_WRITE cover the threat: execution is refused before it
happens, and anything malicious written to disk is quarantined when
the write completes. An interpreted script is caught as it lands
rather than as it is read — the same protection, one step earlier.
`serve` also guards deny-on-exec explicitly, so re-requesting read
events later cannot silently restore the old behaviour.
3. Hound did not exclude its own state. /var/lib/hound and /run/hound
are now always excluded; the vault holds live malware by definition.
Henry asked whether the single watchdog rescue was queue pressure or
scan time. It was scan time: the gate inherited the on-demand 100 MB
limit and tried to read and match a multi-megabyte transcript inline
while holding a process. A gate's budget is a deadline, not a size, so
it now caps at 32 MB — anything larger is allowed through unread rather
than turned into a rescue, which is a process released unscanned and
worse than never having looked.
Dropping read events made everything faster, because most opens on a
running machine are reads:
latency +1.38 -> +0.79 ms per exec
throughput 2,680 -> 4,178 execs/sec (58% of ungated, was 36%)
events 1,179 in five seconds on an idle tmpfs -> 1
Re-verified on the tmpfs: an ELF miner is quarantined before it can even
be made executable, a document naming every one of its strings is
readable, and a clean binary runs.
297 tests pass. The gate stays off; stage 3 gets attempted again with
these fixes and fresh numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Armed the execution gate on the live server for the first time. It
reported itself armed on a dedicated tmpfs, and then let EICAR execute.
Counters: 0 allowed, 0 blocked. Not one event was ever delivered.
Cause: systemd gives the service a PRIVATE MOUNT NAMESPACE. Several
perfectly ordinary hardening options force one — ProtectProc,
ProtectKernelTunables, ProtectControlGroups — and none of them mention
it. FAN_MARK_MOUNT marks a vfsmount, and a private namespace holds its
own vfsmount for the same filesystem. So the daemon marked its copy,
every other process on the machine used the host's copy, and the gate
protected nothing while claiming to be armed.
That is the worst way for a security feature to fail: silently, with a
reassuring status line. Nothing in the unit tests could have caught it —
they run in the host namespace, where the mount mark works.
Fixed by always using FAN_MARK_FILESYSTEM, which marks the SUPERBLOCK.
A superblock is shared across namespaces, so events arrive from
everywhere, and scoping still works because a superblock is exactly one
filesystem: marking a dedicated mount covers that mount and nothing
else. mark_mount is kept for the smoke-test example, which runs outside
systemd, with a doc comment about when it lies to you.
Two more that only appeared once the gate was actually armed:
* SystemCallFilter=@system-service kills the daemon with SIGSYS the
moment the gate is switched on. fanotify_init and fanotify_mark live
in @privileged, which @system-service deliberately excludes. Granted
individually rather than by adding @privileged, which would also admit
setuid, chroot, bpf and kexec_load. Invisible until armed — the
service starts fine with the gate off.
* The capability reduction reported "60 capabilities could not be
dropped" while the end state was perfectly correct. systemd's
CapabilityBoundingSet had already done the work, and the service does
not hold CAP_SETPCAP afterwards, so every redundant drop failed EPERM.
It now checks what is actually present, attempts only that, and judges
by the end state rather than by return codes.
Also removed AmbientCapabilities from the unit. Ambient capabilities are
inherited by children, the daemon shells out to freshclam/rpm/pacman on
some paths, and a root process already receives the bounding set as
permitted — so it bought nothing except a way for CAP_SYS_ADMIN to leak
into a subprocess.
Performance, measured on the live server rather than guessed at:
+2.70 ms/exec as first written
+1.47 ms/exec after the reader blocked on poll() instead of sleeping
a millisecond between empty reads — that sleep sat on
the critical path of every execve
+1.38 ms/exec after answering cache hits in the reader thread, with
no channel handoff or worker wakeup
2,680 execs/sec sustained through the gate, 16-way parallel, with
ZERO watchdog rescues — the queue never fell behind. Ungated is 7,455.
Caddy stayed at sub-millisecond throughout and load did not rise.
Joe and Henry are right that the exec-heavy paths on this box — Docker
overlays, agent workspaces, PM2 — are the performance bar rather than an
exclusion list. Protecting agent workspaces from injected payloads is
the product. 2,680/sec with no backlog is roughly ten times what this
machine generates, so the bar looks clearable; stage 2 will say for sure.
295 tests pass, and the three-phase gate smoke test still passes
including the fail-open case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes Phase 1. The gate now asks for FAN_CLOSE_WRITE alongside the
permission events, so a threat written to disk is quarantined and a
threat being executed is refused — one mechanism, one mark, no
watch-descriptor ceiling and no blind spots outside a configured list.
Verified live. The nicest evidence is an error message:
$ chmod +x /tmp/hound-live/malware.sh
chmod: cannot access '/tmp/hound-live/malware.sh': No such file or directory
Hound had already quarantined it. `hound quarantine list` shows the
entry, the clean binary beside it still runs, and CapPrm/CapEff/CapBnd
read 000000000020000e.
Three bugs, each of which looked like working code:
* A file descriptor number is not an identity. The kernel allocates an
fd per event and recycles the number the moment we close it, so one
write arrives as FAN_OPEN_PERM on fd 6 and then FAN_CLOSE_WRITE on fd
6 again. Idempotency keyed on the fd treated the second as a duplicate
of the first and dropped it — detection ran, matched EICAR, and threw
the result away. Events now carry a monotonic seq that is never reused.
* rename(2) fails EXDEV across filesystems, and for quarantine that is
the common case rather than the exotic one: the vault is under
/var/lib while threats land on /home, in a tmpfs, on a USB stick or
in a container overlay. Quarantine now falls back to copy-then-unlink,
unlinking only once the copy is safely down, and seals the stored file
at 0600 with every execute bit cleared.
* The capability set was too small to do the job. CAP_DAC_READ_SEARCH
lets us read a threat but not unlink it, so quarantine failed EACCES
as root. The set is now four capabilities — SYS_ADMIN, DAC_READ_SEARCH,
DAC_OVERRIDE, FOWNER. DAC_OVERRIDE is close to "write anywhere" and
that is worth being honest about; an antivirus that quarantines cannot
avoid it, because the threat is by definition in a directory somebody
else owns. What the reduction still buys is what it excludes, and
there is a test asserting SYS_MODULE, SYS_BOOT, SYS_PTRACE, NET_ADMIN,
NET_RAW, AUDIT_CONTROL and SETUID never creep back in. Narrowing
further means a separate privileged helper for quarantine.
realtime.rs is now documented as the unprivileged fallback and does not
start when the gate is armed — running both would scan everything twice
and quarantine the same file from two threads.
99 tests pass. HOUNDD_GATE_DEBUG=1 dumps every event and decision.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FAN_OPEN_EXEC_PERM hands us the open and waits for an answer, so a
binary can be refused before it runs. inotify could only report what
had already happened.
Verified end to end as root against a dedicated tmpfs (examples/
gate-smoke.rs, three phases):
benign binary ran 7.2 ms
EICAR binary blocked 1.8 ms never executed
scanner stalled 5 s ran 1.6 s watchdog rescued 3 events
The third phase is the one that matters. A gate that can hold a process
forever is a machine-wedging bug wearing a feature's clothes, so the
watchdog answers ALLOW for anything unanswered past DEADLINE and counts
it. A missed detection is a bad day; a frozen machine ends the product.
Two things this cost, both worth recording:
* Scanning by re-opening the path deadlocks the daemon against itself.
The open() lands on the watched mount and queues a permission event
behind the one we are currently answering, and we cannot answer that
one until we finish this one. Allowing our own pid does not help —
the thread never gets back to the queue to apply the rule. The gate
reads through the descriptor the kernel already handed it, with
pread so the gated process still sees its own file offset. This is
what hung the first smoke run.
* The watchdog can only rescue events it has been told about, and it
learns of them when the queue is drained. Scanning on the draining
thread makes every event behind a slow scan invisible to the
deadline. Reader and workers are therefore separate threads: the
reader never blocks on a scan, so every event is registered within
microseconds of arriving.
Also:
- ScanEngine::scan_bytes — the seam the gate needs, since it must never
scan by path. Engines that cannot do it return None and simply are
not usable behind the gate.
- Settings gain exec_gate and exec_gate_paths, defaulting to OFF. It
needs CAP_SYS_ADMIN and a root-filesystem mark holds every process on
the box; that is not a default to ship before Phase 2 soak testing.
- ABI constants are defined locally rather than taken from libc, so a
version bump cannot quietly change what we ask the kernel for.
78 tests pass, up from 57.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hound