Armed the gate on / on the live server. Aborted after about twenty
seconds. The box was never at risk — Caddy stayed sub-millisecond and
load never rose — but the gate blocked reads of an AI agent's session
transcript, reporting it as Linux.Coinminer.XMRig.
It was not wrong about the bytes. That transcript contains
"stratum+tcp://", "donate-level" and "xmrig" because the miner rule was
being written in that session. The rule matched a document ABOUT
malware.
Three bugs, none of which the tmpfs stage could have shown:
1. The miner rule had no file-type condition, so any text mentioning
mining tripped it: threat-intelligence reports, security blog posts,
support tickets, an antivirus's own logs. It now requires ELF magic,
as the rootkit rule always did. Two regression tests: a transcript
discussing the rule is clean, and an ELF carrying the same strings
still matches — the fix must not cost the detection it exists for.
2. The gate requested FAN_OPEN_PERM, so it held every OPEN, not every
execve. A matching file could not be read by anything. That is a
different product from the one advertised, and on a multi-tenant box
it is a denial of service against the operator rather than a defence.
Read events are no longer requested at all. FAN_OPEN_EXEC_PERM and
FAN_CLOSE_WRITE cover the threat: execution is refused before it
happens, and anything malicious written to disk is quarantined when
the write completes. An interpreted script is caught as it lands
rather than as it is read — the same protection, one step earlier.
`serve` also guards deny-on-exec explicitly, so re-requesting read
events later cannot silently restore the old behaviour.
3. Hound did not exclude its own state. /var/lib/hound and /run/hound
are now always excluded; the vault holds live malware by definition.
Henry asked whether the single watchdog rescue was queue pressure or
scan time. It was scan time: the gate inherited the on-demand 100 MB
limit and tried to read and match a multi-megabyte transcript inline
while holding a process. A gate's budget is a deadline, not a size, so
it now caps at 32 MB — anything larger is allowed through unread rather
than turned into a rescue, which is a process released unscanned and
worse than never having looked.
Dropping read events made everything faster, because most opens on a
running machine are reads:
latency +1.38 -> +0.79 ms per exec
throughput 2,680 -> 4,178 execs/sec (58% of ungated, was 36%)
events 1,179 in five seconds on an idle tmpfs -> 1
Re-verified on the tmpfs: an ELF miner is quarantined before it can even
be made executable, a document naming every one of its strings is
readable, and a clean binary runs.
297 tests pass. The gate stays off; stage 3 gets attempted again with
these fixes and fresh numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Armed the execution gate on the live server for the first time. It
reported itself armed on a dedicated tmpfs, and then let EICAR execute.
Counters: 0 allowed, 0 blocked. Not one event was ever delivered.
Cause: systemd gives the service a PRIVATE MOUNT NAMESPACE. Several
perfectly ordinary hardening options force one — ProtectProc,
ProtectKernelTunables, ProtectControlGroups — and none of them mention
it. FAN_MARK_MOUNT marks a vfsmount, and a private namespace holds its
own vfsmount for the same filesystem. So the daemon marked its copy,
every other process on the machine used the host's copy, and the gate
protected nothing while claiming to be armed.
That is the worst way for a security feature to fail: silently, with a
reassuring status line. Nothing in the unit tests could have caught it —
they run in the host namespace, where the mount mark works.
Fixed by always using FAN_MARK_FILESYSTEM, which marks the SUPERBLOCK.
A superblock is shared across namespaces, so events arrive from
everywhere, and scoping still works because a superblock is exactly one
filesystem: marking a dedicated mount covers that mount and nothing
else. mark_mount is kept for the smoke-test example, which runs outside
systemd, with a doc comment about when it lies to you.
Two more that only appeared once the gate was actually armed:
* SystemCallFilter=@system-service kills the daemon with SIGSYS the
moment the gate is switched on. fanotify_init and fanotify_mark live
in @privileged, which @system-service deliberately excludes. Granted
individually rather than by adding @privileged, which would also admit
setuid, chroot, bpf and kexec_load. Invisible until armed — the
service starts fine with the gate off.
* The capability reduction reported "60 capabilities could not be
dropped" while the end state was perfectly correct. systemd's
CapabilityBoundingSet had already done the work, and the service does
not hold CAP_SETPCAP afterwards, so every redundant drop failed EPERM.
It now checks what is actually present, attempts only that, and judges
by the end state rather than by return codes.
Also removed AmbientCapabilities from the unit. Ambient capabilities are
inherited by children, the daemon shells out to freshclam/rpm/pacman on
some paths, and a root process already receives the bounding set as
permitted — so it bought nothing except a way for CAP_SYS_ADMIN to leak
into a subprocess.
Performance, measured on the live server rather than guessed at:
+2.70 ms/exec as first written
+1.47 ms/exec after the reader blocked on poll() instead of sleeping
a millisecond between empty reads — that sleep sat on
the critical path of every execve
+1.38 ms/exec after answering cache hits in the reader thread, with
no channel handoff or worker wakeup
2,680 execs/sec sustained through the gate, 16-way parallel, with
ZERO watchdog rescues — the queue never fell behind. Ungated is 7,455.
Caddy stayed at sub-millisecond throughout and load did not rise.
Joe and Henry are right that the exec-heavy paths on this box — Docker
overlays, agent workspaces, PM2 — are the performance bar rather than an
exclusion list. Protecting agent workspaces from injected payloads is
the product. 2,680/sec with no backlog is roughly ten times what this
machine generates, so the bar looks clearable; stage 2 will say for sure.
295 tests pass, and the three-phase gate smoke test still passes
including the fail-open case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Installed the .deb on the live server. The daemon did not start, and
everything below is what that one command surfaced.
1. MemoryDenyWriteExecute=yes stopped the service dead.
yara-x compiles rules to WebAssembly and JITs them, so it needs pages
that go writable then executable. With W^X enforced the daemon aborts
at startup: "unable to make memory executable". The unit had passed
systemd-analyze verify, which checks syntax and cannot know this.
Now off, with the reasoning in the unit rather than in a commit
nobody will read: a hardening directive that stops the service is
worse than the exposure it prevents, because the machine ends up with
no antivirus at all. What compensates is spelled out beside it.
2. Hound detected itself.
The goodware gate reported /usr/bin/houndd as Linux.Coinminer.XMRig
and Linux.Rootkit.Preload. Correctly: the built-in pack matches on
"stratum+tcp://", "xmrig", "RTLD_NEXT" and "ld.so.preload", and the
pack was embedded verbatim, so the daemon's own binary contained all
of them.
Not cosmetic. With the execution gate armed, Hound would have refused
to execute itself or quarantined its own binary — a scanner that eats
its own daemon the moment protection is switched on.
The pack is now XOR-masked at build time (build.rs) and unmasked at
startup. Not a secret — the rules are open source — the only job is
keeping the literal bytes out of the executable. Two regression tests:
the embedded blob carries no plaintext rule strings, and a built
daemon binary in target/ carries none either.
3. The postinst copied the built-in pack into /var/lib/hound/rules,
where the daemon compiled it a second time and logged a duplicate
declaration on every start. That directory is for ADDITIONAL packs;
the built-ins live in the binary. Removed from deb, rpm and AUR.
4. The daemon and the CLI disagreed about the socket. systemd gives the
service /run/hound; the CLI looked in $XDG_RUNTIME_DIR and reported
the daemon unreachable — technically true, entirely unhelpful.
default_socket_path() now prefers /run/hound when it exists, the unit
states HOUNDD_SOCK explicitly, and a permission error on the socket
says "try: sudo hound" instead of "Permission denied".
Also: Recommends: clamav-daemon was wrong and apt duly installed clamd,
which took 970 MB of RSS on the live server. clamd is an optional
arm's-length engine, so it is a Suggests. I stopped and disabled the
copy my install pulled in.
Verified on the server after fixing: service active, status reports the
engine and the gate, EICAR caught, rootkit scan clean, and the goodware
gate passes across 3,955 system binaries including the now-installed
houndd.
The gate remains OFF. Turning it on for the host that serves Caddy is a
separate decision.
295 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes Phase 5. Half of a Linux compromise is not a file on disk, it
is a line added to a startup file — a curl in a shell profile, a systemd
unit with a dull name, one extra key in authorized_keys. The payload is
often unremarkable; what makes it an incident is that it survives a
reboot and nobody reads those files from one year to the next.
So this is not a scanner but an inventory with a memory. It records
systemd units (system and per-user), cron in all its locations,
autostart entries, shell profiles, authorized_keys and ld.so.preload,
then reports what CHANGED.
Three decisions, all of which are the difference between a report people
read and one they turn off:
* Content is hashed, not stat'd. An mtime can be set backwards with one
touch, and someone editing a startup file is exactly the person who
would. Verified: a backdated edit is still caught.
* A first run reports no changes and says so. Everything would be a
change, and a first-run report full of alarms is one nobody reads.
What a first run can honestly say is how many entries no package
claims — 97 of 1,026 on this machine — because that is true
regardless of history.
* Writing the baseline is an explicit act (`--accept`, or
update_baseline on the wire). A plain check must never quietly record
whatever is currently installed as normal; that is how a compromise
becomes the new baseline.
Package ownership decides what is ordinary: a unit that arrived with a
package is the system working, the same unit unowned is somebody's
decision. Reuses the merged-/usr-aware index from the rootkit rewrite,
with a test asserting most units resolve to a package — if that ratio
collapses, ownership lookup has broken and the whole report is noise.
Exercised end to end against this machine: baseline of 1,026 items,
a planted user unit caught as ADDED, an in-place edit with a backdated
mtime caught as CHANGED, and its deletion caught as REMOVED. The test
artifact was removed afterwards.
Cross-distro verification of the rootkit rewrite is now MET. Henry ran
the suite on the Ubuntu box (26.04, glibc 2.43): 20/20, including both
unowned_setuid_does_not_fire_on_a_healthy_system and
no_false_positives_on_system_binaries.
214 tests pass across the workspace.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 2. "Distro-agnostic" was a claim with no packaging behind it.
.deb and AppImage are both built and tested here; the rpm spec and
PKGBUILD are written but not yet built, since neither rpmbuild nor
makepkg exists on this machine.
The AppImage is deliberately not a packaged daemon. An AppImage is
unprivileged by design — no install, no root, no systemd — so the
execution gate cannot exist in one, and pretending otherwise would be
worse than saying so. What it is instead is the try-before-you-install
build: on-demand scanning, quarantine under the user's own data dir,
rootkit and supply-chain checks, all with nothing left behind. Asking
it for the gate prints why it cannot and how to get it. Verified with
an isolated HOME: status, scan, EICAR caught, gate refusal.
Two packaging bugs caught by looking rather than assuming:
* mktemp -d creates 0700 and dpkg applies the staging root's mode to
"/". Installing that package would have chmodded the root directory
to 0700 and broken the machine.
* AppRun wrote its log before creating the directory, and built a
socket path that can exceed sun_path (108 bytes) when
XDG_RUNTIME_DIR is long. Both fixed; the socket falls back rather
than failing with an error nobody can act on.
The systemd unit is hardened as far as this daemon can be. Notably
ProtectSystem=strict is WRONG here and is left off on purpose: it
remounts everything read-only, and quarantine has to remove a threat
from wherever it landed. ProtectHome is off for the same reason. The
CapabilityBoundingSet mirrors what caps.rs drops to, so the machine is
protected even if the binary is replaced by one that does not reduce
itself.
App icon: the mark in white on a periwinkle tile, per Joe. Small sizes
are not the same artwork scaled down — the mark is line-weight, so at
16px a 62% inset leaves about a pixel and a half of stroke and the head
turns to mush. Each size is authored with its own inset and corner
radius, and the ground goes flat below 32px because a gradient across
16 pixels is just noise. The tray ladder is untouched: those glyphs stay
transparent and re-tint per state so they can sit on any panel colour.
Package installs are NOT enabled by default beyond the daemon itself —
exec_gate stays off until the operator turns it on, in every packaging
format.
99 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Hound