gate: the mark was invisible to every process but our own

Armed the execution gate on the live server for the first time. It
reported itself armed on a dedicated tmpfs, and then let EICAR execute.
Counters: 0 allowed, 0 blocked. Not one event was ever delivered.

Cause: systemd gives the service a PRIVATE MOUNT NAMESPACE. Several
perfectly ordinary hardening options force one — ProtectProc,
ProtectKernelTunables, ProtectControlGroups — and none of them mention
it. FAN_MARK_MOUNT marks a vfsmount, and a private namespace holds its
own vfsmount for the same filesystem. So the daemon marked its copy,
every other process on the machine used the host's copy, and the gate
protected nothing while claiming to be armed.

That is the worst way for a security feature to fail: silently, with a
reassuring status line. Nothing in the unit tests could have caught it —
they run in the host namespace, where the mount mark works.

Fixed by always using FAN_MARK_FILESYSTEM, which marks the SUPERBLOCK.
A superblock is shared across namespaces, so events arrive from
everywhere, and scoping still works because a superblock is exactly one
filesystem: marking a dedicated mount covers that mount and nothing
else. mark_mount is kept for the smoke-test example, which runs outside
systemd, with a doc comment about when it lies to you.

Two more that only appeared once the gate was actually armed:

* SystemCallFilter=@system-service kills the daemon with SIGSYS the
  moment the gate is switched on. fanotify_init and fanotify_mark live
  in @privileged, which @system-service deliberately excludes. Granted
  individually rather than by adding @privileged, which would also admit
  setuid, chroot, bpf and kexec_load. Invisible until armed — the
  service starts fine with the gate off.

* The capability reduction reported "60 capabilities could not be
  dropped" while the end state was perfectly correct. systemd's
  CapabilityBoundingSet had already done the work, and the service does
  not hold CAP_SETPCAP afterwards, so every redundant drop failed EPERM.
  It now checks what is actually present, attempts only that, and judges
  by the end state rather than by return codes.

Also removed AmbientCapabilities from the unit. Ambient capabilities are
inherited by children, the daemon shells out to freshclam/rpm/pacman on
some paths, and a root process already receives the bounding set as
permitted — so it bought nothing except a way for CAP_SYS_ADMIN to leak
into a subprocess.

Performance, measured on the live server rather than guessed at:

  +2.70 ms/exec   as first written
  +1.47 ms/exec   after the reader blocked on poll() instead of sleeping
                  a millisecond between empty reads — that sleep sat on
                  the critical path of every execve
  +1.38 ms/exec   after answering cache hits in the reader thread, with
                  no channel handoff or worker wakeup

  2,680 execs/sec sustained through the gate, 16-way parallel, with
  ZERO watchdog rescues — the queue never fell behind. Ungated is 7,455.
  Caddy stayed at sub-millisecond throughout and load did not rise.

Joe and Henry are right that the exec-heavy paths on this box — Docker
overlays, agent workspaces, PM2 — are the performance bar rather than an
exclusion list. Protecting agent workspaces from injected payloads is
the product. 2,680/sec with no backlog is roughly ten times what this
machine generates, so the bar looks clearable; stage 2 will say for sure.

295 tests pass, and the three-phase gate smoke test still passes
including the fail-open case.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

crates/houndd/examples/gate-smoke.rs

@@ -113,6 +113,9 @@ fn run(mount: &Path) -> bool {
         Arc::new(|p: &Path, name: &str, v: fanotify::Verdict| {
             println!("    detect: {} ({name}) {v:?}", p.display());
         }),
+        // No fast path in the smoke test: every event must reach a worker,
+        // so phase 3 can genuinely stall one.
+        Arc::new(|_e: &fanotify::Event| None),
     );
     println!("gate armed on {MOUNT} — 1 reader, 4 workers, watchdog live");
     std::thread::sleep(Duration::from_millis(100));

crates/houndd/src/caps.rs

@@ -118,22 +118,41 @@ pub fn retain_only(caps: &[u32]) -> io::Result<bool> {
     }
 
     let (lo, hi) = to_words(caps);
+    let want = lo as u64;
 
-    // 1. Bounding set first, while CAP_SETPCAP is still held. Dropping one
+    // 1. Bounding set first, while CAP_SETPCAP is still held.
-    //    we never had, or one this kernel does not define, returns EINVAL
+    //
-    //    and is not interesting; anything else is worth knowing about.
+    //    Only capabilities actually PRESENT are attempted. Under systemd
+    //    the unit's CapabilityBoundingSet= has already narrowed the set,
+    //    and the service does not hold CAP_SETPCAP afterwards — so
+    //    blindly dropping all sixty-odd fails EPERM sixty-odd times and
+    //    reports a scary error about work that was already done
+    //    correctly. Checking first makes the operation idempotent, which
+    //    is what it should have been.
+    let present = bounding_now().unwrap_or(u64::MAX);
     let mut refused = 0usize;
     for c in 0..=CAP_LAST_CAP_CEILING {
         if caps.contains(&c) {
             continue;
         }
+        if c < 64 && present & (1u64 << c) == 0 {
+            continue; // already gone
+        }
         // SAFETY: prctl with a constant option and a capability number.
         let rc = unsafe { libc::prctl(PR_CAPBSET_DROP, c as libc::c_ulong, 0, 0, 0) };
         if rc != 0 && io::Error::last_os_error().raw_os_error() != Some(libc::EINVAL) {
             refused += 1;
         }
     }
-    if refused > 0 {
+    // Judge by the end state, not by the return codes: what matters is
+    // that nothing beyond `caps` remains, however it got that way.
+    if let Some(after) = bounding_now() {
+        if after & !want != 0 {
+            return Err(io::Error::other(format!(
+                "{refused} capabilities could not be dropped; CapBnd is {after:#x}, wanted {want:#x}"
+            )));
+        }
+    } else if refused > 0 {
         return Err(io::Error::other(format!(
             "{refused} capabilities could not be dropped from the bounding set"
         )));

crates/houndd/src/engine.rs

@@ -50,6 +50,16 @@ pub trait ScanEngine: Send + Sync {
     fn scan_bytes(&self, _bytes: &[u8]) -> Option<String> {
         None
     }
+
+    /// Answer from memory alone, without reading the file.
+    ///
+    /// `Some(verdict)` means we have judged this exact file version
+    /// before; `None` means it must be scanned. The execution gate uses
+    /// this to answer repeat executions without waking a worker, which is
+    /// most of the traffic on a machine that is actually doing something.
+    fn cached_verdict(&self, _md: &std::fs::Metadata) -> Option<Option<String>> {
+        None
+    }
 }
 
 /// The ClamAV-backed engine: `clamscan` + `freshclam` over their

crates/houndd/src/fanotify.rs

@@ -142,6 +142,19 @@ impl Event {
         std::fs::read_link(format!("/proc/self/fd/{}", self.fd)).ok()
     }
 
+    /// Full metadata for the descriptor, for cache lookups that must not
+    /// read the file.
+    pub fn metadata(&self) -> Option<std::fs::Metadata> {
+        use std::os::unix::io::{AsRawFd, FromRawFd};
+        // SAFETY: borrowed for the duration of the call and immediately
+        // forgotten, so the descriptor is not closed twice.
+        let f = unsafe { std::fs::File::from_raw_fd(self.fd) };
+        let md = f.metadata().ok();
+        let _ = f.as_raw_fd();
+        std::mem::forget(f);
+        md
+    }
+
     /// Size of the file behind the descriptor, via `fstat`.
     pub fn size(&self) -> Option<u64> {
         // SAFETY: zeroed stat is a valid initial value; fd is ours.
@@ -385,14 +398,29 @@ impl Gate {
         Arc::clone(&self.responder)
     }
 
-    /// Watch a whole mount. Used in tests against a dedicated tmpfs so a
+    /// Watch a mount, by its vfsmount.
-    /// bug cannot reach the real filesystem.
+    ///
+    /// **Only safe when the watcher shares a mount namespace with the
+    /// processes it is watching.** A vfsmount is per-namespace, so a
+    /// daemon in a private namespace marks its own copy and receives
+    /// nothing from anybody else — while still reporting itself armed.
+    /// Under systemd that is the default situation, because several
+    /// ordinary hardening options force a private mount namespace.
+    ///
+    /// Kept for the smoke-test example, which runs outside systemd.
+    /// Production uses [`Gate::mark_filesystem`].
     pub fn mark_mount(&self, path: &Path) -> io::Result<()> {
         self.mark(path, FAN_MARK_ADD | FAN_MARK_MOUNT)
     }
 
-    /// Watch an entire filesystem. This is the production mark, and the
+    /// Watch an entire filesystem, by its superblock.
-    /// reason the watchdog is not optional.
+    ///
+    /// A superblock is shared across mount namespaces, so this sees every
+    /// process on the machine regardless of where the watcher lives. It
+    /// also still scopes correctly: a superblock is exactly one
+    /// filesystem, so marking a dedicated mount covers that and nothing
+    /// else. This is the production mark, and the reason the watchdog is
+    /// not optional.
     pub fn mark_filesystem(&self, path: &Path) -> io::Result<()> {
         self.mark(path, FAN_MARK_ADD | FAN_MARK_FILESYSTEM)
     }
@@ -417,6 +445,35 @@ impl Gate {
         Ok(())
     }
 
+    /// Wait until the group has events, or `timeout_ms` elapses.
+    ///
+    /// The reader used to poll: read, get `EAGAIN`, sleep a millisecond,
+    /// try again. That millisecond lands on the critical path of every
+    /// `execve` on a watched filesystem — measured at +2.7 ms per exec on
+    /// this machine, nearly all of it waiting for a sleep to finish rather
+    /// than doing any work. Blocking on the descriptor means the reader
+    /// wakes when the kernel has something to say and not before.
+    ///
+    /// The timeout exists only so the loop can notice `stop()`.
+    pub fn wait_readable(&self, timeout_ms: i32) -> io::Result<bool> {
+        let mut fds = libc::pollfd {
+            fd: self.responder.fan.as_raw_fd(),
+            events: libc::POLLIN,
+            revents: 0,
+        };
+        // SAFETY: one valid descriptor, and the struct outlives the call.
+        let rc = unsafe { libc::poll(&mut fds, 1, timeout_ms) };
+        if rc < 0 {
+            let err = io::Error::last_os_error();
+            // A signal interrupting the wait is not an error.
+            if err.kind() == io::ErrorKind::Interrupted {
+                return Ok(false);
+            }
+            return Err(err);
+        }
+        Ok(rc > 0 && fds.revents & libc::POLLIN != 0)
+    }
+
     /// Read whatever events are pending. Returns empty on `EAGAIN`.
     pub fn read_events(&self) -> io::Result<Vec<Event>> {
         let mut buf = [0u8; 8192];
@@ -521,6 +578,7 @@ impl Gate {
         cfg: GateConfig,
         scan: Arc<dyn Fn(&Path, &[u8]) -> Option<String> + Send + Sync>,
         on_detect: Arc<dyn Fn(&Path, &str, Verdict) + Send + Sync>,
+        fast: Arc<dyn Fn(&Event) -> Option<bool> + Send + Sync>,
     ) -> Vec<std::thread::JoinHandle<()>> {
         let (tx, rx) = std::sync::mpsc::channel::<Event>();
         let rx = Arc::new(Mutex::new(rx));
@@ -535,13 +593,41 @@ impl Gate {
                     .name("hound-gate-reader".into())
                     .spawn(move || {
                         while running.load(Ordering::SeqCst) {
-                            match gate.read_events() {
+                            // Block until the kernel has something, so no
-                                Ok(events) if events.is_empty() => {
+                            // exec ever waits on a timer of ours.
-                                    std::thread::sleep(Duration::from_millis(1));
+                            match gate.wait_readable(100) {
+                                Ok(false) => continue,
+                                Ok(true) => {}
+                                Err(e) => {
+                                    eprintln!("gate: poll failed: {e}");
+                                    std::thread::sleep(Duration::from_millis(20));
+                                    continue;
                                 }
+                            }
+                            match gate.read_events() {
+                                Ok(events) if events.is_empty() => {}
                                 Ok(events) => {
                                     for e in events {
                                         let (seq, fd) = (e.seq, e.fd);
+
+                                        // Fast path: a verdict already in
+                                        // memory is answered here, without
+                                        // the channel handoff or a worker
+                                        // wakeup. Re-executing a binary we
+                                        // have already judged is the
+                                        // overwhelmingly common event on a
+                                        // busy machine, and it costs an
+                                        // fstat and a map lookup — the
+                                        // reader still never scans, which
+                                        // is what keeps the watchdog
+                                        // meaningful.
+                                        if e.needs_response() {
+                                            if let Some(allow) = fast(&e) {
+                                                gate.answer_and_retire(seq, fd, allow);
+                                                continue;
+                                            }
+                                        }
+
                                         if tx.send(e).is_err() {
                                             // No workers left: fail open.
                                             gate.answer_and_retire(seq, fd, true);

crates/houndd/src/main.rs

@@ -223,15 +223,26 @@ impl DaemonState {
         } else {
             s.exec_gate_paths.clone()
         };
+        // ALWAYS mark the filesystem, never the mount — even when the
+        // operator scoped the gate to one path.
+        //
+        // systemd gives this service a private mount namespace (any of
+        // ProtectProc, ProtectKernelTunables or ProtectControlGroups is
+        // enough to force one). FAN_MARK_MOUNT marks a vfsmount, and a
+        // private namespace holds its own vfsmount for the same
+        // filesystem — so the daemon marks its copy, every other process
+        // on the machine uses the host's copy, and not one event is ever
+        // delivered. The gate reports itself armed and silently protects
+        // nothing, which is the worst way for a security feature to fail.
+        //
+        // FAN_MARK_FILESYSTEM marks the SUPERBLOCK, which is shared
+        // across namespaces. Scoping still works, because a superblock is
+        // exactly one filesystem: marking a dedicated mount covers that
+        // mount and nothing else.
         let mut marked = Vec::new();
         for p in &paths {
             let path = std::path::Path::new(p);
-            let result = if s.exec_gate_paths.is_empty() {
+            match gate.mark_filesystem(path) {
-                gate.mark_filesystem(path)
-            } else {
-                gate.mark_mount(path)
-            };
-            match result {
                 Ok(()) => marked.push(p.clone()),
                 Err(e) => {
                     events.push("gate", "warn", format!("could not watch {p}: {e}"));
@@ -329,6 +340,15 @@ impl DaemonState {
                     }
                 },
             ),
+            // Fast path: fstat the descriptor the kernel already gave us
+            // and ask the engine whether it has judged this exact file
+            // version before. No read, no scan, no worker.
+            std::sync::Arc::new(|event: &fanotify::Event| {
+                let md = event.metadata()?;
+                engine::engine()
+                    .cached_verdict(&md)
+                    .map(|verdict| verdict.is_none())
+            }),
         );
 
         events.push(

crates/houndd/src/native.rs

@@ -162,6 +162,12 @@ impl ScanEngine for HoundEngine {
         })
     }
 
+    fn cached_verdict(&self, md: &std::fs::Metadata) -> Option<Option<String>> {
+        self.cache
+            .get(&FileKey::from_metadata(md))
+            .map(|v| v.map(|name| name.to_string()))
+    }
+
     fn scan_bytes(&self, bytes: &[u8]) -> Option<String> {
         let set = self.rules.current();
         let mut scanner = yara_x::Scanner::new(&set.rules);

packaging/systemd/houndd.service

@@ -25,7 +25,12 @@ RestartSec=2s
 # someone else needs DAC_OVERRIDE; stripping the execute bit off a file
 # we do not own needs FOWNER.
 CapabilityBoundingSet=CAP_SYS_ADMIN CAP_DAC_READ_SEARCH CAP_DAC_OVERRIDE CAP_FOWNER
-AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_READ_SEARCH CAP_DAC_OVERRIDE CAP_FOWNER
+# AmbientCapabilities is deliberately NOT set. Ambient capabilities are
+# inherited by child processes, and the daemon shells out to freshclam,
+# rpm and pacman on some paths — none of which should start life holding
+# CAP_SYS_ADMIN. A process running as root already receives everything in
+# the bounding set as permitted and effective, so Ambient adds nothing
+# here except a way for it to leak.
 NoNewPrivileges=yes
 
 # ── Filesystem ───────────────────────────────────────────────────────
@@ -74,6 +79,16 @@ LockPersonality=yes
 RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
 SystemCallArchitectures=native
 SystemCallFilter=@system-service
+# fanotify_init and fanotify_mark live in @privileged, which
+# @system-service deliberately excludes — so the base filter kills the
+# daemon with SIGSYS the moment the execution gate is switched on. This
+# was invisible until the gate was armed for the first time on a real
+# install: the service starts fine with the gate off.
+#
+# Granted individually rather than by adding @privileged, which would
+# also admit setuid, chroot, bpf, kexec_load and pivot_root. Two
+# syscalls is the whole requirement.
+SystemCallFilter=fanotify_init fanotify_mark
 SystemCallFilter=~@clock @cpu-emulation @debug @module @mount @obsolete @raw-io @reboot @swap
 UMask=0077