Antivirus/crates/hound-defs/src/pack.rs
Hound 3be16eeef9 hound-defs: OSV ingest, the IOC index, and signed packs
Phase 3's foundation. Three jobs that share a data model:

  osv    parse the ossf/malicious-packages feed (Apache-2.0, ~226k
         records, daily) into indicators
  index  answer "is this package known bad?" fast enough to ask it
         thousands of times per project sweep
  pack   sign a definitions pack, and verify one before loading it

Validated against the real feed rather than fixtures: the whole
crates.io OSV export, 2,749 records, parsed with zero failures — 3,885
indicators across ten ecosystems, 19 of them MAL-. `rustdecimal` (the
real typosquat of rust_decimal) resolves in crates.io and stays clean
in npm and PyPI, which is the ecosystem isolation working.

Notes on the three:

* A malicious-package record is not a vulnerability record. It almost
  always carries introduced:"0" with no fix, meaning EVERY version is
  malicious — the package exists only to be malware, so there is no safe
  version to upgrade to. Conflating that with a version-bounded
  vulnerability either misses real hits or condemns safe versions of
  legitimate packages, so the two are modelled separately.

* The index is a cuckoo filter in front of a map. Cuckoo rather than
  bloom specifically because a definitions feed needs DELETION: OSV
  withdraws records — it once withdrew 157 malware reports after a
  false-positive incident — and a filter you cannot remove from means a
  withdrawn record costs a probe forever or forces a rebuild. 226k
  indicators fit in under 4 MB; the crates.io set is 8 KB.

  The property that must never break is no false negatives, and it has
  its own test. A false positive costs a hash lookup; a false negative
  is malware reported as clean. That is also why a fingerprint hashing
  to zero is nudged to one — zero marks an empty slot, so without the
  nudge one key in 65,536 would silently vanish.

* Signing is not about entitlement; the subscription gates the server.
  It answers "is this pack really from us?", because someone who can
  substitute a definitions file can add an entry for /usr/bin/sudo and
  have Hound quarantine it on every machine that updates — a supply
  chain attack delivered through the security product. Verification
  happens on the raw bytes BEFORE anything parses them, so a hostile
  pack never reaches the parser at all.

256 tests pass across the workspace.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 07:06:00 -05:00

355 lines
13 KiB
Rust

//! Signed definition packs.
//!
//! Hound's agent is Apache-2.0 and anyone can build it. The subscription
//! holds because the *server* only serves packs to a valid licence, not
//! because a boolean in an open binary says so — that gets patched out in
//! ten minutes.
//!
//! Signing is therefore not about entitlement. It answers a different and
//! more important question: **is this pack really from us?** A definitions
//! file is a list of things the scanner will act on. Someone who can
//! substitute one can add an entry for `/usr/bin/sudo` and have Hound
//! quarantine it on every machine that updates — a supply-chain attack
//! delivered through the security product, which is the worst shape this
//! can take.
//!
//! So: Ed25519 over the exact bytes, the public key compiled into the
//! agent, and verification **before** parsing. Not after, and not "parse,
//! then check" — a malformed pack must never reach the parser at all.
use crate::osv::Indicator;
use ed25519_dalek::{Signature, Signer, SigningKey, Verifier, VerifyingKey};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
/// What a pack carries.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct Pack {
/// Feed version, e.g. "2026.08.21". This is what `hound status` shows.
pub version: String,
/// When it was built, RFC3339.
pub created: String,
/// Free-text provenance, so a user can see where content came from.
pub sources: Vec<String>,
pub indicators: Vec<Indicator>,
}
impl Pack {
/// Serialise to the exact bytes that get signed.
///
/// Deterministic on purpose: the same input must produce the same
/// bytes on every machine, or the signature is unverifiable and the
/// hash is not a version. `serde_json` preserves struct field order
/// and `Vec` order, so the only requirement is that the caller does
/// not reorder indicators between build and sign.
pub fn to_bytes(&self) -> Result<Vec<u8>, Error> {
serde_json::to_vec(self).map_err(|e| Error::Encode(e.to_string()))
}
pub fn sha256(&self) -> Result<String, Error> {
let bytes = self.to_bytes()?;
let mut h = Sha256::new();
h.update(&bytes);
Ok(format!("{:x}", h.finalize()))
}
}
/// A pack plus its detached signature.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SignedPack {
/// The pack, as the exact bytes that were signed. Kept as bytes rather
/// than as a `Pack` so verification happens against what was actually
/// signed, not against a re-encoding of a parsed value.
#[serde(with = "base64_bytes")]
pub payload: Vec<u8>,
#[serde(with = "base64_bytes")]
pub signature: Vec<u8>,
/// Which key signed it, so keys can be rotated without a flag day.
pub key_id: String,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Error {
Encode(String),
/// The signature did not verify. The pack is discarded untouched.
BadSignature,
/// Signed by a key we do not trust.
UnknownKey(String),
Malformed(String),
}
impl std::fmt::Display for Error {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Error::Encode(e) => write!(f, "encoding the pack: {e}"),
Error::BadSignature => write!(
f,
"the definitions pack is not signed by Hound and was discarded"
),
Error::UnknownKey(id) => write!(f, "pack signed by unknown key {id}"),
Error::Malformed(e) => write!(f, "malformed pack: {e}"),
}
}
}
impl std::error::Error for Error {}
/// Sign a pack. Build-side only; the agent never holds a signing key.
pub fn sign(pack: &Pack, signing_key: &SigningKey, key_id: &str) -> Result<SignedPack, Error> {
let payload = pack.to_bytes()?;
let signature = signing_key.sign(&payload);
Ok(SignedPack {
payload,
signature: signature.to_bytes().to_vec(),
key_id: key_id.to_string(),
})
}
/// Verify and decode a pack.
///
/// The order is the point: the signature is checked against the raw bytes
/// **before** anything parses them. A pack that fails verification is
/// never handed to the JSON parser, so a hostile pack cannot reach the
/// parser's attack surface at all.
pub fn verify(signed: &SignedPack, trusted: &[(&str, VerifyingKey)]) -> Result<Pack, Error> {
let Some((_, key)) = trusted.iter().find(|(id, _)| *id == signed.key_id) else {
return Err(Error::UnknownKey(signed.key_id.clone()));
};
let sig_bytes: [u8; 64] = signed
.signature
.as_slice()
.try_into()
.map_err(|_| Error::BadSignature)?;
let signature = Signature::from_bytes(&sig_bytes);
key.verify(&signed.payload, &signature)
.map_err(|_| Error::BadSignature)?;
serde_json::from_slice(&signed.payload).map_err(|e| Error::Malformed(e.to_string()))
}
/// Base64 for the byte fields, so a signed pack is a plain JSON file.
mod base64_bytes {
use serde::{Deserialize, Deserializer, Serializer};
const ALPHABET: &[u8; 64] =
b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
pub fn encode(data: &[u8]) -> String {
let mut out = String::with_capacity(data.len().div_ceil(3) * 4);
for chunk in data.chunks(3) {
let b = [chunk[0], *chunk.get(1).unwrap_or(&0), *chunk.get(2).unwrap_or(&0)];
let n = ((b[0] as u32) << 16) | ((b[1] as u32) << 8) | b[2] as u32;
out.push(ALPHABET[(n >> 18) as usize & 63] as char);
out.push(ALPHABET[(n >> 12) as usize & 63] as char);
out.push(if chunk.len() > 1 {
ALPHABET[(n >> 6) as usize & 63] as char
} else {
'='
});
out.push(if chunk.len() > 2 {
ALPHABET[n as usize & 63] as char
} else {
'='
});
}
out
}
pub fn decode(s: &str) -> Option<Vec<u8>> {
let mut rev = [255u8; 256];
for (i, c) in ALPHABET.iter().enumerate() {
rev[*c as usize] = i as u8;
}
let clean: Vec<u8> = s.bytes().filter(|b| *b != b'=' && !b.is_ascii_whitespace()).collect();
let mut out = Vec::with_capacity(clean.len() * 3 / 4);
for chunk in clean.chunks(4) {
let mut n = 0u32;
for (i, b) in chunk.iter().enumerate() {
let v = rev[*b as usize];
if v == 255 {
return None;
}
n |= (v as u32) << (18 - 6 * i);
}
out.push((n >> 16) as u8);
if chunk.len() > 2 {
out.push((n >> 8) as u8);
}
if chunk.len() > 3 {
out.push(n as u8);
}
}
Some(out)
}
pub fn serialize<S: Serializer>(data: &[u8], s: S) -> Result<S::Ok, S::Error> {
s.serialize_str(&encode(data))
}
pub fn deserialize<'de, D: Deserializer<'de>>(d: D) -> Result<Vec<u8>, D::Error> {
let s = String::deserialize(d)?;
decode(&s).ok_or_else(|| serde::de::Error::custom("invalid base64"))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::osv::Versions;
fn test_key() -> SigningKey {
// A fixed key so tests are deterministic. Never a real one.
SigningKey::from_bytes(&[7u8; 32])
}
fn a_pack() -> Pack {
Pack {
version: "2026.08.21".into(),
created: "2026-08-21T12:00:00Z".into(),
sources: vec!["ossf/malicious-packages".into()],
indicators: vec![Indicator {
ecosystem: "cratesio".into(),
name: "rustdecimal".into(),
versions: Versions::All,
id: "MAL-2022-1".into(),
summary: "malicious crate rustdecimal".into(),
}],
}
}
#[test]
fn a_signed_pack_round_trips() {
let key = test_key();
let signed = sign(&a_pack(), &key, "hound-2026").unwrap();
let trusted = [("hound-2026", key.verifying_key())];
let back = verify(&signed, &trusted).unwrap();
assert_eq!(back, a_pack());
}
// ── the attack this exists to stop ──
#[test]
fn a_tampered_payload_is_rejected() {
// Someone who can substitute a pack can add an entry for
// /usr/bin/sudo and have Hound quarantine it everywhere. This is
// the single most important test in the crate.
let key = test_key();
let mut signed = sign(&a_pack(), &key, "hound-2026").unwrap();
let pos = signed.payload.len() / 2;
signed.payload[pos] ^= 0x01;
let trusted = [("hound-2026", key.verifying_key())];
assert_eq!(verify(&signed, &trusted), Err(Error::BadSignature));
}
#[test]
fn a_pack_signed_by_a_different_key_is_rejected() {
let attacker = SigningKey::from_bytes(&[9u8; 32]);
let signed = sign(&a_pack(), &attacker, "hound-2026").unwrap();
let trusted = [("hound-2026", test_key().verifying_key())];
assert_eq!(verify(&signed, &trusted), Err(Error::BadSignature));
}
#[test]
fn an_unknown_key_id_is_rejected_by_name() {
let key = test_key();
let signed = sign(&a_pack(), &key, "somebody-elses-key").unwrap();
let trusted = [("hound-2026", key.verifying_key())];
assert!(matches!(
verify(&signed, &trusted),
Err(Error::UnknownKey(id)) if id == "somebody-elses-key"
));
}
#[test]
fn a_truncated_signature_is_rejected_rather_than_panicking() {
let key = test_key();
let mut signed = sign(&a_pack(), &key, "hound-2026").unwrap();
signed.signature.truncate(10);
let trusted = [("hound-2026", key.verifying_key())];
assert_eq!(verify(&signed, &trusted), Err(Error::BadSignature));
}
#[test]
fn a_hostile_payload_never_reaches_the_parser() {
// Verification happens on raw bytes first. Garbage that would
// upset the JSON parser is discarded before it gets there.
let key = test_key();
let signed = SignedPack {
payload: vec![0xff; 4096],
signature: vec![0u8; 64],
key_id: "hound-2026".into(),
};
let trusted = [("hound-2026", key.verifying_key())];
assert_eq!(verify(&signed, &trusted), Err(Error::BadSignature));
}
#[test]
fn a_validly_signed_but_malformed_pack_is_reported_as_malformed() {
// Distinct from BadSignature: this one is our own bug, not an
// attack, and conflating the two would send us hunting the wrong
// problem.
let key = test_key();
let payload = b"{ not json }".to_vec();
let signature = key.sign(&payload).to_bytes().to_vec();
let signed = SignedPack { payload, signature, key_id: "hound-2026".into() };
let trusted = [("hound-2026", key.verifying_key())];
assert!(matches!(verify(&signed, &trusted), Err(Error::Malformed(_))));
}
// ── determinism ──
#[test]
fn the_same_pack_always_produces_the_same_bytes() {
// A pack's hash is its identity. If encoding varied between
// machines, the signature would be unverifiable and the version
// meaningless.
assert_eq!(a_pack().to_bytes().unwrap(), a_pack().to_bytes().unwrap());
assert_eq!(a_pack().sha256().unwrap(), a_pack().sha256().unwrap());
assert_eq!(a_pack().sha256().unwrap().len(), 64);
}
#[test]
fn changing_one_indicator_changes_the_hash() {
let mut other = a_pack();
other.indicators[0].name = "rust_decimal".into();
assert_ne!(a_pack().sha256().unwrap(), other.sha256().unwrap());
}
// ── the signed pack is a plain JSON file ──
#[test]
fn a_signed_pack_serialises_to_json_and_back() {
let key = test_key();
let signed = sign(&a_pack(), &key, "hound-2026").unwrap();
let json = serde_json::to_string(&signed).unwrap();
let back: SignedPack = serde_json::from_str(&json).unwrap();
assert_eq!(back.payload, signed.payload);
assert_eq!(back.signature, signed.signature);
let trusted = [("hound-2026", key.verifying_key())];
assert!(verify(&back, &trusted).is_ok());
}
#[test]
fn base64_round_trips_every_length() {
for len in 0..200 {
let data: Vec<u8> = (0..len).map(|i| (i * 7 % 256) as u8).collect();
let encoded = base64_bytes::encode(&data);
assert_eq!(base64_bytes::decode(&encoded).as_deref(), Some(&data[..]), "len {len}");
}
}
#[test]
fn invalid_base64_is_rejected_rather_than_guessed_at() {
assert!(base64_bytes::decode("not base64 !!!").is_none());
}
#[test]
fn errors_read_like_something_a_person_can_act_on() {
assert!(Error::BadSignature.to_string().contains("discarded"));
assert!(Error::BadSignature.to_string().contains("Hound"));
}
}