Phase 3's foundation. Three jobs that share a data model:
osv parse the ossf/malicious-packages feed (Apache-2.0, ~226k
records, daily) into indicators
index answer "is this package known bad?" fast enough to ask it
thousands of times per project sweep
pack sign a definitions pack, and verify one before loading it
Validated against the real feed rather than fixtures: the whole
crates.io OSV export, 2,749 records, parsed with zero failures — 3,885
indicators across ten ecosystems, 19 of them MAL-. `rustdecimal` (the
real typosquat of rust_decimal) resolves in crates.io and stays clean
in npm and PyPI, which is the ecosystem isolation working.
Notes on the three:
* A malicious-package record is not a vulnerability record. It almost
always carries introduced:"0" with no fix, meaning EVERY version is
malicious — the package exists only to be malware, so there is no safe
version to upgrade to. Conflating that with a version-bounded
vulnerability either misses real hits or condemns safe versions of
legitimate packages, so the two are modelled separately.
* The index is a cuckoo filter in front of a map. Cuckoo rather than
bloom specifically because a definitions feed needs DELETION: OSV
withdraws records — it once withdrew 157 malware reports after a
false-positive incident — and a filter you cannot remove from means a
withdrawn record costs a probe forever or forces a rebuild. 226k
indicators fit in under 4 MB; the crates.io set is 8 KB.
The property that must never break is no false negatives, and it has
its own test. A false positive costs a hash lookup; a false negative
is malware reported as clean. That is also why a fingerprint hashing
to zero is nudged to one — zero marks an empty slot, so without the
nudge one key in 65,536 would silently vanish.
* Signing is not about entitlement; the subscription gates the server.
It answers "is this pack really from us?", because someone who can
substitute a definitions file can add an entry for /usr/bin/sudo and
have Hound quarantine it on every machine that updates — a supply
chain attack delivered through the security product. Verification
happens on the raw bytes BEFORE anything parses them, so a hostile
pack never reaches the parser at all.
256 tests pass across the workspace.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
355 lines
13 KiB
Rust
355 lines
13 KiB
Rust
//! Signed definition packs.
|
|
//!
|
|
//! Hound's agent is Apache-2.0 and anyone can build it. The subscription
|
|
//! holds because the *server* only serves packs to a valid licence, not
|
|
//! because a boolean in an open binary says so — that gets patched out in
|
|
//! ten minutes.
|
|
//!
|
|
//! Signing is therefore not about entitlement. It answers a different and
|
|
//! more important question: **is this pack really from us?** A definitions
|
|
//! file is a list of things the scanner will act on. Someone who can
|
|
//! substitute one can add an entry for `/usr/bin/sudo` and have Hound
|
|
//! quarantine it on every machine that updates — a supply-chain attack
|
|
//! delivered through the security product, which is the worst shape this
|
|
//! can take.
|
|
//!
|
|
//! So: Ed25519 over the exact bytes, the public key compiled into the
|
|
//! agent, and verification **before** parsing. Not after, and not "parse,
|
|
//! then check" — a malformed pack must never reach the parser at all.
|
|
|
|
use crate::osv::Indicator;
|
|
use ed25519_dalek::{Signature, Signer, SigningKey, Verifier, VerifyingKey};
|
|
use serde::{Deserialize, Serialize};
|
|
use sha2::{Digest, Sha256};
|
|
|
|
/// What a pack carries.
|
|
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
|
pub struct Pack {
|
|
/// Feed version, e.g. "2026.08.21". This is what `hound status` shows.
|
|
pub version: String,
|
|
/// When it was built, RFC3339.
|
|
pub created: String,
|
|
/// Free-text provenance, so a user can see where content came from.
|
|
pub sources: Vec<String>,
|
|
pub indicators: Vec<Indicator>,
|
|
}
|
|
|
|
impl Pack {
|
|
/// Serialise to the exact bytes that get signed.
|
|
///
|
|
/// Deterministic on purpose: the same input must produce the same
|
|
/// bytes on every machine, or the signature is unverifiable and the
|
|
/// hash is not a version. `serde_json` preserves struct field order
|
|
/// and `Vec` order, so the only requirement is that the caller does
|
|
/// not reorder indicators between build and sign.
|
|
pub fn to_bytes(&self) -> Result<Vec<u8>, Error> {
|
|
serde_json::to_vec(self).map_err(|e| Error::Encode(e.to_string()))
|
|
}
|
|
|
|
pub fn sha256(&self) -> Result<String, Error> {
|
|
let bytes = self.to_bytes()?;
|
|
let mut h = Sha256::new();
|
|
h.update(&bytes);
|
|
Ok(format!("{:x}", h.finalize()))
|
|
}
|
|
}
|
|
|
|
/// A pack plus its detached signature.
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct SignedPack {
|
|
/// The pack, as the exact bytes that were signed. Kept as bytes rather
|
|
/// than as a `Pack` so verification happens against what was actually
|
|
/// signed, not against a re-encoding of a parsed value.
|
|
#[serde(with = "base64_bytes")]
|
|
pub payload: Vec<u8>,
|
|
#[serde(with = "base64_bytes")]
|
|
pub signature: Vec<u8>,
|
|
/// Which key signed it, so keys can be rotated without a flag day.
|
|
pub key_id: String,
|
|
}
|
|
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub enum Error {
|
|
Encode(String),
|
|
/// The signature did not verify. The pack is discarded untouched.
|
|
BadSignature,
|
|
/// Signed by a key we do not trust.
|
|
UnknownKey(String),
|
|
Malformed(String),
|
|
}
|
|
|
|
impl std::fmt::Display for Error {
|
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
|
match self {
|
|
Error::Encode(e) => write!(f, "encoding the pack: {e}"),
|
|
Error::BadSignature => write!(
|
|
f,
|
|
"the definitions pack is not signed by Hound and was discarded"
|
|
),
|
|
Error::UnknownKey(id) => write!(f, "pack signed by unknown key {id}"),
|
|
Error::Malformed(e) => write!(f, "malformed pack: {e}"),
|
|
}
|
|
}
|
|
}
|
|
|
|
impl std::error::Error for Error {}
|
|
|
|
/// Sign a pack. Build-side only; the agent never holds a signing key.
|
|
pub fn sign(pack: &Pack, signing_key: &SigningKey, key_id: &str) -> Result<SignedPack, Error> {
|
|
let payload = pack.to_bytes()?;
|
|
let signature = signing_key.sign(&payload);
|
|
Ok(SignedPack {
|
|
payload,
|
|
signature: signature.to_bytes().to_vec(),
|
|
key_id: key_id.to_string(),
|
|
})
|
|
}
|
|
|
|
/// Verify and decode a pack.
|
|
///
|
|
/// The order is the point: the signature is checked against the raw bytes
|
|
/// **before** anything parses them. A pack that fails verification is
|
|
/// never handed to the JSON parser, so a hostile pack cannot reach the
|
|
/// parser's attack surface at all.
|
|
pub fn verify(signed: &SignedPack, trusted: &[(&str, VerifyingKey)]) -> Result<Pack, Error> {
|
|
let Some((_, key)) = trusted.iter().find(|(id, _)| *id == signed.key_id) else {
|
|
return Err(Error::UnknownKey(signed.key_id.clone()));
|
|
};
|
|
|
|
let sig_bytes: [u8; 64] = signed
|
|
.signature
|
|
.as_slice()
|
|
.try_into()
|
|
.map_err(|_| Error::BadSignature)?;
|
|
let signature = Signature::from_bytes(&sig_bytes);
|
|
|
|
key.verify(&signed.payload, &signature)
|
|
.map_err(|_| Error::BadSignature)?;
|
|
|
|
serde_json::from_slice(&signed.payload).map_err(|e| Error::Malformed(e.to_string()))
|
|
}
|
|
|
|
/// Base64 for the byte fields, so a signed pack is a plain JSON file.
|
|
mod base64_bytes {
|
|
use serde::{Deserialize, Deserializer, Serializer};
|
|
|
|
const ALPHABET: &[u8; 64] =
|
|
b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
|
|
|
|
pub fn encode(data: &[u8]) -> String {
|
|
let mut out = String::with_capacity(data.len().div_ceil(3) * 4);
|
|
for chunk in data.chunks(3) {
|
|
let b = [chunk[0], *chunk.get(1).unwrap_or(&0), *chunk.get(2).unwrap_or(&0)];
|
|
let n = ((b[0] as u32) << 16) | ((b[1] as u32) << 8) | b[2] as u32;
|
|
out.push(ALPHABET[(n >> 18) as usize & 63] as char);
|
|
out.push(ALPHABET[(n >> 12) as usize & 63] as char);
|
|
out.push(if chunk.len() > 1 {
|
|
ALPHABET[(n >> 6) as usize & 63] as char
|
|
} else {
|
|
'='
|
|
});
|
|
out.push(if chunk.len() > 2 {
|
|
ALPHABET[n as usize & 63] as char
|
|
} else {
|
|
'='
|
|
});
|
|
}
|
|
out
|
|
}
|
|
|
|
pub fn decode(s: &str) -> Option<Vec<u8>> {
|
|
let mut rev = [255u8; 256];
|
|
for (i, c) in ALPHABET.iter().enumerate() {
|
|
rev[*c as usize] = i as u8;
|
|
}
|
|
let clean: Vec<u8> = s.bytes().filter(|b| *b != b'=' && !b.is_ascii_whitespace()).collect();
|
|
let mut out = Vec::with_capacity(clean.len() * 3 / 4);
|
|
for chunk in clean.chunks(4) {
|
|
let mut n = 0u32;
|
|
for (i, b) in chunk.iter().enumerate() {
|
|
let v = rev[*b as usize];
|
|
if v == 255 {
|
|
return None;
|
|
}
|
|
n |= (v as u32) << (18 - 6 * i);
|
|
}
|
|
out.push((n >> 16) as u8);
|
|
if chunk.len() > 2 {
|
|
out.push((n >> 8) as u8);
|
|
}
|
|
if chunk.len() > 3 {
|
|
out.push(n as u8);
|
|
}
|
|
}
|
|
Some(out)
|
|
}
|
|
|
|
pub fn serialize<S: Serializer>(data: &[u8], s: S) -> Result<S::Ok, S::Error> {
|
|
s.serialize_str(&encode(data))
|
|
}
|
|
|
|
pub fn deserialize<'de, D: Deserializer<'de>>(d: D) -> Result<Vec<u8>, D::Error> {
|
|
let s = String::deserialize(d)?;
|
|
decode(&s).ok_or_else(|| serde::de::Error::custom("invalid base64"))
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use crate::osv::Versions;
|
|
|
|
fn test_key() -> SigningKey {
|
|
// A fixed key so tests are deterministic. Never a real one.
|
|
SigningKey::from_bytes(&[7u8; 32])
|
|
}
|
|
|
|
fn a_pack() -> Pack {
|
|
Pack {
|
|
version: "2026.08.21".into(),
|
|
created: "2026-08-21T12:00:00Z".into(),
|
|
sources: vec!["ossf/malicious-packages".into()],
|
|
indicators: vec![Indicator {
|
|
ecosystem: "cratesio".into(),
|
|
name: "rustdecimal".into(),
|
|
versions: Versions::All,
|
|
id: "MAL-2022-1".into(),
|
|
summary: "malicious crate rustdecimal".into(),
|
|
}],
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn a_signed_pack_round_trips() {
|
|
let key = test_key();
|
|
let signed = sign(&a_pack(), &key, "hound-2026").unwrap();
|
|
let trusted = [("hound-2026", key.verifying_key())];
|
|
let back = verify(&signed, &trusted).unwrap();
|
|
assert_eq!(back, a_pack());
|
|
}
|
|
|
|
// ── the attack this exists to stop ──
|
|
|
|
#[test]
|
|
fn a_tampered_payload_is_rejected() {
|
|
// Someone who can substitute a pack can add an entry for
|
|
// /usr/bin/sudo and have Hound quarantine it everywhere. This is
|
|
// the single most important test in the crate.
|
|
let key = test_key();
|
|
let mut signed = sign(&a_pack(), &key, "hound-2026").unwrap();
|
|
let pos = signed.payload.len() / 2;
|
|
signed.payload[pos] ^= 0x01;
|
|
|
|
let trusted = [("hound-2026", key.verifying_key())];
|
|
assert_eq!(verify(&signed, &trusted), Err(Error::BadSignature));
|
|
}
|
|
|
|
#[test]
|
|
fn a_pack_signed_by_a_different_key_is_rejected() {
|
|
let attacker = SigningKey::from_bytes(&[9u8; 32]);
|
|
let signed = sign(&a_pack(), &attacker, "hound-2026").unwrap();
|
|
let trusted = [("hound-2026", test_key().verifying_key())];
|
|
assert_eq!(verify(&signed, &trusted), Err(Error::BadSignature));
|
|
}
|
|
|
|
#[test]
|
|
fn an_unknown_key_id_is_rejected_by_name() {
|
|
let key = test_key();
|
|
let signed = sign(&a_pack(), &key, "somebody-elses-key").unwrap();
|
|
let trusted = [("hound-2026", key.verifying_key())];
|
|
assert!(matches!(
|
|
verify(&signed, &trusted),
|
|
Err(Error::UnknownKey(id)) if id == "somebody-elses-key"
|
|
));
|
|
}
|
|
|
|
#[test]
|
|
fn a_truncated_signature_is_rejected_rather_than_panicking() {
|
|
let key = test_key();
|
|
let mut signed = sign(&a_pack(), &key, "hound-2026").unwrap();
|
|
signed.signature.truncate(10);
|
|
let trusted = [("hound-2026", key.verifying_key())];
|
|
assert_eq!(verify(&signed, &trusted), Err(Error::BadSignature));
|
|
}
|
|
|
|
#[test]
|
|
fn a_hostile_payload_never_reaches_the_parser() {
|
|
// Verification happens on raw bytes first. Garbage that would
|
|
// upset the JSON parser is discarded before it gets there.
|
|
let key = test_key();
|
|
let signed = SignedPack {
|
|
payload: vec![0xff; 4096],
|
|
signature: vec![0u8; 64],
|
|
key_id: "hound-2026".into(),
|
|
};
|
|
let trusted = [("hound-2026", key.verifying_key())];
|
|
assert_eq!(verify(&signed, &trusted), Err(Error::BadSignature));
|
|
}
|
|
|
|
#[test]
|
|
fn a_validly_signed_but_malformed_pack_is_reported_as_malformed() {
|
|
// Distinct from BadSignature: this one is our own bug, not an
|
|
// attack, and conflating the two would send us hunting the wrong
|
|
// problem.
|
|
let key = test_key();
|
|
let payload = b"{ not json }".to_vec();
|
|
let signature = key.sign(&payload).to_bytes().to_vec();
|
|
let signed = SignedPack { payload, signature, key_id: "hound-2026".into() };
|
|
let trusted = [("hound-2026", key.verifying_key())];
|
|
assert!(matches!(verify(&signed, &trusted), Err(Error::Malformed(_))));
|
|
}
|
|
|
|
// ── determinism ──
|
|
|
|
#[test]
|
|
fn the_same_pack_always_produces_the_same_bytes() {
|
|
// A pack's hash is its identity. If encoding varied between
|
|
// machines, the signature would be unverifiable and the version
|
|
// meaningless.
|
|
assert_eq!(a_pack().to_bytes().unwrap(), a_pack().to_bytes().unwrap());
|
|
assert_eq!(a_pack().sha256().unwrap(), a_pack().sha256().unwrap());
|
|
assert_eq!(a_pack().sha256().unwrap().len(), 64);
|
|
}
|
|
|
|
#[test]
|
|
fn changing_one_indicator_changes_the_hash() {
|
|
let mut other = a_pack();
|
|
other.indicators[0].name = "rust_decimal".into();
|
|
assert_ne!(a_pack().sha256().unwrap(), other.sha256().unwrap());
|
|
}
|
|
|
|
// ── the signed pack is a plain JSON file ──
|
|
|
|
#[test]
|
|
fn a_signed_pack_serialises_to_json_and_back() {
|
|
let key = test_key();
|
|
let signed = sign(&a_pack(), &key, "hound-2026").unwrap();
|
|
let json = serde_json::to_string(&signed).unwrap();
|
|
let back: SignedPack = serde_json::from_str(&json).unwrap();
|
|
assert_eq!(back.payload, signed.payload);
|
|
assert_eq!(back.signature, signed.signature);
|
|
|
|
let trusted = [("hound-2026", key.verifying_key())];
|
|
assert!(verify(&back, &trusted).is_ok());
|
|
}
|
|
|
|
#[test]
|
|
fn base64_round_trips_every_length() {
|
|
for len in 0..200 {
|
|
let data: Vec<u8> = (0..len).map(|i| (i * 7 % 256) as u8).collect();
|
|
let encoded = base64_bytes::encode(&data);
|
|
assert_eq!(base64_bytes::decode(&encoded).as_deref(), Some(&data[..]), "len {len}");
|
|
}
|
|
}
|
|
|
|
#[test]
|
|
fn invalid_base64_is_rejected_rather_than_guessed_at() {
|
|
assert!(base64_bytes::decode("not base64 !!!").is_none());
|
|
}
|
|
|
|
#[test]
|
|
fn errors_read_like_something_a_person_can_act_on() {
|
|
assert!(Error::BadSignature.to_string().contains("discarded"));
|
|
assert!(Error::BadSignature.to_string().contains("Hound"));
|
|
}
|
|
}
|