3be16eeef9
Phase 3's foundation. Three jobs that share a data model:
osv parse the ossf/malicious-packages feed (Apache-2.0, ~226k
records, daily) into indicators
index answer "is this package known bad?" fast enough to ask it
thousands of times per project sweep
pack sign a definitions pack, and verify one before loading it
Validated against the real feed rather than fixtures: the whole
crates.io OSV export, 2,749 records, parsed with zero failures — 3,885
indicators across ten ecosystems, 19 of them MAL-. `rustdecimal` (the
real typosquat of rust_decimal) resolves in crates.io and stays clean
in npm and PyPI, which is the ecosystem isolation working.
Notes on the three:
* A malicious-package record is not a vulnerability record. It almost
always carries introduced:"0" with no fix, meaning EVERY version is
malicious — the package exists only to be malware, so there is no safe
version to upgrade to. Conflating that with a version-bounded
vulnerability either misses real hits or condemns safe versions of
legitimate packages, so the two are modelled separately.
* The index is a cuckoo filter in front of a map. Cuckoo rather than
bloom specifically because a definitions feed needs DELETION: OSV
withdraws records — it once withdrew 157 malware reports after a
false-positive incident — and a filter you cannot remove from means a
withdrawn record costs a probe forever or forces a rebuild. 226k
indicators fit in under 4 MB; the crates.io set is 8 KB.
The property that must never break is no false negatives, and it has
its own test. A false positive costs a hash lookup; a false negative
is malware reported as clean. That is also why a fingerprint hashing
to zero is nudged to one — zero marks an empty slot, so without the
nudge one key in 65,536 would silently vanish.
* Signing is not about entitlement; the subscription gates the server.
It answers "is this pack really from us?", because someone who can
substitute a definitions file can add an entry for /usr/bin/sudo and
have Hound quarantine it on every machine that updates — a supply
chain attack delivered through the security product. Verification
happens on the raw bytes BEFORE anything parses them, so a hostile
pack never reaches the parser at all.
256 tests pass across the workspace.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
70 lines
2.3 KiB
Rust
70 lines
2.3 KiB
Rust
//! Ingest a directory of OSV JSON records and report what came out.
|
|
//!
|
|
//! Used to validate the parser against the real feed rather than against
|
|
//! fixtures. Fixtures test the shape you expected; the feed tests the
|
|
//! shape that ships.
|
|
//!
|
|
//! cargo run -p hound-defs --example ingest-osv -- <dir> [lookup-name]
|
|
|
|
use hound_defs::{osv, Index};
|
|
|
|
fn main() {
|
|
let mut args = std::env::args().skip(1);
|
|
let Some(dir) = args.next() else {
|
|
eprintln!("usage: ingest-osv <dir-of-osv-json> [name-to-look-up]");
|
|
std::process::exit(2);
|
|
};
|
|
let probe = args.next();
|
|
|
|
let mut files = 0usize;
|
|
let mut unparsed = 0usize;
|
|
let mut indicators = Vec::new();
|
|
let mut malicious = 0usize;
|
|
|
|
for entry in std::fs::read_dir(&dir).expect("readable directory").flatten() {
|
|
let path = entry.path();
|
|
if path.extension().is_none_or(|e| e != "json") {
|
|
continue;
|
|
}
|
|
files += 1;
|
|
let Ok(text) = std::fs::read_to_string(&path) else { continue };
|
|
let parsed = osv::parse_record(&text);
|
|
if parsed.is_empty() {
|
|
unparsed += 1;
|
|
continue;
|
|
}
|
|
if parsed[0].id.starts_with("MAL-") {
|
|
malicious += parsed.len();
|
|
}
|
|
indicators.extend(parsed);
|
|
}
|
|
|
|
let mut ecosystems: std::collections::BTreeMap<String, usize> = Default::default();
|
|
let mut all_versions = 0usize;
|
|
for i in &indicators {
|
|
*ecosystems.entry(i.ecosystem.clone()).or_default() += 1;
|
|
if i.versions == osv::Versions::All {
|
|
all_versions += 1;
|
|
}
|
|
}
|
|
|
|
println!("files read {files}");
|
|
println!("yielded nothing {unparsed}");
|
|
println!("indicators {}", indicators.len());
|
|
println!(" of those MAL- {malicious}");
|
|
println!(" all-versions {all_versions}");
|
|
println!("ecosystems {ecosystems:?}");
|
|
|
|
let index = Index::build(indicators);
|
|
println!("index keys {}", index.len());
|
|
println!("filter {:.1} KB", index.filter_bytes() as f64 / 1024.0);
|
|
|
|
if let Some(name) = probe {
|
|
for eco in ["cratesio", "npm", "pypi"] {
|
|
match index.any_version(eco, &name) {
|
|
Some(hit) => println!("\nLOOKUP {eco}:{name} -> {} — {}", hit.id, hit.summary),
|
|
None => println!("\nLOOKUP {eco}:{name} -> clean"),
|
|
}
|
|
}
|
|
}
|
|
}
|