Detection benchmark

A malware detector is only worth what it catches, and what it wrongly flags. Both are measured, and the numbers are published here, misses included.

Real-world results

Measured on 95 real malicious samples in supported languages (malicious PyPI and npm packages from the DataDog malicious-software-packages dataset and PHP webshells from tennc/webshell), plus a benign control of 10 popular, trusted PyPI packages:

Configuration	Gate	Recall	Precision	False positives
scan (Semgrep + heuristics)	MEDIUM	73%	94%	4
scan (Semgrep + heuristics)	HIGH	45%	100%	0
malcontent alone	MEDIUM	77%	—	3
scan --with-malcontent (hybrid)	MEDIUM	91%	93%	6
scan --with-malcontent (hybrid)	HIGH	51%	100%	0

How to read this:

• The hybrid catches more than either engine alone. At 91% it beats Semgrep (73%) and malcontent (77%) on their own, which is the reason malware-detector combines engines.
• --fail-on HIGH is a very low-noise gate. It had zero false positives on trusted packages, which makes it a safe default for blocking CI.
• The default MEDIUM gate favours catching more. It reports more, and some of those findings need human review.

The remaining misses are mostly heavily obfuscated PHP webshells and npm dependency-confusion packages that only use child_process. These are the current focus of rule work.

Regression gate in CI

Every push also runs a synthetic benchmark of safe, illustrative samples, with a malicious set and a benign set. CI fails if any malicious sample is missed or any benign one is flagged at the MEDIUM gate, so detections that work today keep working.

Reproduce it yourself

git clone https://github.com/benzid-wael/malware-detector && cd malware-detector
make install

# Synthetic benchmark (what CI runs)
uv run python benchmark/run.py --refresh

# Real-world corpus: downloads live malware into a gitignored corpus/ directory.
# Do this in a disposable VM or container.
uv run python benchmark/fetch_corpus.py --pypi 40 --npm 40 --webshells 30
uv run python benchmark/run.py corpus --refresh --format html > benchmark.html

Live malware

fetch_corpus.py downloads real malicious packages. Don't run anything from corpus/, and only work with it in an isolated environment.

Full methodology: benchmark/README.md.