Improving Cisco Skill Scanner Across Real-World Agent-Safety Datasets
Summary
We evaluated a proposed modernization of Cisco AI Defense Skill Scanner across labeled packages, risky skills, benign hard negatives, prompt-injection corpora, and popular bundled skills. In this latest data-driven update to the Skill Scanner, we overhauled the deterministic core to better correlate signals across dataflow, YARA, and AST analyzers. We shifted syntax-aware Python and JavaScript detection to follow active aliases and direct dynamic imports, explicitly connecting behaviors like download-to-execution and cross-file staging. The result is a highly selective, auditable detection engine that more than doubles our development F1 score to 47.73% and slashes false positives to 1.05%.
Results
On the 6,594-package MaliciousSkillBench development population, core-only package-block F1 rose from 18.69% on origin/main to 47.73%, while blocking false-positive rate fell from 4.56% to 1.05%. Within the pull request, further detector tuning raised F1 from an initial 32.92% snapshot to 47.73%. A separate source-disjoint evaluation was harder: F1 improved from 7.40% to 13.74%, but blocking FPR increased from 3.67% to 7.71%. The development improvement is material, but it did not fully generalize.
Figure 1. F1 and blocking false positive rate across completed scanner snapshots. The initial PR snapshot was available only for the development population.
Datasets and protocol
No single corpus answers every detection question, so each dataset had a defined role:
| Dataset | What it tests | What it does not establish |
|---|---|---|
| MaliciousSkillBench | Labeled malicious-versus-benign package classification, plus source-disjoint evaluation | Complete coverage of unseen ecosystems |
| HarmfulSkillBench | Harmful-content diagnostic coverage | Precision or FPR; it has no benign controls |
| OpenSkillRisk | Risky-package recall and category coverage | Precision or FPR; it has no benign controls |
| NotInject | Benign prompt-injection hard negatives | Package-level benign FPR |
| InjecAgent and In-Page Prompt Injection in the Wild | Injection-signal recall and regression coverage | Whole-package classification quality |
| DataDog's malicious software packages | Quarantined package-recall cases | A balanced classification benchmark |
We pinned upstream revisions and never executed samples. The MaliciousSkillBench development population was used for iteration; three quarantined records were excluded from its declared 6,597 records to form the 6,594-package usable population. Its source-disjoint population was evaluated separately and is now frozen from further tuning. For the positive-only HarmfulSkillBench and OpenSkillRisk diagnostics, unavailable, quarantined, and analyzer-failure samples remained in their denominators.
Results by dataset
The measurements below use different units and should not be combined into one aggregate score.
| Dataset / population | Evaluated population | Valid headline metric | Latest result |
|---|---|---|---|
| MaliciousSkillBench development | 6,594 packages | HIGH+ package classification | Precision 99.16%; recall 31.43%; F1 47.73%; FPR 1.05% |
| MaliciousSkillBench source-disjoint | 1,384 packages | HIGH+ package classification | Precision 60.75%; recall 7.75%; F1 13.74%; FPR 7.71% |
| HarmfulSkillBench | 200 positive-risk packages | Severity-threshold coverage | 7 MEDIUM+ (3.50%); 6 HIGH+ (3.00%) |
| OpenSkillRisk | 263 positive-risk packages | Severity-threshold coverage | 94 MEDIUM+ (35.74%); 76 HIGH+ (28.90%); 22 CRITICAL (8.37%) |
| NotInject | 339 benign text cases | Target injection flag rate | 0/339 at MEDIUM+ (0.00%; 95% CI 0.00–1.12%) |
| InjecAgent | 1,054 canonical signals | Target signal recall | 1,054/1,054 (100.00%) |
| In-Page Prompt Injection | 1,101 canonical groups | Target signal recall | 1,094/1,101 (99.36%) |
| DataDog case-mining subset | 5 selected positives | Diagnostic coverage | Signal on 5/5; HIGH+ on 4/5 |
| Committed golden corpus | 24 curated packages | Exact regression contract | 24/24 package verdicts; 23/23 finding identities |
| Official bundled-skill snapshot | 111 installed skills | Compatibility flag rate | 30 MEDIUM+ (27.03%); 8 HIGH+ (7.21%) |
Only MaliciousSkillBench supplies both malicious and benign package labels for a complete TP/FP/TN/FN classification scorecard. The other rows report the narrower metric their labels support.
MaliciousSkillBench: labeled package classification
These deterministic, core-only results disable ATR and LLM analyzers. Precision, recall, F1, and blocking FPR classify packages at a HIGH/CRITICAL threshold. The separately reported MEDIUM+ actionable flag rate is a broader operational-sensitivity measure, not the blocking FPR.
The table reports completed benchmark snapshots. The latest rows are exact for pushed revision 52796e1a, before the final detector-only working-tree fixes, and should be refreshed against the merge commit before publication.
| Population | Scanner snapshot | TP / FP / TN / FN | Precision | Recall | F1 | Blocking FPR |
|---|---|---|---|---|---|---|
| Development | origin/main (a49c8d9f) |
548 / 61 / 1,277 / 4,708 | 89.98% | 10.43% | 18.69% | 4.56% |
| Development | Initial PR, before tuning (4e96c879) |
1,045 / 48 / 1,290 / 4,211 | 95.61% | 19.88% | 32.92% | 3.59% |
| Development | Latest completed (52796e1a) |
1,652 / 14 / 1,324 / 3,604 | 99.16% | 31.43% | 47.73% | 1.05% |
| Source-disjoint | origin/main (a49c8d9f) |
33 / 20 / 525 / 806 | 62.26% | 3.93% | 7.40% | 3.67% |
| Source-disjoint | Latest completed (52796e1a) |
65 / 42 / 503 / 774 | 60.75% | 7.75% | 13.74% | 7.71% |
| Latest completed population | Accuracy | Macro-F1 | Signal recall | MEDIUM+ benign flag rate |
|---|---|---|---|---|
| Development | 45.13% | 45.00% | 33.37% | 2.17% |
| Source-disjoint | 41.04% | 34.48% | 8.46% | 8.44% |
Against true origin/main, development F1 increased 29.04 percentage points, recall increased 21.00 points, and blocking FPR fell 3.51 points. The initial-PR row shows the effect of the later tuning alone: another 14.81-point F1 gain and a 2.54-point FPR reduction. On source-disjoint data, recall roughly doubled and F1 gained 6.34 points, but precision declined slightly and FPR increased 4.04 points.
For the latest snapshots, 95% Wilson intervals were 98.59–99.50% for development precision, 30.19–32.70% for development recall, and 0.62–1.75% for development blocking FPR. Source-disjoint intervals were 51.28–69.47%, 6.12–9.75%, and 5.75–10.25%, respectively.
HarmfulSkillBench and OpenSkillRisk: positive-risk coverage
These datasets have no benign control population. Their rates measure how often the deterministic scanner produced findings at each threshold; they do not measure precision, F1, true negatives, or FPR.
| Dataset | Metric | Count | Rate | 95% Wilson interval |
|---|---|---|---|---|
| HarmfulSkillBench | Any finding | 192 / 200 | 96.00% | 92.31–97.96% |
| HarmfulSkillBench | MEDIUM+ package | 7 / 200 | 3.50% | 1.71–7.05% |
| HarmfulSkillBench | HIGH+ package | 6 / 200 | 3.00% | 1.38–6.39% |
| OpenSkillRisk | MEDIUM+ package | 94 / 263 | 35.74% | 30.19–41.70% |
| OpenSkillRisk | HIGH+ package | 76 / 263 | 28.90% | 23.75–34.65% |
| OpenSkillRisk | CRITICAL package | 22 / 263 | 8.37% | 5.59–12.34% |
HarmfulSkillBench scanned 199/200 packages; one unavailable sample remained in the denominator. OpenSkillRisk scanned 261/263, with two quarantines and five partial analyzer-failure samples retained in the denominator. OpenSkillRisk remains supplemental while its complete artifact-manifest hash is pending.
The HarmfulSkillBench any-finding rate includes LOW and INFO findings and must not be read as harmful-behavior recall.
The separate DataDog case-mining run found a signal in 5/5 selected packages from its 204-package corpus and a HIGH/CRITICAL finding in 4/5. With only five positives, no benign controls, two scanner-owned historical fixtures, and an older scanner snapshot, that is diagnostic evidence—not a recall, precision, F1, or FPR estimate.
Figure 2. Observed package finding rates on the two positive only risk datasets. The values are diagnostic rates, not classification recall estimates.
NotInject: benign hard negatives
NotInject produced 0/339 target indirect-injection matches at the MEDIUM+ diagnostic threshold: a 0.00% observed target-flag rate with a 95% Wilson interval of 0.00–1.12%. There were no scan or analyzer failures. Non-target wrapper findings were excluded from this purpose-specific metric.
NotInject is valuable prompt-text hard-negative evidence, but it is not independently labeled benign package gold and therefore does not establish package-level FPR.
InjecAgent and In-Page Prompt Injection: signal recall
| Dataset | Canonical signals recovered | Signal recall | 95% Wilson interval |
|---|---|---|---|
| InjecAgent | 1,054 / 1,054 | 100.00% | 99.64–100.00% |
| In-Page Prompt Injection | 1,094 / 1,101 | 99.36% | 98.69–99.69% |
InjecAgent produced no target injection signal on its 1,054 paired controls. The In-Page corpus is positive-only and has no paired controls. Its rule work recovered 30 additional canonical groups over the earlier 1,064/1,101 snapshot, raising signal recall from 96.64% to 99.36%. InjecAgent is a differential signal-recall diagnostic; In-Page is a positive-only deduplicated signal-recall diagnostic. Neither measures package-block classification. Their artifacts pin the source datasets but do not identify the final scanner Git revision, so they remain exploratory PR-era evidence.
Golden corpus and bundled-skill compatibility
The committed golden corpus matched 24/24 package verdicts and 23/23 exact expected finding identities across five runs. This is a deterministic regression contract, not an estimate of real-world accuracy; its saved output artifacts also do not embed the final scanner source revision.
The last lock-valid compatibility run scanned 111/111 installed Codex, Claude Code, and Cursor skills without loader or scanner errors:
| Source | Packages | MEDIUM+ flag rate | HIGH+ flag rate |
|---|---|---|---|
| Codex | 54 | 27 / 54 (50.00%) | 6 / 54 (11.11%) |
| Claude Code | 33 | 2 / 33 (6.06%) | 2 / 33 (6.06%) |
| Cursor | 24 | 1 / 24 (4.17%) | 0 / 24 (0.00%) |
| Total | 111 | 30 / 111 (27.03%) | 8 / 111 (7.21%) |
These are review-queue flag rates, not proof that a bundled skill is malicious and not an authoritative FPR. This last lock-valid run was orchestrated after c5cc6a64, but its report does not embed a scanner Git revision. The installed corpus later changed and could not be reproduced exactly, so it is compatibility evidence rather than final-head benchmark evidence.
What drove the measured change
The proposal retains the Python, signature, YARA/YARA-X, AST, dataflow, archive, document, dependency, and binary analyzers. The measured lift came from improving those deterministic layers:
- Syntax-aware Python and JavaScript execution detection recovered active aliases and direct dynamic imports without treating every code mention as execution.
- Dataflow and correlation added download-to-execution, sensitive-source-to-network, obfuscation-to-execution, cross-file staging, and hidden-executable context.
- Documentation and example awareness, manifest-capability checks, tighter evidence validation, and benign near-miss fixtures reduced noisy candidates.
- Locked revisions, source-aware splits, exact finding identities, fixed denominators, and five-run determinism checks made benchmark changes auditable instead of self-referential.
LLM and Meta analyzers also received stricter evidence validation, but they were disabled for the package-classification results above. This keeps the published comparison reproducible and attributes the measured change to the deterministic core.
Existing scan commands and rule-pack selection remain compatible: core remains the default, optional packs remain selectable, and custom YARA remains supported. The compatibility change is a CPython 3.11–3.14 requirement and a bundled native cel-go helper.
CEL rollout caveat
CEL (Common Expressive Language, https://cel.dev/) did not produce the reported lift: analyzer, signature, YARA, dataflow, and deterministic-correlation changes did. It cannot create findings or change their identity, category, or severity.
On the latest pushed MaliciousSkillBench snapshot, two valid completed development shadow runs each evaluated 150 candidates, proposed no suppressions, and failed open on four candidates whose manifest projection was incomplete; a third report was invalidated by producer-build drift. The source-disjoint run evaluated 25 candidates with no suppression or fallback. An earlier five-run development snapshot had evaluated 154 candidates without fallback, but the later projection failures show that result was not yet stable across revisions.
All eight bundled CEL rules therefore remain in shadow. The latest development run does not satisfy the zero-fallback promotion gate, and neither labeled MaliciousSkillBench population showed false-positive reduction from CEL. We will promote rules individually only after labeled evidence shows safe suppression without lost HIGH/CRITICAL malicious recall. Until then, the built-in CEL gates supply decision telemetry for evaluation but do not suppress user-visible findings.
Reproduce and contribute
The pull request contains the implementation, evaluation methodology, dataset lock, and benchmark details. Reproducible malicious skills, benign near misses, and new structural families are welcome contributions.