Independent field evaluation · Cactus Compute
Needle 3.0.3: a tiny model is not a tiny general-purpose agent
Needle became much more useful when we stopped asking it to be a broad router and instead put it behind deterministic gates, gave it one to three closely related tools, and validated the result afterwards.
Practical conclusion: use Needle 3.0.3 as a narrow local specialist for routing/pass-through and grounded argument extraction. Do not make it the owner of broad tool routing, archive retrieval or unconstrained extraction.
About “Needle Free”: this report covers the freely available/open-source cactus-needle Python package. “Needle Free” is descriptive here, not an official product-tier name.
Routing: broad was weak, narrow was useful
| Experiment | Observed result | Our disposition |
|---|---|---|
| 8 natural narrow tools, base Needle | 50% routing | Reject as broad router |
| Generic / vague schemas | 10% routing | Reject |
| Deterministic prefilter → 3 candidate tools | 75% routing; 100% abstention | Promising hybrid |
| Hand-written deterministic rules on known patterns | 100% | Deterministic code wins |
| Two separate 3-tool specialist suites with deterministic triggers | 8/8 in each suite | Useful narrow specialist |
The lesson was architectural rather than just numerical. Needle got better when the surrounding code first decided which small neighbourhood of tools was plausible. When the intent was already a known pattern, ordinary code was both more accurate and easier to reason about.
Grounded extraction: promising, but compare against code
| Extraction regime | Needle 3.0.3 | Comparator / note |
|---|---|---|
| Four simple grounded cases | 91.7% field accuracy | Promising |
| 72 noisy grounded cases | 87.5% | Tailored deterministic parser: 100% |
| Open-ended paraphrase extraction | 22.4% | Minimal deterministic baseline: 44.9% |
| Narrow specialist grounded slice | 100% | Strong specialist signal |
Schema design mattered. In one structured operational extraction task, bare schemas produced 75% field accuracy, field descriptions lifted that to 86.1%, and supplying the live allowed values as an enum grammar lifted it to 97.2%. A stable tailored parser still reached 100%, so maintenance cost matters as much as raw model accuracy.
Retrieval was not strong enough to own search
On an initial retrieval set, Needle reached recall@1/3/5 of .40 / .70 / .80. On a deliberately confusable corpus it fell to .20 / .40 / .60; a simple lexical baseline scored .30 / .40 / .40. A many-tool document-retrieval test was worse: 0% with opaque identifiers and 10% with semantic identifiers.
That was not enough evidence for us to replace ordinary search/retrieval machinery with Needle 3.0.3.
The ladder is genuinely useful
Needle 3's layer ladder let us trade capability for resource use. These figures are measurements on our test host, not universal device benchmarks.
| Depth | Peak RAM | Decode throughput | Observed role |
|---|---|---|---|
| 4 layers | ~53 MB | ~1,252 tok/s on one specialist; ~1,449 tok/s on another | Narrow routing / pass-through |
| 8 layers | ~81 MB | ~736 tok/s | More headroom, still compact |
| 20 layers (full) | ~99 MB | ~337 tok/s | Use selectively for harder grounded extraction |
The 4-layer rung achieved 100% expected-call coverage on the small trigger suites, but only 35% overall on a multi-field extraction stress test. Cheap depth should therefore be treated as a specialist route, not a general extractor.
Abstention is valuable
The deterministic-prefilter + three-tool configuration achieved 100% abstention on the negative cases we supplied. For embedded or local automation, the ability to return no call is often more valuable than squeezing out another speculative match.
The architecture we would use
- Deterministic/domain gate handles known intent and obvious negatives.
- Expose only a narrow specialist environment, preferably one to three tools.
- Use Needle for grounded span extraction and argument assembly, not general reasoning.
- Validate identifiers, ranges, state and policy deterministically before execution.
- Escalate only unresolved semantic residue to a stronger semantic/reasoning route.
- When a repeated pattern becomes mechanically known, compile it back into ordinary code.
Version boundary — especially important here
We tested cactus-needle 3.0.3, released 20 September 2026. PyPI already lists 3.0.4 as a newer release. Our installed 3.0.3 build did not expose the Python audio API we were looking for during the test. Current upstream documentation now describes audio arguments, so that limitation should not be projected onto later versions.
Sources and reproducibility notes
- PyPI: cactus-needle 3.0.3
- Cactus Compute Needle repository
- Upstream agent-oriented Needle reference
- Machine-readable version of our measurements
cactus-needle 3.0.3, tested 2026-09-20. The upstream project moved quickly after this test; check the exact package version before applying any limitation reported here.Published 23 September 2026 · The Amateur Limited · All research reports · Home