1.3M requests, and the most serious defect had no error code at all.

Basquin was pointed at real, unmodified JVM web apps — no test code, no instrumentation in the app, no config changes. It found errors, which any load tool would have found too. It also found 3,887 responses that were neither an error nor the thing that was asked for, and — on a target that was still answering 200 in 10 ms — two threads that had been spinning at 100% of a core for five hours.

1.3M

requests driven across three unmodified apps

49k

server errors — the part a crash oracle does report

3,887

3xx responses classified by destination instead of followed

threads spun at 100% of a core for 5 hours, while the app answered 200 in 10 ms

The gap this measures

A crash-only oracle — a plain fuzzer, a load test's error count — asks one question: did it return an error? That question is not useless: it catches the 49k server errors below, and those are real. But it is the only question those tools ask, and the two most serious things on this page do not answer it. A write the app silently threw away returns 302. A thread spinning forever at 100% of a core returns 200 in 10 ms.

Each row splices **two separate campaigns**: the coverage and breach columns come from that app's *explore* run, the traffic columns from a *different*, later *load* run at the highest concurrency tested. They are not two views of one run, and a number in one group says nothing about the other. The tiles above aggregate **every** collected load run (6 of them), so their totals exceed this table's.
	from the explore campaign			from the load campaign — no invariants evaluated
App	Coverage	Invariant breaches	Failed requests	Requests	req/s	p50 / p99 ms	5xx
Apache JSPWiki 2.12.4 wiki CMS · filesystem store	27.1%	2918	17	17k	55.2	2 / 6780	398
MyBatis JPetStore-6 e-commerce · MyBatis + HSQLDB	23.1%	421	470	1.1M	3511.8	1 / 9	39047
Apache Roller 6.1.5 blog CMS · JPA + Postgres	30.5%	1402	0	4,670	15.5	515 / 532	0

These counts are now complete — and it took a fix to make them so

The valve evaluates every invariant inside the app JVM and logs it, then attaches the result to the response — but only if the response has not already been committed, and on a real app most responses have. The driver learned about violations solely by reading that header, so a finding on a committed response was evaluated, logged, and thrown away. DD-040 added a side channel that recovers them and DD-039 carries it across redirects; the numbers on this page are from that fixed channel.

How much was being lost, measured before the fix: Roller reported 0 of the violations it evaluated (one explore window evaluated 1,906 and reported none). On the pre-fix run those two apps reported almost nothing — Roller 0, JSPWiki 1 (summaries preserved at git 5655c46, pruned from the tree by this run) — while JPetStore lost nothing because its small pages rarely commit early, which is why it looked productive and the other two looked clean. On the fixed channel the same explore runs report Roller 1,402, JSPWiki 2,918, and JPetStore 421: Roller, which "looked underwhelming", is the most productive target and has the highest coverage here.

The loss was biased the wrong way — the larger and slower the response, the more likely it committed early, so the most expensive requests were the most likely to be discarded. Evaluation was always intact; only the reporting was lost, and that is what the fix restored. The loss magnitudes are the ones DD-040 measured (see its decision record); the recovered violations are archived under bench-results/violation-logs-2026-07-23/.

Those two columns are split deliberately, because they are not the same kind of evidence and the campaign resource's single findings total blends them (along with inputs saved purely for reaching new code, which are not defects at all). Invariant breaches are the strong signal: the valve, running inside the app JVM, measured a request that exceeded its latency, heap, or thread budget. Failed requests are weaker: the driver's HTTP call threw — a connection reset, a read timeout, or a protocol error. That is an availability symptom, but this run does not attribute it between the app, the driver, and the single-node cluster, so it is reported as what was observed rather than as a defect count. See Methodology.

Silent rejection: the finding a load tool cannot have

JSPWiki's editor is guarded by two per-session tokens — an anti-CSRF field, and an anti-spam hash whose field name is six random letters that rotate. Basquin correlates both out of the edit form and replays the write, so the saves are well-formed: the run's captureMisses counter — reported in the figure below — says how often a required prior capture failed to bind, and the rest were submitted with both tokens.

What comes back is a 302 either way. A save that lands and a save the app throws away are the same status code, the same absence of any error, and the same entry in a load tool's success column. The only thing that separates them is the Location header — which is precisely what a client destroys when it follows the redirect for you. So Basquin stops following, and reads it.

Where 3,887 redirects went — JSPWiki under load, classified by Location

Every one of these was a 3xx: not a 5xx, not a 4xx, and every CSRF and anti-spam token was submitted (captureMisses: 340 — steps skipped because a prior capture never bound, counted separately from the requests below), so a load tool counts all 3,887 as served. Split by destination they resolve into 387 refused writes and 2,820 accepted ones — a distinction carried entirely by a response header that a redirect-following client discards before anyone can read it.

The first version of this chart was wrong in both directions

Worth stating, because it is the reason to trust the second one. The first run of this classifier put Login.jsp at the top and it was read as rejected writes. Probing the running app showed they are not writes at all — they are view requests carrying an empty page name, which JSPWiki bounces to the login screen whether or not a valid session is presented. Meanwhile a bucket named for a page turned out to be successful saves that the fold had missed, because JSPWiki capitalizes the first letter of an unresolved page name (ndws → Ndws) and the fold compared bytes.

The serious one was the opposite mistake. A genuine concurrent-edit conflict redirects to PageModified.jsp?page=<the page being saved>, and because that page name matches the request's own, the rule filed a real rejection into the success bucket — the exact invisibility the feature exists to eliminate, reintroduced by its own classifier. Same-page redirects now key by the destination's path, so every success shares one bounded key while a same-page rejection route keeps its own. Every label in the chart above was verified against the running app rather than inferred from the counts.

What the load actually cost

Latency percentiles from the driver, which measures wall-clock per request against the real app over the whole run — warm-up excluded, no sampling.

Apache JSPWiki 2.12.4 — 17k requests at 55.2 rps (concurrency 8)

Median 2 ms, but a worst single request of 9,568 ms — a 4784× spread that a median-only dashboard hides entirely.

MyBatis JPetStore-6 — 1.1M requests at 3511.8 rps (concurrency 8)

Median 1 ms, but a worst single request of 35 ms — a 35× spread that a median-only dashboard hides entirely.

Apache Roller 6.1.5 — 4,670 requests at 15.5 rps (concurrency 8)

Median 515 ms, but a worst single request of 558 ms — a 1× spread that a median-only dashboard hides entirely.

How much of the app gets reached

Findings only count if the explorer gets somewhere interesting. Coverage is measured by JaCoCo against the app's own classes, driven by a request grammar rather than recorded traffic — so it reaches authenticated write paths, not just the pages a crawler can see.

Bytecode coverage reached by grammar-guided exploration

JaCoCo-measured coverage of the app's own classes, reached without a single line of test code — just a request grammar. Invariant breaches the driver actually RECEIVED in the same runs: JSPWiki reported 2918 · JPetStore-6 reported 421 · Roller reported 1402. These come from the fixed reporting channel, so they are the violations actually evaluated, not a lower bound.

Heap allocated per request

Every number above is visible from outside the app. This one is not, and it is the reason the valve runs in-process.

Heap allocated per request, measured inside the app JVM

This is the measurement no external load tool can take: the valve reads the JVM's own allocation counter on either side of the request, so the number is the app's real cost for that one input — not a process-wide average.

The cold cliff

8408 ms & 10.7 MB on request #1 → ~33 ms steady

The very first request of the soak (JSPWiki front page) took 8408 ms and allocated 10.7 MB — then the app settled to ~33 ms. The first-request-after-deploy cliff (JSP compilation plus cold-cache markup rendering) is an availability pathology a warmed-up load test never sees and a crash test never flags — and it is exactly what a user hits after every deploy, restart, and scale-up event.

Source: bench-results/jspwiki/findings-summary.txt (iteration 1: 8408 ms + 10713 KB) and the k6 steady-state median (33 ms).

Turning findings into changes

A finding is only useful if it names something a team can go fix. Each class of finding maps to a specific kind of work:

Per-request heap over budget

Names the exact input that allocated. Usually a render path building the whole response in memory, or an unbounded result set. The fix is streaming or a bound — and the same run re-measures it.

signal: heapDelta > budget on a specific route

Latency spread, not latency

A p50 that looks fine next to a p99 orders of magnitude worse points at a cliff — a cold cache, a lock, an N+1 — not at general slowness. Capacity planning off the median under-provisions for exactly this.

signal: p99/p50 ratio, plus the worst single input

Silent rejection

Writes that return a success-shaped status and don't land. Points at session handling, CSRF/anti-automation defences, or concurrency conflicts — and invalidates any load number measured on that path, because the expensive half never ran.

signal: 3xx classified by Location, not followed

Retention across a run

Separating "this request is expensive" from "this run is leaking" is the question that matters at 3am. Measuring it needs GC-bracketed sampling, not a raw used-heap delta — see the methodology note below for why this page publishes no drift number.

signal: retained heap after a forced collection, sampled repeatedly

Methodology & honesty

Real runs only. Every figure on this page is generated by deploy/bench/render_page.py from artifacts under bench-results/ — the collected campaign objects and the driver's own terminal summary. The charts are emitted from those numbers, so a figure cannot disagree with the run it cites. Nothing here is projected or hand-tuned.
Apps unmodified. Apache JSPWiki 2.12.4, MyBatis JPetStore-6 and Apache Roller 6.1.5. Each runs on Tomcat 9 with the namespace-free Basquin valve + agent. No code, config, or bytecode changes to the apps.
Budgets are per app, and apply only to the explore runs. JSPWiki latency > 25ms / heapDelta > 512KB; JPetStore-6 latency > 50ms / heapDelta > 512KB; Roller latency > 250ms / heapDelta > 512KB. All soft mode: findings are recorded and the run continues. Load mode evaluates no invariants at all — it is lock-free passthrough by design (DD-029) and its driver is never even given a threshold — so a load run's latency percentiles are measured against no budget, and its violations counts are structurally zero rather than checked. Roller's load p50 of 503 ms is not a 250 ms budget that passed; it is a budget that was never applied.
Load runs are serialized. The bench cluster is a single kind node, so two concurrent drivers would measure each other. deploy/bench/battery.sh blocks on each campaign reaching a terminal phase before starting the next.
Two vantage points (why committed files differ on "max"). The per-request heap and latency findings (10.7 MB, 8.4 s) are measured server-side, inside the app JVM by the valve — the app's true cost for that input. The driver's latency percentiles are the client-side view over the network. They answer different questions and are not interchangeable.
Per-request heap comes from an earlier soak. The heap-per-request figure is from the valve-instrumented run recorded in bench-results/report-data.json (2026-07-21); the campaign-driven runs above measure heap drift across the run, not per request. Both are stated as what they are.
Several reported zeros do not mean "checked and clean". A load run's violations.latency is structurally 0 because load mode is never given the latency budget, and the explore summary's invariants block is the driver measuring itself with no thresholds configured. Neither is a check that passed, so those particular zeros are "not evaluated" rather than "evaluated and clean" — a distinction the summary now marks explicitly (notEvaluated).
Two observations on this run, reported not explained. JSPWiki holds roughly flat throughput (48.8 rps at concurrency 1 vs 55.2 at 8) while its p99 latency blows out from 61 ms to 6780 ms — added concurrency buys queueing, not throughput, the signature of a resource already saturated at c1. And JPetStore returns 5xx on 3.7% of requests under load (48k of 1.3M). Neither has been root-caused; load mode was confirmed on every run (no driftUnavailable), so neither is an artifact of the target sitting in the wrong mode.
No heap-drift number is published, deliberately. The driver's heapDriftKb is a raw totalMemory − freeMemory difference with no collection on either side, so it measures GC phase as much as retention: the same metric came back at +381 MB on one run of this target and −194 MB on another. Bracketing with a forced collection instead showed no unbounded growth on this app, and that repeated unique-text saves retained nothing measurable on-heap (JSPWiki writes versions to disk) — an observation from the drift investigation, not a figure with a retained artifact. GC-bracketed sampling is a tool follow-up; until then this page reports no drift figure rather than a confident one.
Failed requests are not yet attributed. That column counts iterations where the driver's HTTP call threw rather than returning a status. On JPetStore it is a large number and has not been root-caused to the app versus the driver versus the single-node cluster, so no claim is made about it beyond the count. The invariant-breach column is the one measured inside the app, and it is the one to read.
The k6 comparison is indicative, not a controlled head-to-head. The tools measure different things, and the valve deliberately serializes requests (DD-005) so per-request deltas stay accurate under concurrency. The point was never speed — it is that a load tool reports zero defects on requests Basquin flags thousands of times.

Want to run this same battery against your own app? Read Benchmarking & target onboarding → — packaging an unmodified WAR, authoring a request grammar, and running the load comparison.