Benchmarks

Headline

Measured against upstream EasyOCR over a mixed corpus (documents, tables, rotated scans, scene text, receipts) spanning five recognizer groups, on CPU, both engines at their native multi-threaded default.

The table below is generated from the committed benchmarks/published/latest.json by task python:publish, and CI fails if the two drift apart — it is measured output, not prose.

Engine	Throughput (img/s)	Peak RSS (MB)	Mean CER	Mean token-F1
EasyOCR (warm/batch)	0.12	4,963.9	0.569	0.337
sceptre (warm/batch)	0.23 (~1.9×)	4,653.7 (~1.4× lower, per-image median)	0.569	0.358
sceptre (cold CLI run)	0.21 (~1.8×)	4,653.7	0.569	0.358

Measured with sceptre 0.6.0 (ort/cpu, ONNX Runtime 1.28.0) against EasyOCR 1.7.2 (torch 2.13.0+cu130) on runner-medium (Linux/x86_64), over 40 of 43 corpus entries. Regenerate with task python:benchmark then task python:publish.

“Warm/batch” is a fresh subprocess per language group that loads its model/Reader once and processes every image in that group — the fairest like-for-like comparison. “Cold CLI run” is sceptre’s per-invocation cost: a fresh process per image, paying model load every time, which still beats EasyOCR’s warm, already-loaded reader.

Quality (CER, WER, token-F1) is averaged over the labeled subset of the corpus only, scored against ground truth. The accuracy columns are the parity check: sceptre reads the corpus the same way EasyOCR does. Peak RSS is dominated by the CRAFT detector both engines share, so its ratio narrows toward parity as page size grows — throughput is the more stable difference between the two.

Methodology

Both engines are measured identically: each runs as a fresh subprocess per language group under /usr/bin/time, so peak RSS is a like-for-like whole-process figure. EasyOCR’s figure legitimately includes the Python + torch runtime — that is its real cost, not an artifact of measurement. Timings use --repeats 2 (median wall time, max peak RSS observed).

Three images in the corpus (alpha, test, Hadley_Crater) are capability gaps — formats sceptre cannot decode that EasyOCR can — reported separately and excluded from the speed/quality aggregates so a fast-failing decode never inflates sceptre’s numbers. See Image formats.

Full methodology, including the RSS-measurement rationale, threading choice, and the quality signal, is recorded in ADR 0021.

Reproducing the numbers

task python:benchmark

re-measures and writes benchmark-results/comparison.{json,md}, which is gitignored and machine-local. Useful flags:

Flag	Purpose
--group labeled --limit 3 --repeats 1	A fast inner-loop run over a small subset.
--baseline <prior.json>	See per-image deltas against a previous run.
--assert	Enforce the regression gate (see below).

Measuring and publishing are separate steps. To turn a run into the numbers shown above:

task python:publish

distils the report into the committed benchmarks/published/latest.json — headline figures plus the provenance block naming the sceptre, ONNX Runtime, EasyOCR and torch versions behind them — and regenerates the marked table regions in the README and on this page. A report with no provenance block is refused rather than published. task python:publish:check re-renders without writing and fails if the committed tables have drifted from the artifact; CI runs it on every pull request, and it needs neither torch nor a built binary. See ADR 0030.

cargo bench covers internal hot paths (criterion microbenchmarks); the head-to-head harness above measures end-to-end OCR against EasyOCR. Parity fixtures backing the quality signal live under crates/sceptre/tests/data/golden/.

Note

The regression gate (--assert) enforces a floor on warm speedup against EasyOCR, and a ceiling on sceptre’s own peak RSS relative to the published baseline for the same host — a cross-engine memory ratio makes a poor gate, since peak RSS is dominated by the CRAFT detector both engines share (ADR 0042). Labeled quality is held by per-image guardrail contracts. The exact thresholds live in the harness, not here. Run manually via workflow_dispatch in CI, not on every PR (the reference run needs torch, easyocr, and the full model set).