Backends

The ModelBackend seam

All model execution — the CRAFT detector and every gen2 recognizer — goes through one trait: inference::ModelBackend. The detection, recognition, and engine code call this trait only; they never call ort, tract, or candle APIs directly. config, types, and the geometry code stay entirely backend-agnostic, so a config or result type is never conditionally compiled per backend.

This keeps the backend choice non-load-bearing: a new backend implements the trait and slots in without touching the pipeline. See ADR 0004.

Choosing a backend

Backend	Feature	Target	Notes
ort	ort, ort-bundled, ort-dynamic	Desktop / server	Native ONNX Runtime, and the fastest by a wide margin. ort-bundled fetches a prebuilt runtime at build time (zero-config, and the CLI default); ort-dynamic dlopens libonnxruntime at runtime via ORT_DYLIB_PATH, which is the path on targets with no prebuilt.
tract	tract	WASM / Android, and targets with no prebuilt ONNX Runtime	Pure-Rust ONNX interpreter — no native library to ship or load. CPU-only.
candle	candle, candle-metal, candle-cuda	GPU, or a build with no ONNX Runtime	Does not interpret the ONNX graph: it runs hand-written CRAFT and CRNN networks over the weights read from the same ONNX files. Runs on the CPU, Apple Metal, or CUDA.

Pick ort for CPU/RSS-optimized execution on desktop or server; pick tract when you need a pure-Rust build with no native dependency — WASM, Android, or any target ONNX Runtime ships no prebuilt for; pick candle when you want a GPU without provisioning a provider-specific ONNX Runtime, or a native build with no ONNX Runtime at all. See ADR 0012 for the ort-bundled/ort-dynamic provisioning tradeoff, ADR 0029 for why the CLI defaults to ort-bundled and how the accelerator is selected, and ADR 0031 for how candle runs these models and what it costs.

The active backend is selected through ModelConfig::backend (Backend::Ort / Backend::Tract / Backend::Candle), or --backend on the CLI.

Accelerator selection

ModelConfig::accelerator (--accelerator on the CLI) chooses the hardware the backend runs the graph on: cpu (the default), auto, coreml, directml, metal, or cuda. The vocabulary is named for the seam rather than for ONNX Runtime, but not every backend answers every value — ort reaches hardware through ONNX Runtime execution providers while candle addresses devices directly:

Backend	Accelerators	Cargo feature
ort	coreml, directml, cuda	ort-coreml, ort-directml, ort-cuda
tract	— (CPU only)	—
candle	metal, cuda	candle-metal, candle-cuda

cuda appears on both because it names hardware rather than an execution provider. coreml and metal are not interchangeable: both drive the same Apple GPU, but through different frameworks with different numerics, so each belongs to one backend. Any other pairing is rejected at config-validation time, with the equivalent on your backend named in the error. See ADR 0032.

The default is cpu rather than auto deliberately. The same ONNX graph produces different numeric output on different providers, and sceptre’s published parity and benchmark figures are CPU figures; defaulting to auto would silently change a user’s results on upgrade.

Caution

An explicitly requested accelerator that cannot be used is a hard error, on either backend. ONNX Runtime’s ExecutionProviderDispatch defaults to failing silently, which would make “asked for CoreML, got CPU” look like success, so explicit requests register with error_on_failure; candle refuses a device its build did not compile in rather than opening the CPU. Only auto falls back — it walks candidates and reports the one that actually took.

runtime_info() (and sceptre env --format json) reports the backend, the requested and the registered accelerator, the ONNX Runtime provisioning strategy and version, and the target arch — the environment any parity or benchmark artifact has to record alongside its numbers.

Caution

candle is not the fast option, on a GPU or otherwise. Measured on a local Apple Silicon machine at the default canvas, its CPU path is roughly 10x slower than ort on the CPU, and Metal closes only part of that — still ~3-4x slower than ort. The size of that gap is host-dependent: on a core-constrained CI runner it narrows to roughly 2x and 1x, because ort has fewer cores to exploit. The ordering holds regardless. Choose candle for what it removes (no ONNX Runtime, no provider-specific runtime build), not for speed. It is excluded from the published benchmark gate for that reason.

Caution

GPU numerics are not covered by CI. No hosted runner has a GPU, so the Metal and CUDA paths are compiled in CI and never executed there. Metal is validated against ort on real hardware — at the tensor level and on end-to-end word agreement — and CUDA is compiled only.

One concurrency budget

Parallelism flows through a single ConcurrencyConfig::max_threads budget. Rayon’s thread pool and the backend’s own intra-op threads (once wired) draw from the same budget, so nested parallelism — Rayon fanning out across images while the backend also parallelizes inside a single model call — cannot oversubscribe the CPU. max_threads defaults to num_cpus, capped at 8. See Configuration.