Benchmarks, measured

Every number on this page is derived from the IR by one piece of code, for every design, by the same method. Regenerate it:

cargo test -p ferrite-lithic-corpus --test graph_metrics -- --nocapture --ignored

The point of saying that first is that most hardware size figures in documentation are estimates written in a doc comment. They are honest as estimates and useless as measurements: nothing re-derives them when the design changes. These are re-derived every run.

How to read the columns

Two of these are routinely misread, so both are stated as what they are not.

Area, power and frequency are not derivable from the IR and are not claimed anywhere in this project. A number on this page is a fact about the graph; what a synthesiser builds from that graph is a different question that needs a different tool.

Every design, measured

How much slower? ×-ratios

The question every engineer actually asks about a hardware implementation: is it faster than just doing it in software? Measured per byte, against the crate that defines each algorithm — crc32fast, crc, hex, base64, memchr — all already dev-dependencies of the corpus, so the reference is the real thing rather than a reimplementation.

Each row asserts that both sides produce identical output before either is timed. Without that, a fast design would win by doing less work.

Two of the five beat their reference implementation. That is not the result the project expected to publish, and it is why the benchmark was written.

A ratio needs a clock: the design is measured in cycles, the software in nanoseconds, and converting between them requires a frequency. Everything above is at an assumed 1 GHz, so read the ratio with that attached. At 2 GHz every design figure halves.

Why the hardware is slower, and when it isn't

Because the comparison is rigged, and knowing how is the point.

memchr's design is 31 nodes and 3 flops. An AVX2 memchr is a CPU core with hundreds of millions of transistors, 32 bytes per instruction, a deep memory hierarchy and a branch predictor. So "hardware loses" is really "one fingernail-sized circuit lost to an entire core". The ratio measures how much silicon the design was allowed, not how good it is.

Three reasons it loses:

1. The CPU amortises everything. Software runs the algorithm once, then applies it to a megabyte with the code in L1, the table in registers, the branch predictor trained. None of that setup is charged per byte. A circuit has no program to amortise — its state is the program, and it is rewritten every cycle.

2. Software is allowed to be cleverer than hardware. Software can be asymmetric: process 32 bytes at once, skip the common case, specialise on a length it can see. Hardware has no equivalent of "do the thing that is usually right" — a mux is a mux, and you pay for it whether or not the select is ever anything but zero.

3. The design is deliberately serial. crc32 accumulates one byte per edge. Its recurrence is linear, so software can fold several bytes and break the dependency chain; the design as built cannot, and a pipelined variant would be a different design.

Then why do crc3 and hex win? Because the win comes from doing less, not going faster. The crc crate is generic over width and pays per-byte machinery for a computation whose real state is three bits. The design is three flops, a table, and one byte per edge, with no generality to pay for.

Hardware loses when the CPU has more parallelism available than the design uses, and wins when the design's smallness is the point.

The assumption is load-bearing: crc32 will not make 1 GHz

crc32's measured combinational depth is 34 operator levels — a table-driven CRC is 32 XORs deep. Logic that deep does not close at 1 GHz on a modern process; it runs at a few hundred megahertz. So its true figure is worse than 8.6×, by the ratio of its real clock to 1 GHz.

This is why the depth column in the table above and the ratio column here have to be read together. sha256, at 644 levels, could not close at any clock worth having.

What hardware is actually for

None of this is an argument against hardware, only against general-purpose hardware in a general-purpose comparison. A dedicated circuit earns its place when the CPU cannot do the job at all (streaming, no memory to buffer into), when latency must be bounded (a fixed pipeline has no cache misses, no mispredicts, no preemption), when the CPU is busy with something else, or when the algorithm is regular enough that a small circuit covers it — which is exactly the crc3 and hex case.

Against a modern CPU on the same data, with both free to use everything, a small straightforward circuit usually loses. That is a fact about transistors.

Three structural findings

1. mem bits is zero everywhere, which is the memory gap with evidence behind it

There is no Mem node in the entire corpus. Design::rom and Design::mem emit Case and Constant nodes instead, and you can read that straight off the memchr pair:

The ROM variant is 256 constants and a multiplexer tree, not a memory. That is honest, it simulates correctly, and it is the wrong thing for an FPGA block RAM. This is the project's largest known gap, and it is now a measurement rather than an assertion.

2. sha256 and chacha20 have combinational paths hundreds of levels long

sha256 is 644 operator levels deep. chacha20 is 484. aes is 98.

Both ciphers unroll every round on purpose, to get one block per cycle. That is the trade, and the depth is its price: no synthesis flow meets timing on a 644-level combinational path, and neither design is pipelined. The project is not claiming these are shippable as-is.

This is worth stating rather than hiding because it is the kind of cost that otherwise surfaces in a timing report months later. combinational_depth_is_bounded pins the worst case at 650 levels, so an unrolled design that grows a round cannot pass unnoticed.

3. The wall-clock harness measures the simulator, not the hardware

tests/memchr.rs wraps the software simulator in a timer and compares its wall clock against the crate's. It does not assert the ratio, and the number moves by more than 8x between build profiles on one host — because it depends on host load, build profile and clock rather than on the design.

What it measures is the simulator: a software interpreter stepping one node per cycle costs far more wall time than the single clock edge it is modelling. It is a debugging signal, not a hardware figure, and tests/speed.rs exists to replace it.

The claim that survives needs no wall clock at all:

The design retires one byte per clock edge by construction. The crate retires 32 bytes per AVX2 instruction.

Reproducing

# Structural metrics, all 24 design/variant pairs. Deterministic, no timing.
cargo test -p ferrite-lithic-corpus --test graph_metrics -- --nocapture --ignored

# Design cycles per unit vs the reference crate's wall clock, at a stated 1 GHz.
cargo test -p ferrite-lithic-corpus --test speed --release -- --nocapture --ignored

Both are excluded from the default suite. The first because it prints rather than asserts. The second because a timing assertion is a flake generator — and the assertion it does make is deliberately almost empty, checking only that the measured figure is finite and positive.

CI runs the corpus with --nocapture so the printed ratio reaches the job log, and fails if anything reported SKIPPED. Without that rule a green run in which every equivalence check did nothing would be green and meaningless.

What is not measured

Stated plainly, because a benchmarks page that only lists wins is marketing:

• Area and power. Not derivable from the IR.
• Frequency. Would need a synthesis flow and a target process.
• Throughput per design, uniformly. Only some designs have a cycle-rate assertion (deflate at exactly 9 cycles per byte, aes at one block per cycle, hex at 4 bits per cycle, huffman at one symbol per code length). Building a fair rate benchmark for the other sixteen would mean writing a driver per design, which is most of a test suite.
• Anything about real silicon. Every figure here is about the graph.