A normalized accelerator comparison is one of this project’s stated pillars, and
it is real analytical work rather than a table anyone can copy. This page
establishes the method; the filled comparison follows once the HBM vertical is
finished.
The problem is that a headline TOPS or TOPS/W figure is not a measurement. It is
an arithmetic identity evaluated under conditions the vendor chose, and two
vendors almost never choose the same ones.
The axes that make two numbers incomparable
Precision. An operation counted at INT4 is not the operation counted at
INT8, which is not the FP16 one. Doubling the headline number by halving the
precision is free and extremely common.
Sparsity. Many figures assume a structured sparsity ratio and count the
skipped multiplies as though performed. Whether the workload exhibits that
sparsity is a separate question, usually unaddressed.
Peak versus achieved. Peak is the MAC array multiplied by the clock. It
assumes every unit is fed every cycle, which the memory system generally cannot
do. The gap between the two is the memory wall, expressed as a number.
Clock and power conditions. Boost or sustained; typical or worst-case;
with or without the memory system’s power in the denominator of TOPS/W.
Batch size. Large batches raise arithmetic intensity and flatter throughput
figures, at a latency cost that a latency-bound deployment cannot pay.
Why peak is the wrong number
The TPU paper is the useful reference point because it documents its own
conditions unusually honestly. It reports a peak of 92 TOPS from a 65,536-unit
8-bit MAC array, and then shows via roofline analysis that most of the evaluated
production workloads sit in the memory-bandwidth-limited region rather than the
compute-limited one. The paper’s own counterfactual is the clearest statement of
the problem: substituting the GPU’s GDDR5 memory into the same accelerator would
roughly triple achieved throughput. The compute did not change. The memory did.
Eyeriss makes the same point from the design side. Its dataflow is organised
around minimising operand movement rather than maximising arithmetic, on the
argument that fetching an operand costs far more energy than the multiply that
consumes it. An accelerator’s energy efficiency is largely a statement about
its memory hierarchy.
What this project will record
For each accelerator entered into the comparison, the normalized table will
carry the headline claim and the conditions attached to it: precision,
sparsity assumption, peak versus achieved, clock condition, batch size, and
whether memory power is included. Where a vendor does not disclose a condition,
that will be recorded as undisclosed rather than assumed.
Where two sources give different figures for the same part, both are recorded
and the conflict is flagged. Never averaged, never silently resolved.
Sources
paperPaywalled — preprint availableNorman P. Jouppi, Cliff Young, Nishant Patil, David Patterson · ISCA 2017 (44th Annual International Symposium on Computer Architecture), pp. 1-12, 2017
The memory wall measured in a production accelerator rather than argued in the abstract. Its roofline analysis shows most of the evaluated neural network workloads sitting in the memory-bandwidth-limited region, and the abstract states that substituting the GPU's GDDR5 memory into the TPU would triple achieved TOPS — an accelerator whose headline compute number is gated by the memory it is attached to. 75 authors; the arXiv preprint is complete and freely readable. Also the reference point for how a vendor constructs a TOPS figure, since it documents its own conditions unusually honestly.
Yu-Hsin Chen, Tushar Krishna, Joel S. Emer, Vivienne Sze · IEEE Journal of Solid-State Circuits 52(1), pp. 127-138, 2017
The academic accelerator design that made data movement, rather than arithmetic, the explicit optimisation target — its dataflow exists to avoid DRAM accesses, on the argument that moving an operand costs far more energy than the multiply that consumes it. The clearest single citation for why accelerator design is memory-wall design. Extended from the ISSCC 2016 version (doi:10.1109/ISSCC.2016.7418007).