The demo begins cold. The camera wakes, the accelerator races through its first thousand frames, and the latency chart looks terrific. Twenty minutes later, inside a sealed roadside enclosure in August, the same model misses its response target.
Nothing mysterious happened. The product team sold peak compute while the customer bought sustained behavior.
TOPS, or trillions of operations per second, can help compare silicon. It is a weak product promise on its own. Real edge systems share power across cameras, radios, storage, CPUs, and accelerators. They live inside enclosures. Dust clogs vents. Sunlight heats the box before inference begins.
The useful promise is a sustained performance envelope: the latency, throughput, accuracy, and power a complete product can maintain under named operating conditions.
Cold benchmarks borrow from the future
A short benchmark can spend thermal capacity that the deployed product does not have. The accelerator runs fast while the heat sink absorbs energy. Once temperatures settle, clocks may fall, fans may accelerate, or other components may lose power headroom.
Call the difference between cold-start performance and steady behavior heat debt. A team incurs heat debt when a demo uses temporary thermal headroom and presents the result as continuous capacity.
NVIDIA makes the limitation unusually explicit for Jetson Orin. Its MAXN setting allows broad access to cores and clock frequencies, yet NVIDIA says hardware throttling can engage when module power exceeds the thermal design power budget. MAXN therefore does not guarantee the best performance, and prolonged heavy workloads require conservative thermal settings.
That is a product warning, not a footnote for electrical engineers.
Put the enclosure into the requirement
An edge AI requirement should describe the system that ships. At minimum, record:
- ambient temperature and solar exposure;
- enclosure, heat sink, fan policy, and airflow;
- input voltage and selected power mode;
- camera count, resolution, frame rate, and codec;
- concurrent CPU, storage, radio, and accelerator work;
- test duration, tail latency, throughput, and accuracy.
The duration matters because thermal equilibrium has a clock. Raspberry Pi documents an 85°C throttling point and identifies persistent workloads, high ambient temperatures, and airtight enclosures as conditions that may need added cooling. The board protects itself, but protection reduces performance. A two-minute test and a two-hour shift answer different questions.
This also changes architecture choices. The product boundary set by camera interfaces determines cable reach and compute placement; it also determines where decoding heat, memory traffic, and inference load accumulate.
Measure the whole box
Chip power is not product power. A realistic test includes the conversion losses, storage writes, network activity, and cooling system that arrive with the accelerator.
MLCommons takes this boundary seriously in its Edge inference results. It defines workload scenarios with latency and throughput metrics. For power submissions, it measures average AC power for the complete system during the performance run, and it warns that a component rating or stated TDP is not a validated power measurement.
Product teams can borrow that discipline without running a formal MLPerf submission. Use a production image, production sensors, the intended enclosure, and representative background services. Measure at the wall or battery terminals. Keep the warmup period, then run until temperature and latency stop drifting.
Average latency alone is still too polite. Report a high percentile, missed deadlines, dropped frames, temperature, clock state, and energy per useful inference. A system that maintains 30 frames per second but violates the response deadline every few minutes may be unsuitable for control.
Make degradation visible
The sustained envelope should include what happens outside its boundary. The device might reduce frame rate, select a smaller model, defer nonessential work, increase fan speed, or ask for remote help. Each response has a customer cost: lower accuracy, extra noise, slower service, or lost autonomy.
Expose those tradeoffs before launch. A thermal alarm that only appears in an engineering console cannot support field operations. Fleet software should show which devices are throttling, which workload triggered the change, and whether performance returned after conditions improved.
Peak TOPS still belongs on a data sheet. The product promise belongs elsewhere: a repeatable workload, a real enclosure, a defined ambient range, and performance that survives long enough to do the job.
Comments
Comments are powered by GitHub Discussions through Giscus.