Packet-based neural processing units, architected for the compute-bound convolutional neural networks of the last decade, now face workloads dominated by memory constraints as large language models move to the edge and into vehicles. The shift reframes how engineers must evaluate, architect, and benchmark NPUs.

Semiconductor Engineering examines the transition in a technical analysis of NPU design in the LLM era. The core argument: the performance bottleneck has migrated. CNNs of the earlier deep-learning generation kept arithmetic units busy; transformer-based LLM inference does not.

Why were CNN-era NPUs compute-bound?

Convolutional workloads exhibit high arithmetic intensity. The same input feature maps feed many multiply-accumulate operations, so a well-designed NPU could stream data once and reuse it across thousands of MACs. Under those conditions, performance scaled with the number of arithmetic units and their clock frequency.

Packet-based NPU architectures fit this regime. They decompose neural network execution into packets of work — data movements and computation scheduled as discrete, routable units rather than monolithic program flows. That structure gave designers fine-grained control over dataflow and let compute engines stay saturated.

What changes when the workload is an LLM?

Large language models invert the arithmetic-intensity picture. Transformer inference, particularly the decode phase of autoregressive generation, performs relatively few operations per byte of weight and activation data moved. The result: memory bandwidth, not MAC count, sets the ceiling on tokens per second.

This has direct consequences for NPU design targets:

• Weight fetching dominates energy and latency, since billions of parameters must stream through the engine.
• On-chip memory capacity and hierarchy become first-order design constraints, not afterthoughts.
• Data reuse strategies that worked for convolutions map poorly onto attention and matrix-by-vector operations.
• Benchmarking on FLOPS alone becomes misleading; bytes-per-second per watt is the metric that tracks user-visible performance.

Why do edge and automotive make it harder?

The analysis centers on edge and automotive as the environments where this transition bites hardest. Both impose constraints that datacenter LLM serving does not.

Edge devices carry strict power budgets and thermal limits. A memory-bound workload under a tight power cap forces a direct trade-off: every joule spent moving weights is a joule unavailable for anything else. Automotive adds certification, determinism, and functional-safety requirements on top, and vehicle platforms must run inference for the lifetime of the car on fixed hardware.

For packet-based architectures specifically, the scheduling and routing machinery that once maximized compute utilization must now be redirected toward keeping memory pipelines full and minimizing redundant data movement across the chip.

What does this mean for NPU architects?

The article's framing suggests designers should treat the NPU as a memory system with arithmetic attached, rather than the reverse. Practically, that pushes attention toward:

• Sizing on-chip buffers against real LLM layer shapes rather than CNN feature maps.
• Scheduling packets to maximize weight reuse before eviction.
• Accounting for the decode-phase memory traffic that dominates interactive inference.
• Selecting benchmarks that reflect token-generation latency, not top-line compute throughput.

The broader market read

The CNN-to-LLM workload transition reaches every silicon vendor building inference engines for phones, vehicles, and embedded systems. Architectures tuned to saturate arithmetic arrays risk looking strong on legacy benchmarks while underdelivering on the generative workloads customers now ship. Vendors that rebalance their designs around memory bandwidth, on-chip capacity, and data-movement efficiency are positioned for the workloads of the current cycle.

For engineering teams, the takeaway is a evaluation-methodology change as much as a silicon change: procurement and benchmarking criteria built on CNN-era throughput figures will misrank candidate NPUs for LLM deployment. Memory-bound behavior must sit at the center of the assessment.

The full technical discussion appears in Semiconductor Engineering's coverage of packet-based NPU design in the LLM era.