Samsung's In-Memory AI Compute Needs the CPU's Caches Switched Off

Samsung presented its processing-in-memory work at Hot Chips, and the write-up published early on 29 August puts numbers on both sides of the trade. The bandwidth figure travelled; the architectural cost did not.

Where the bandwidth is

The large multiple compares bandwidth available to compute units inside the memory device against the bandwidth the same device exposes across its external interface. That is the entire point of putting compute in memory: the wide internal bus is the resource, and it was always there — what changes is that arithmetic units now sit on the near side of it. It is not extra bandwidth delivered to the host processor.

The cost the host pays

For the scheme to work, the memory holding PIM operands must be mapped as non-cacheable and non-speculative. A cached copy would be stale the moment the in-memory unit wrote to it, and a speculative read could trigger a computation that architecturally never happened. So the processor addressing that region loses caching, loses hardware prefetch, and cannot reorder around those accesses. Three of the mechanisms that make modern cores fast are switched off for exactly the data being accelerated.

What the common framing gets wrong

Reporting has treated the internal figure as a system-level speedup — as though a machine gains eight times the memory bandwidth. It does not. The correct claim is narrower and still interesting: for operations that are bandwidth-bound and arithmetically trivial, chiefly the elementwise and reduction work in LLM decode, moving the operation to the data avoids a round trip that the external bus would otherwise dominate. For anything compute-dense, PIM is the wrong place and the disabled cache is pure loss.

The throughput in context

Per-package compute exposed outside is a couple of INT8 TOPS; a set of eight lands in single-digit tens of TOPS — comparable to the NPU in a mainstream laptop processor. The distinguishing feature is not raw throughput but that the throughput is attached to the full memory capacity of those devices, which is where large-model decode actually hurts.