CPU-ARC: Agentic Reasoning Caching for On-CPU Inference Reuse

What Happened

A new ArXiv paper introduces CPU-ARC, a caching and reuse technique for agentic reasoning traces executed on CPU rather than GPU. The method stores intermediate reasoning outputs from agent runs and retrieves them when semantically equivalent sub-tasks recur, avoiding redundant inference. The work targets test-time compute efficiency specifically for agent workloads where repeated task structures produce overlapping reasoning paths.

Why It Matters

Agent deployments frequently re-derive the same intermediate reasoning — the same tool-selection logic, the same decomposition steps, the same validation checks — across runs that differ only in surface inputs. Each redundant derivation carries full inference cost. CPU-ARC treats reasoning traces as a cacheable artifact, which converts a per-call compute expense into a one-time cost plus retrieval overhead. For operators running high-volume agent pipelines, this reframes reasoning from a variable cost that scales linearly with requests into something closer to a fixed asset with amortization. The technique lands on CPU because reasoning traces are often small, structured, and latency-tolerant relative to matrix operations — a placement decision that also avoids competing with GPU capacity for model serving. The practical implication is that agent economics become sensitive to cache hit rate rather than raw token throughput, which shifts optimization effort toward workload homogeneity and trace governance.

Technical Details

CPU-ARC operates by capturing reasoning traces — sequential intermediate outputs produced during agent execution — and indexing them for semantic retrieval on subsequent runs. The cache layer sits on CPU, leveraging the fact that similarity matching and trace storage are memory-bound rather than compute-bound, keeping GPU resources dedicated to model inference. The paper frames the objective as reducing redundant inference during test-time compute, a category that has grown as reasoning models spend more tokens per query. Precise benchmark numbers and cache hit-rate curves are reported in the source; the architecture assumes the agent's reasoning is decomposable into retrievable segments rather than monolithic. Key constraints follow from that assumption: workloads with high semantic diversity or strict determinism requirements around freshness will see lower hit rates, and cache invalidation for evolving tool outputs or changing world state is a non-trivial design surface. Integration requires instrumenting the agent loop to emit structured traces, not just final outputs.

Operational Impact

Day-to-day, builders gain a new tunable in the cost model: trace cache size, eviction policy, and similarity threshold become first-class deployment parameters alongside model choice and context window. Workflows that today re-run full reasoning for near-duplicate requests — batch classification, recurring monitoring tasks, templated multi-step pipelines — can route through the cache and cut per-request inference. This also changes observability requirements: teams need hit-rate metrics, staleness tracking, and trace lineage to trust cached reasoning in production. The technique makes homogeneous agent fleets cheaper to operate and makes heterogeneous, long-tail workloads comparatively more expensive, which may push operators toward standardizing task templates. Existing prompt-caching and KV-cache approaches address a different layer (token-level state within a session); CPU-ARC targets cross-session reasoning reuse, so it complements rather than replaces them.