Is One Layer Enough? Training Single Transformer Layer Matches Full RL

What Happened

A research team trained a single transformer layer with reinforcement learning and matched the benchmark performance of full-depth transformer policies across standard RL evaluation suites. The result holds on tasks where depth was previously treated as a fixed architectural requirement. The finding isolates layer count as a variable that does not determine policy quality under RL training regimes.

Why It Matters

If depth is not load-bearing for RL policy quality, then a large fraction of current architecture selection is untested convention rather than empirical constraint. Training cost in RL scales with parameter count and forward-backward pass depth; collapsing a 12- to 24-layer stack to a single layer reduces both simultaneously. For operators running large-scale RL pipelines—robotics manipulation, simulated game environments, continuous control—this is a direct cost lever rather than a marginal efficiency gain. The immediate problem it solves is iteration throughput: shorter training runs mean more policy updates inside a fixed wall-clock budget, which compounds across a research or deployment cycle. Beneficiaries are teams whose compute is dominated by RL rollouts and gradient updates rather than inference.

Technical Details

The single-layer configuration was evaluated on standard RL benchmarks and matched full-depth baselines on return and convergence behavior. Depth reduction cuts per-step compute proportionally to the number of layers removed, with memory footprint falling alongside activation storage. The result appears to hold across the tested task distribution, but the paper does not establish whether it generalizes to tasks requiring compositional or hierarchical credit assignment—precisely the regimes where depth is theoretically motivated. Integration is straightforward: existing RL training code accepts a depth parameter, so the change is a config edit rather than a rewrite. Limitations center on task scope and on whether the parity is asymptotic or holds across the full training trajectory, including early-training instability regimes.

Operational Impact

Builders should run a single-layer ablation on their own domains before committing to another full-depth training run. The concrete workflow change is treating depth as a hyperparameter to sweep rather than a fixed inheritance from supervised-learning conventions. For a pipeline currently training 12-layer policies, a validated single-layer configuration reduces per-run cost by roughly an order of magnitude in FLOPs, or enables proportionally more parallel seeds within the same budget. This also shortens wall-clock iteration cycles, which matters more than raw cost for teams doing frequent policy updates. Infrastructure already optimized for parameter efficiency—quantization, small-model serving, edge deployment—becomes directly applicable to RL policies that previously assumed larger backbones.

What To Watch

The second-order question is whether the same depth-irrelevance holds for attention mechanisms and other components currently treated as fixed requirements; if depth was never load-bearing, other inherited architectural defaults deserve the same ablation. Over the next 6–12 months, expect replication attempts on harder task distributions—long-horizon planning, sparse-reward environments—where the result may or may not survive. If it does, model selection conventions in RL shift from depth-first to ablation-first, and the compute savings get redirected into environment scale and rollout diversity rather than policy capacity.