44M Parameter Quantized LLM Trained from Scratch in 19.8 MB at 1,900 tok/s

What Happened

An r/MachineLearning author reported training a 44M parameter quantized language model from scratch on 45 billion tokens. The final artifact ships in 19.8 MB and runs at approximately 1,900 tokens per second on CPU without GPU acceleration. The model was trained entirely from scratch rather than fine-tuned from an existing checkpoint, meaning the tokenizer, architecture, and quantization scheme were all defined by the author.

Why It Matters

The deployment envelope for language models has been bounded by VRAM and accelerator availability, which excludes a large class of devices: embedded controllers, air-gapped terminals, legacy x86 fleets, and consumer hardware without discrete GPUs. A 19.8 MB artifact fits in L3 cache on many modern CPUs and can be embedded directly in application binaries, firmware images, or browser bundles. For builders shipping offline-capable or privacy-constrained products, this removes the dependency on inference endpoints and the latency, cost, and data-egress surface that come with them. The 1,900 tok/s figure also shifts CPU inference from "technically possible" to "interactive," which reopens use cases previously ceded to GPU-hosted APIs.

Technical Details

At 44M parameters and 19.8 MB, the effective footprint implies roughly 3.6 bits per parameter, consistent with an aggressive post-training quantization scheme applied to a compact transformer. Training on 45 billion tokens against a 44M parameter model represents a token-to-parameter ratio near 1,000:1, far above Chinchilla-optimal, indicating deliberate overtraining to compensate for capacity constraints and improve quantization robustness. Throughput of 1,900 tok/s on CPU implies heavy reliance on SIMD kernels, cache-resident weights, and likely a small vocabulary or restricted context window; the reported figures do not specify sequence length, batch size, or CPU SKU. Practical limitations follow from scale: 44M parameters cannot match mid-size models on reasoning, multi-step instruction following, or long-context tasks — the model is suited to classification, extraction, templated generation, and narrow-domain assistants.

Operational Impact

Inference cost for suitable workloads drops to zero marginal dollars, since the model runs on hardware already deployed. Teams currently paying per-token for classification or routing layers can collapse those calls into local processes, eliminating network round-trips and reducing p50 latency from hundreds of milliseconds to single-digit milliseconds. Packaging changes: a 19.8 MB model fits inside app installers, container images, and firmware partitions that were previously off-limits, removing the need for a sidecar inference service. The quantization and training pipeline becomes the reusable asset — operators who internalize the recipe can produce task-specific small models at low cost, which weakens the case for general-purpose API calls in high-volume, low-complexity paths. Evaluation burden shifts from prompt engineering to dataset curation and quantization-aware training.