SLAI T-Rex – Full-parameter post-training of DeepSeek-V4 on Ascend SuperPOD

What Happened

A technical paper from SLAI documents full-parameter post-training of DeepSeek-V4 on an Ascend SuperPOD cluster, using Huawei's accelerator stack rather than NVIDIA GPUs. The writeup has accumulated 31 upvotes on Hugging Face, indicating practitioner-level engagement rather than passive visibility. The run covers the post-training phase specifically—not pretraining—on a model in the DeepSeek-V4 class.

Why It Matters

Post-training is where most teams now spend the majority of their compute budget, and it is also the phase most sensitive to orchestration, precision handling, and checkpointing behavior rather than peak FLOPS. A documented, reproducible run on Ascend materially lowers the perceived risk of routing that phase away from NVIDIA hardware. For operators stuck in GPU allocation queues or operating under procurement constraints, this converts "alternative accelerator" from a theoretical hedge into a validated option with a public reference. The second-order effect is organizational: compute planning shifts from single-vendor dependency toward multi-vendor portfolios, which changes how teams think about toolchain lock-in, staffing, and contract structure. It also gives builders a concrete baseline against which to benchmark their own workflow portability, which reduces the cost of evaluating a migration.

Technical Details

The run executes full-parameter post-training—not LoRA or other parameter-efficient variants—which means optimizer states, gradient buffers, and activation memory all scale with the full model rather than a low-rank adapter. This matters because full-parameter tuning on non-NVIDIA hardware exercises the entire memory hierarchy and collective communication path, not a reduced subset. Orchestration on Ascend SuperPOD relies on Huawei's CANN stack and associated collective libraries (HCCL), which differ from CUDA/NCCL semantics in ways that surface during gradient synchronization, mixed-precision loss scaling, and checkpoint sharding. The paper does not claim throughput parity with NVIDIA clusters; the operative claim is that the run completed with acceptable convergence behavior. Convergence fidelity, not raw speed, is the load-bearing result for post-training, since a faster run that drifts from the reference loss curve is unusable.

Operational Impact

Teams should now audit their training stack for Ascend compatibility as a concrete near-term task: enumerate dependencies on CUDA-specific kernels, NCCL assumptions, FlashAttention variants, and any custom fused operators. Anything that compiles through standard frameworks (PyTorch with device-agnostic abstractions) migrates with moderate effort; anything hand-written in CUDA does not, and that is the real cost surface. Checkpoint formats, sharding strategies, and data pipeline throughput need explicit validation, since these fail silently and expensively late in a run. The immediate benefit is optionality: a team that can articulate its Ascend portability posture negotiates GPU allocation from a different position, and can route post-training bursts to whichever cluster is available rather than waiting on a single vendor. Pretraining remains largely NVIDIA-anchored for now, so the pragmatic split is post-training on Ascend, pretraining where it currently lives.