Twitter/X

Layer streaming loads one layer at a time from disk; AirLLM claims to run Llama…

2026-08-06 · 00:52 UTC ·@hasantoxr ·1 min read

Brief

AirLLM uses layer-streaming—loading one layer at a time from disk so the rest stays on disk—to run very large models with tiny GPUs. It claims Llama 3.3 70B on 4GB, Llama 3.1 405B on 8GB, DeepSeek‑V3 671B on ~12GB and Kimi K3 (2.8T) on 3.72GB. It adds a single AutoModel for Hugging Face, 4/8‑bit block compression, prefetching, CPU and FP8 support, MoE expert streaming, Apple Silicon support, and auto-delete of original checkpoints without quantization.

Why it matters

Layer streaming loads one layer at a time from disk; AirLLM claims to run Llama 3.3 70B on 4GB, Llama 3.1 405B on 8GB, DeepSeek‑V3 671B on ~12GB, Qwen3‑235B on ~3GB, and Kimi K3 (2.8T parameters) on 3.72GB of VRAM.

Key details

  • AirLLM provides performance features: a single AutoModel line for Hugging Face, 4-bit and 8-bit block-wise compression (advertised as 3× faster inference), prefetching that overlaps disk loading with compute, CPU-only inference capability, and FP8 support.
  • Platform and model support includes streaming MoE models one expert at a time, macOS/Apple Silicon support, auto-delete of original checkpoints to save disk space, and claims no quantization, distillation, or pruning is required to fit models.
Source evidence

Layer streaming is the next step after quantization.

Most people still think running a 70B model means renting an A100 or waiting for the next Nvidia release.

AirLLM works a different way. It does not load the whole model at once. It loads just one layer at a time, and the rest stays on the disk.

Inside:

→ Llama 3.3 70B on a 4GB GPU
→ Llama 3.1 405B on 8GB
→ DeepSeek-V3 671B on ~12GB
→ Qwen3-235B on ~3GB
→ Kimi K3 (2.8T parameters) on 3.72GB of VRAM
→ MoE models that stream one expert at a time instead of a whole layer
→ MacOS support on Apple Silicon

It also gives you:

→ One AutoModel line that works for almost any Hugging Face model
→ 4-bit and 8-bit block-wise compression for 3x faster inference
→ Prefetching that overlaps disk loading with compute
→ CPU inference for machines with no GPU at all
→ FP8 support for the newest models
→ Auto-delete of the original checkpoint to save disk space
→ No quantization, distillation, or pruning required to fit