ArXiv

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

2026-08-05 · 17:58 UTC ·Indraneil Paul, Falko Helm, Goran Glavaš... ·1 min read

Authors: Indraneil Paul, Falko Helm, Goran Glavaš...
Categories: cs.AI, cs.LG, cs.SE
arXiv: https://arxiv.org/abs/2608.05141v1
PDF: https://arxiv.org/pdf/2608.05141v1

Brief

OctoLong engineers dependency-aware, recursive code-context retrieval (AST + LSP + package-manager) to produce dependency-rich code contexts millions of tokens long. The authors mid-train OctoLong-Instruct models (600M–14B) on a ~50B-token mixture (≈6.2B OctoLong tokens) plus ~10B instruction-tuning tokens; replacing 12% of context-extension corpora yields measurable gains versus 18 open-weight long-context LMs. Only the abstract was available.

Why it matters

OctoLong instruments an AST parser, a language-server backend, and a package manager to recursively retrieve code references and curate dependency-rich code contexts millions of tokens long.

Key details

OctoLong-Instruct (models 600M–14B) was mid-trained on a ~50B-token mixture containing ≈6.2B OctoLong tokens and followed by ≈10B tokens of instruction tuning; supplanting 12% of traditional context-extension data with OctoLong improved long-range retrieval, long-term state tracking, repository-level code understanding, agentic tasks, and short-context API usage versus 18 open-weight baselines.

Cleaned source text

Abstract