ArXiv

SparseDitto: Customizing GPU Kernels for Different Sparsity Patterns with LLM-Based Agentic System

2026-08-05 · 16:41 UTC ·Shiyang Li, Guangyan Sun, Jinwei Tang... ·1 min read
Authors
Shiyang Li, Guangyan Sun, Jinwei Tang...
Categories
cs.DC, cs.LG
arXiv
https://arxiv.org/abs/2608.05033v1
PDF
https://arxiv.org/pdf/2608.05033v1

Brief

SparseDitto constructs per-matrix GPU kernels for sparse operators (SpMV, SpMM, SpGEMM) via an LLM-driven pipeline: a lightweight additive model ranks established strategies from matrix structural features, an architecture-aware planner proposes candidate designs, and coding/verification agents implement and refine kernels using on-device measurements. Based on the abstract, it delivers geometric-mean speedups of 2.68x on an NVIDIA RTX PRO 6000 (max 146.61x) and 2.79x on an H200 (max 78.5x), and accelerates full-batch GCN training up to 3.39x.

Why it matters

cuSPARSE can exhibit up to a 350x performance gap between CSR and Blocked-ELL for the same SpMM on the same matrix, motivating per-matrix/kernel customization.

Key details

  • SparseDitto is an LLM-based agentic system that builds a GPU kernel per matrix/operator/GPU using a lightweight additive model to rank strategies from matrix structural features, an architecture-aware planner to propose designs, and coding+verification agents that refine implementations with on-device measurements; it supports SpMV, SpMM, and SpGEMM.
  • On an NVIDIA RTX PRO 6000 SparseDitto achieves a geometric-mean 2.68x speedup over cuSPARSE (maximum 146.61x); on an NVIDIA H200 it achieves 2.79x (maximum 78.5x); its generated SpMM kernels speed up full-batch GCN training by up to 3.39x.
Source evidence

Abstract

Sparse matrix kernels are fundamental to scientific computing, graph analytics, and machine learning. Their GPU performance depends strongly on the input sparsity pattern and execution strategy. For the same SpMM on the same matrix, cuSPARSE exhibits a 350x performance gap between CSR and Blocked-ELL. Our study of multiple data formats, specialized systems, and sparse compilers shows that no single implementation consistently dominates across sparsity patterns and operators. This motivates a system that can adapt its representation, execution strategy, and hardware mapping to each workload and target GPU. We present SparseDitto, an LLM-based system that constructs a GPU kernel for each matrix, operator, and target GPU. SparseDitto supports SpMV, SpMM, and SpGEMM within a unified design framework. A lightweight additive model ranks established strategies using structural features of the input matrix. An architecture-aware planner then proposes several candidate designs. Coding and verification agents implement and refine them using measurements from the target GPU. Across three sparse operators and a diverse set of matrices, SparseDitto achieves a geometric-mean speedup of 2.68x over cuSPARSE on an NVIDIA RTX PRO 6000 GPU, with a maximum of 146.61x. On an NVIDIA H200 GPU, it achieves 2.79x, with a maximum of 78.5x. Its generated SpMM kernels also accelerate full-batch GCN training by up to 3.39x.