Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference

Tianxiang Xia 1
Kai Lion 1
Niao He 1
Michael Muehlebach 2,3
Melanie Zeilinger 1
Hao Ma 4 (corresponding author)
1ETH Zürich, Zürich 2Max Planck Institute for Intelligent Systems, Tübingen 3Vienna University of Economics and Business, Vienna 4The University of Hong Kong
{xiatia, mzeilinger}@ethz.ch, {kai.lion, niao.he}@inf.ethz.ch, michael.muehlebach@wu.ac.at, haoma@hku.hk
Paper Hugging Face GitHub

Rank ratio target: 15%. Sparsity target: 95%. In the example, S-SALAAD is 1.73 times faster and reduces peak VRAM from 3,713 MB to 1,961 MB compared with dense inference.

Abstract

Large models increasingly need to run on the device itself. Unreliable connectivity, communication delays and privacy requirements make the cloud a poor fit for many applications, so models must fit into limited memory and still run fast on local hardware.

Fewer parameters do not automatically mean faster inference. Learning sparse-plus-low-rank weights lets a single checkpoint be compressed to any memory budget. But when the sparse part is unstructured, indexing overhead and irregular memory access can erase the expected speedup.

S-SALAAD learns sparsity that hardware can actually use. During training, every weight matrix is learned as a low-rank part plus a structured sparse part, such as blocks, rows, columns or N ⁣: ⁣MN\!:\!M patterns. Budget-aware compression and dedicated CPU and GPU kernels then turn that structure into real savings, from one checkpoint across all memory budgets.

The result: elastic, smaller and faster models. On Llama2 and Qwen3, in both pretraining and distillation, S-SALAAD advances the Pareto frontier in perplexity, memory and latency on multiple hardware platforms.

Qwen3-1.7B elastic compression

S-SALAAD runs up to 1.62× faster on GPU and 2.20× faster on CPU than dense inference. Perplexity versus prefill throughput. Marker size represents peak memory; labels report GB1.

Qwen3-1.7B perplexity versus negative throughput on a Quadro RTX 6000. The leftmost S-SALAAD point is 1.62 times faster than dense inference; labels show peak GPU memory in GB.
Quadro RTX 6000 · batch 32 × sequence length 512 · FP16.
Qwen3-1.7B perplexity versus negative throughput on four AMD EPYC 7662 CPUs. The leftmost S-SALAAD point is 2.20 times faster than dense inference; labels show peak RAM in GB.
4× AMD EPYC 7662 · batch 8 × sequence length 64 · FP32.

Citation

@misc{xia2026structured,
  title  = {Learning Structured Sparsity for Large Models: Bridging Model Compressibility and Efficient Inference},
  author = {Xia, Tianxiang and Lion, Kai and He, Niao and Muehlebach, Michael and Zeilinger, Melanie and Ma, Hao},
  year   = {2026},
  note   = {Preprint}
}