This repository is the official implementation of "Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression".

📢 This paper has been accepted at NeurIPS 2025.

To set up the environment and install dependencies:

conda create -n ultradelta_vit python=3.10
conda activate ultradelta_vit

git clone https://github.com/xiaohuiwang000/UltraDelta.git
cd UltraDelta

pip install -r requirements.txt

Please follow the AdaMerging repository for detailed instructions on downloading datasets: 🔗 AdaMerging # Datasets

You can download the fine-tuned checkpoints from the Task Vectors repository:
🔗 task_vectors # checkpoints

The corresponding Google Drive folder is available here:
🔗 task_vectors_checkpoints

UltraDelta/
├── data/  
│   ├── dtd/
│   │   ├── test/
│   │   │   ├── banded/
│   │   │   ├── blotchy/
│   │   │   ├── ...
│   │   ├── ...
│   ├── EuroSAT_splits/
│   ├── gtsrb/
|   ├── ...
│
├── checkpoints/ 
│   ├── ViT-B-32/
│   │   ├── Cars/
│   │   │   ├── finetuned.pt
│   │   ├── DTD/
│   │   │   ├── finetuned.pt
│   │   ├── ...
│   ├── ViT-L-14/
│   │   ├── ...
│
└── ...

To evaluate, simply run:

bash run.sh

• Integrate LLM compression pipeline (e.g., LLaMA, Qwen)
• Integrate NLP models compression pipeline (e.g., T5, RoBERTa)

If you find this work useful, please cite:

@inproceedings{
  wang2025breaking,
  title={Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression},
  author={Xiaohui Wang and Peng Ye and Chenyu Huang and Shenghe Zheng and Bo Zhang and Lei Bai and Wanli Ouyang and Tao Chen},
  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
  year={2025}
}

Our implementation references the following excellent open-source projects — many thanks to their authors:

• Task Arithmetic: https://github.com/mlfoundations/task_vectors
• AdaMerging: https://github.com/EnnengYang/AdaMerging
• EMR-Merging: https://github.com/harveyhuang18/EMR_Merging