• 🧠 Intent-Driven Compression (ATA): Uses a Small VLM as an O(1) dynamic router to allocate dense token bandwidth to query-critical moments while rapidly fast-forwarding redundant backgrounds.
• ⚡ Extreme Efficiency: Achieves aggressive dynamic compression (0.5–16 tokens/frame), bypassing the lost-in-the-middle phenomenon without breaking causality.
• 🏆 State-of-the-Art Performance: Our compact Tempo-6B model scores 52.3 on the extreme-long LVBench under a strict 8K visual token budget (53.7 with 12K budget), outperforming proprietary baselines like GPT-4o and Gemini 1.5 Pro.

(Click play to see our interactive UI, dynamic token allocation visualization, and real-time inference)

• [2026.04] 📦 We have released the Intermediate Checkpoints for Stages 0, 1, and 2! You can find them in our Hugging Face Collection.
• [2026.04] 📊 The full Evaluation Pipeline is now open-sourced! Our evaluation scripts, integrated with the standard lmms-eval framework for LVBench, Video-MME, MLVU, and LongVideoBench, are ready to use. Please refer to the Evaluation Section for detailed instructions.
• [2026.04] 📄 Our paper is officially out! You can read it on arXiv and check out our page on Hugging Face Papers.
• [2026.04] 🚀 We have released the Tempo-6B inference code, interactive Gradio UI, and the final checkpoints (Stage 3)!
• [TODO] 🛠️ Training Code: The complete training scripts for all 4 stages will be open-sourced in the following weeks. Stay tuned!

⭐ Tip: Please Watch or Star this repository to keep an eye on our latest updates and code releases!

Create a new conda environment and install all required dependencies:

# Clone our repository
git clone https://github.com/FeiElysia/Tempo.git
cd Tempo

# Create environment
conda create -n tempo python=3.12 -y
conda activate tempo

# Install all packages (PyTorch 2.6.0 + CUDA 12.4)
pip install -r requirements.txt

Since flash-attn installation can be highly environment-dependent, please install it manually using one of the methods below:

# Method 1
pip install flash-attn==2.7.4.post1

# Method 2: Without Build Isolation
pip install flash-attn==2.7.4.post1 --no-build-isolation

# Method 3: If you are unable to build from source, you can directly download and install the pre-built wheel:
wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl
pip install flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl
rm flash_attn*.whl

💡 If you are unable to install flash-attn, you can still run Tempo by disabling it:

1. Set use_flash_attn=False when calling load_pretrained_model.
2. Comment out the line self.config._attn_implementation = "flash_attention_2" in qwen3vl_encoder.py.

Theoretically, the numerical differences should be minimal. We have visually verified that the model produces excellent qualitative results without Flash-Attention. However, please note that we have not rigorously evaluated its impact on the benchmarks we reported in the paper.

To fully support the open-source community and facilitate future research, we have released the weights for our final model alongside the intermediate checkpoints from all 4 stages of our training pipeline.

💡 Note on Token Budgets: Tempo's Adaptive Token Allocation (ATA) is dynamically controlled at inference time. The 4K and 8K budget configurations reported in our paper use the exact same final weights (Stage 3). You simply adjust the budget hyperparameter during inference.

(Note: If you only want to run inference or evaluate our model, simply download the Stage 3 weights. The intermediate checkpoints for Stages 0, 1, and 2 are provided for researchers who wish to reproduce our training pipeline, conduct ablation studies, or perform custom fine-tuning.)

To run the inference script successfully, you need to download two components: our final Tempo-6B weights, and the base Qwen3-VL-2B-Instruct model (for Tempo initialization).

We highly recommend using the huggingface-cli for fast and resumable downloads:

mkdir -p checkpoints

# 1. Download the final Tempo-6B model
huggingface-cli download --resume-download Vision-CAIR/Tempo-6B --local-dir ./checkpoints/Tempo-6B

# 2. Download the base Qwen3-VL model (Required for architecture initialization)
# 💡 Note: To avoid caching Qwen3-VL in the default system drive during inference, 
# you can modify Tempo-6B's `config.json`: change "Qwen/Qwen3-VL-2B-Instruct" to "./checkpoints/Qwen3-VL-2B-Instruct" and run:
huggingface-cli download --resume-download Qwen/Qwen3-VL-2B-Instruct --local-dir ./checkpoints/Qwen3-VL-2B-Instruct

We provide multiple ways to interact with Tempo, from web UI to batch scripts.

To launch the local Gradio application with interactive visualizations of the Token Allocation distribution:

python app.py

Navigate to the generated local or public URL in your browser. Our UI features dynamic token compression visualization and one-click example testing.

⚠️ Note: We use Git LFS to manage the demo videos in this repository. If you encounter a "Video not playable" error, it means the actual video files were not downloaded during git clone (this will not happen if your system has already installed Git LFS).

• Option 1 (Recommended): Install Git LFS (git lfs install) and run git lfs pull inside the repository to fetch the real video files.
• Option 2: Manually download the specific .mp4 files directly from our repository web interface.

Run the default example: We provide a quick-start script to test a pre-configured.

sh ./scripts/infer/infer.sh

Run your own custom video: To test your own videos, call the Python script directly. Make sure to point the --model_path to your downloaded local checkpoint.

python infer.py \
    --model_path "./checkpoints/Tempo-6B" \
    --video_path "/path/to/your/custom_video.mp4" \
    --query "Your detailed question here."

Run all default examples: To sequentially reproduce all the qualitative examples shown on our Project Page, run:

sh ./scripts/infer/infer_all_demos.sh

Run a custom batch: For testing across multiple custom videos, we highly recommend our JSON-based pipeline. Simply edit the test configurations in ./examples/demo_cases.json:

[
  {
    "video_path": "/path/to/custom1.mp4",
    "query": "Question 1"
  },
  {
    "video_path": "/path/to/custom2.mp4",
    "query": "Question 2"
  }
]

Once configured, execute the same batch script to automatically process them all:

sh ./scripts/infer/infer_all_demos.sh

Here are some unedited inference results generated directly by our Tempo-6B model. These examples demonstrate the model's capability in video summarization and complex QA.

Tempo bridges the gap between hour-long videos and bounded LLM context windows by casting long video understanding as an end-to-end, query-aware cross-modal distillation process.

Instead of blindly sampling frames, our pipeline operates in three highly efficient phases:

• 1. Local Compressor (Early Distillation): A Small VLM (SVLM) processes video segments alongside the user query. It uses learnable memory tokens to inherently distill visual semantics, dropping query-irrelevant backgrounds early on.
• 2. Adaptive Token Allocation (ATA): Operating as a training-free, O(1) dynamic router during inference, ATA intercepts zero-shot relevance scores from the SVLM. It allocates dense token bandwidth to query-critical segments while rapidly fast-forwarding redundancies into minimal temporal anchors.
• 3. Global Decoder (Synthesis): The highly compressed, filtered memory tokens are assembled with explicit temporal tags (e.g., <t=2.0s>). A large global LLM then synthesizes this condensed storyline to generate precise answers without suffering from attention dilution.

Tempo achieves state-of-the-art performance on long video benchmarks while using a fraction of the token budget compared to traditional models.

💡 Note: While configured with a theoretical dynamic range of 0.5–16 tokens, Tempo's Adaptive Token Allocation (ATA) operates substantially below the maximum limits in practice (see the actual avg. rows). For the complete leaderboard and metrics, please visit our Project Page.

Our evaluation scripts are organized under ./scripts/eval/.

💡 Note: By default, the scripts are configured for a 4-GPU setup. Before running, please ensure you adjust NUM_GPUS and CUDA_VISIBLE_DEVICES inside the scripts to match your local machine environment.

To reproduce the main results across all benchmarks, run the script for your desired visual token budget:

# For 4K budget
sh ./scripts/eval/4k_budgets/eval_all.sh

# For 8K budget
sh ./scripts/eval/8k_budgets/eval_all.sh

To run evaluation on a specific benchmark (e.g., LVBench under the 4K budget), execute the corresponding script:

sh ./scripts/eval/4k_budgets/eval_lvbench.sh

For a quick sanity check to ensure your environment and model loading are properly configured:

sh ./scripts/eval/eval_debug.sh

The lmms-eval framework only reports the Overall accuracy for Video-MME. To obtain the detailed breakdown across different video lengths (Short, Medium, Long), use our provided script:

python ./scripts/eval/split_videomme.py

⚠️ Before running, please open split_videomme.py and modify the jsonl_file path to point to your recently generated *_samples_videomme.jsonl log file. The overall score calculated by this script matches the lmms-eval output.

We are actively cleaning up and organizing the training codebase for public release.

• Training Scripts: The complete source code and configuration files for all 4 stages of our progressive training curriculum.

Stay tuned! Please watch/star the repository to get notified as soon as the training code drops.

While Tempo provides a strong foundation for long video understanding, it opens up several exciting possibilities for the community. Potential avenues for future research include:

• Routing Post-Training: Enhancing the SVLM's zero-shot routing precision via RL to elicit stronger relevance priors.
• Autoregressive Compression: Exploring reasoning-driven, dynamic length token generation for query-aware segment compression.
• Multi-Turn Efficiency: Implementing hierarchical, on-demand visual extraction to support extremely fast multi-turn dialogue.

📖 For more detailed insights, please refer to the Discussion and Future Works section in our Paper.

🤝 Call for Collaboration: We warmly welcome community contributions! If you are interested in exploring these directions, building upon Tempo, or collaborating on future research, please feel free to reach out to us directly at junjiefei@outlook.com. Let's push the boundaries of long video understanding together!

If you find our work useful for your research and applications, please consider citing our paper:

@article{fei2026small,
  title={Small Vision-Language Models are Smart Compressors for Long Video Understanding},
  author={Fei, Junjie and Chen, Jun and Liu, Zechun and Xiong, Yunyang and Zhou, Chong and Wen, Wei and Han, Junlin and Zhuge, Mingchen and Suri, Saksham and Qian, Qi and others},
  journal={arXiv preprint arXiv:2604.08120},
  year={2026}
}

Junjie Fei, Mingchen Zhuge, Shuming Liu, and Mohamed Elhoseiny were supported by funding from the KAUST Center of Excellence for Generative AI.

We extend our sincere gratitude to the open-source community for their invaluable contributions that made this research possible:

• Evaluation Benchmarks: LVBench, Video-MME, MLVU, and LongVideoBench.
• Codebase Foundations: Our codebase is built upon the excellent architectures of LongVU, VideoChat-Flash, LLaVA, LMMs-Eval.
• Pre-trained Weights: Our models are initialized using the powerful foundational models from Qwen3-VL and Qwen3-LM.
• Project Page: Our website template is adapted from the Nerfies project.

• ⚖️ Framework & Code: The Tempo framework's source code is open-sourced under the Apache-2.0 License to foster research and community development.
• 🚫 Model Weights: The pre-trained model weights and checkpoints are distributed strictly for academic and non-commercial research purposes. Any commercial use is explicitly prohibited without prior written consent.
• 🎬 Video & Data Assets: All visual media (video clips, images) showcased on our project page and within evaluation pipelines are utilized under the doctrine of Fair Use exclusively for non-profit academic research and scientific illustration. We claim no ownership over the original, copyrighted media assets.

Take-down Notice: We deeply respect the intellectual property rights of creators. If you are a copyright holder and believe that any content hosted in this repository or our project page infringes upon your rights, please contact us at junjiefei@outlook.com or open an Issue. We will promptly investigate and remove the identified content upon verification.