Latency-aware Robust LLM Inference Workload Allocation under Precision-Dependent Uncertainty

Efficient optimization framework for allocating LLM inference workloads across heterogeneous GPU resources while handling uncertainty in processing delays and error rates. Supports tensor parallelism, decision-dependent uncertainty, and multi-precision GPU configurations.

• ✅ Robust Optimization with decision-dependent uncertainty sets
• ✅ Tensor Parallelism support (TP degrees: 1, 2, 4, 8)
• ✅ Multi-Precision GPU configurations (FP16, INT8, INT4)
• ✅ 6 Query Types: Summarization, Code Gen, Translation, Math, Image Gen, Video Gen
• ✅ 6 LLM Models: Llama-3.2 (1B-70B) with vision support
• ✅ 10 GPU Tiers: RTX 4090, A6000, A100, H100 variants
• ✅ Comprehensive Sensitivity Analysis tools

# Install dependencies
pip install -r requirements.txt

Requirements: Python 3.8+, Gurobi (Academic License)

from RODIU_LLM import DataGenerator, LLMInferenceOptimizer

# Generate problem instance
generator = DataGenerator(seed=42)
data = generator.generate()

# Build and solve optimization model
optimizer = LLMInferenceOptimizer(data)
solution = optimizer.build_and_solve_optimization_problem(
    time_limit=300,  # seconds
    mip_gap=0.01     # 1% optimality tolerance
)

# Display results
optimizer.display_results(solution)

# Analyze GPU cost vs budget tradeoffs
python sensitivity_analysis_cost_budget.py

# Analyze delay vs error threshold impacts
python sensitivity_analysis_delay_error_threshold.py

# Analyze GPU cost vs error rate sensitivity
python sensitivity_analysis_gpu_cost_error.py

# Analyze memory capacity vs error rate
python sensitivity_analysis_memory_error.py

Minimize: Total Cost = GPU Rental + Storage + Robust Delay Penalty + Unmet Demand Penalty

Subject to:

• Supply-demand balance
• Budget constraints
• Memory capacity (with tensor parallelism)
• Compute capacity
• Storage limits
• Robust delay constraints (worst-case guarantees)
• Robust error rate constraints (precision-dependent)
• Tensor parallelism selection
• Logical consistency

Actual Delay = d_bar[i,j,k] + d_hat[i,j,k] × ξ_d
Actual Error = e_bar[i,j,k] + e_hat[i,j,k] × ξ_e

Where: ξ ∈ [0,1], and Σ ξ ≤ Γ (uncertainty budget)

Our model captures:

• GPU tier impact on delays: Higher compute power (H100 vs A6000) reduces delay uncertainty
• Precision impact on errors: Quantization (INT4/INT8 vs FP16) increases error uncertainty
• Model-GPU pairing effects: Optimal pairings reduce both delay and error variability

H100 provides ~36× speedup over A6000, while INT8 quantization offers 1.5× additional acceleration

Results are stored in sensitivity_results/ with visualizations:

• Heatmaps: Cost, GPU allocation, performance metrics
• Trend plots: Parameter sensitivity curves
• Breakdown charts: Cost component stacked bars
• TP distribution: Tensor parallelism selection patterns

ICC conference/
├── 📦 Core Models
│   ├── RODIU_LLM.py                          ⭐ Main robust optimization
│   ├── LLM_DET.py                            Deterministic baseline
│   └── parameter_setup.py                    Parameter generation
│
├── 🔬 Sensitivity Analysis
│   ├── sensitivity_analysis_cost_budget.py
│   ├── sensitivity_analysis_delay_error_threshold.py
│   ├── sensitivity_analysis_gpu_cost_error.py
│   └── sensitivity_analysis_memory_error.py
│
├── 📓 Notebooks
│   └── Experiment_RO.ipynb                   Robust vs Deterministic
│
├── 📁 Results
│   └── plot/*.jpg                            Publication figures
│
└── 📄 Documentation
    ├── README.md                             This file
    ├── PROJECT_STRUCTURE.md                  File organization & flows
    ├── DATA_SOURCES_AND_PARAMETERS.md        Parameter generation details

See DATA_SOURCES_AND_PARAMETERS.md for complete parameter details and data sources.

All parameters are generated with realistic distributions based on:

• GPU Specifications: NVIDIA official technical whitepapers
• Rental Costs: vast.ai, Lambda Labs, RunPod marketplace (Q4 2024)
• Model Sizes: Meta AI Llama model cards
• Workload Patterns: "Characterizing LLM Workloads" (Patel et al., 2024)
• Quantization Impact: "LLM.int8()" (Dettmers et al., 2022)
• SLA Pricing: OpenAI, Anthropic API documentation (2024)

Processing Delay (ms/token):

d[i,j,k] = base_delay[i] × model_multiplier[j] × (reference_power / P_gpu[k])

Error Rate (fraction):

e[i,j,k] = (base_error[i] / model_capacity[j]) × precision_factor[k]

Where:
  precision_factor[FP16] = 1.0
  precision_factor[INT8] = 1.15  # +15% quantization error
  precision_factor[INT4] = 1.35  # +35% quantization error

Uncertainty Deviations:

d_hat[i,j,k] ~ Uniform(0.10, 0.25) × d_bar[i,j,k]  # 10-25% of nominal
e_hat[i,j,k] ~ Uniform(0.10, 0.25) × e_bar[i,j,k]

1. Cost-Budget Sensitivity (sensitivity_analysis_cost_budget.py)

   • Parameters: GPU rental cost × Budget threshold
   • Grid: 6 × 5 = 30 scenarios
   • Insights: Budget utilization, cost breakdown, GPU allocation patterns

2. Delay-Error Threshold Sensitivity (sensitivity_analysis_delay_error_threshold.py)

   • Parameters: Delay threshold × Error threshold
   • Grid: 5 × 11 = 55 scenarios
   • Insights: Constraint utilization, QoS tradeoffs, TP distribution

3. GPU Cost-Error Sensitivity (sensitivity_analysis_gpu_cost_error.py)

   • Parameters: GPU cost scaling × Error threshold
   • Grid: 11 × 11 = 121 scenarios
   • Insights: Cost-accuracy tradeoffs, GPU tier selection

4. Memory-Error Sensitivity (sensitivity_analysis_memory_error.py)

   • Parameters: Memory capacity × Error threshold
   • Insights: Resource-quality tradeoffs

$ python sensitivity_analysis_cost_budget.py

SENSITIVITY ANALYSIS: GPU RENTAL COST & BUDGET
================================================================================
Running 30 scenarios...
[1/30] p_c_scale=0.4, delta_scale=0.3
  ✓ OPTIMAL: Total Cost = $1,247.89, Gap = 0.0032

Results saved to: sensitivity_results/sensitivity_cost_budget_20251124_153042.csv

Generating visualizations...
  Saved: heatmaps_cost_budget_20251124_153042.png
  Saved: cost_trends_cost_budget_20251124_153042.png
  Saved: cost_breakdown_cost_budget_20251124_153042.png
  Saved: budget_analysis_20251124_153042.png

# More conservative (higher robustness)
data.Gamma_d = 30  # Allow 30/60 configurations to reach worst-case delay
data.Gamma_e = 30  # Allow 30/60 configurations to reach worst-case error

# Less conservative (lower cost)
data.Gamma_d = 5   # Only 5/60 worst-case scenarios
data.Gamma_e = 5

# Adjust arrival rates
data.lambda_i = np.array([150, 120, 200, 80, 50, 40])  # Higher traffic

# Relax QoS constraints
data.Delta_i *= 1.5  # +50% delay tolerance
data.epsilon *= 1.2  # +20% error tolerance

# Scale budget
data.delta = 5000

solution = optimizer.build_and_solve_optimization_problem(
    time_limit=600,    # 10 minutes (default: 300s)
    mip_gap=0.005      # 0.5% optimality (default: 0.01)
)

This project is for academic research purposes only.

Gurobi License: Academic license required. For commercial use, obtain a commercial Gurobi license.

⭐ Star this repository if you find it helpful!