Build AI Agents That Understand, Simulate, and Act in the Real World
Quick Start • Features • Arena • Docs • Contributing • Discord
Spatial Agents is an open-source framework for building AI agents with true spatial intelligence—agents that can understand the physical world, simulate environments, and take actions in reality.
┌────────────────────────────────────────────────────────────────────────────────────┐
│ │
│ 🧠 SPATIAL AGENTS │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ UNDERSTAND │ → │ SIMULATE │ → │ ACT │ │
│ │ │ │ │ │ │ │
│ │ 3D Scenes │ │ Physics │ │ Navigation │ │
│ │ Objects │ │ Worlds │ │ Manipulation│ │
│ │ Spatial │ │ Scenarios │ │ Real-world │ │
│ │ Relations │ │ What-ifs │ │ Execution │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────────┐ │
│ │ 🌍 QAI EARTH ENGINE │ │
│ │ Real-world geospatial data • Physics • 3D environments │ │
│ └──────────────────────────────────────────────────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────────────────┘
Current AI is blind to the physical world. Spatial Agents gives AI the ability to:
- See — Understand 3D scenes, object relationships, and spatial layouts
- Think — Reason about physics, predict outcomes, plan paths
- Simulate — Test actions in virtual environments before execution
- Act — Navigate, manipulate, and operate in the real world
Perceive and comprehend spatial environments—3D scene understanding, object recognition, spatial relationship extraction, and depth perception.
Run physics-accurate simulations, test hypothetical scenarios, predict outcomes, and train agents in virtual worlds before real deployment.
Execute real-world actions—navigation, object manipulation, robotic control, and autonomous decision-making in physical environments.
Benchmark and compete AI models in spatial reasoning challenges. Compare performance across navigation, physics, and 3D understanding.
Plug in any LLM/VLM—OpenAI, Anthropic, Google, DeepSeek, Qwen, or your own custom model.
Connect to real-world geospatial data, maps, terrain, and environmental information.
pip install spatial-agentsfrom spatial_agents import SpatialAgent, Environment
from spatial_agents.capabilities import Vision, Navigation, Manipulation
# Create an agent with spatial capabilities
agent = SpatialAgent(
model="claude-sonnet-4-20250514",
capabilities=[Vision, Navigation, Manipulation]
)
# Connect to an environment
env = Environment.from_location("san_francisco", radius_km=5)
# Agent understands, simulates, and acts
observation = agent.perceive(env)
plan = agent.reason("Navigate to the nearest coffee shop avoiding traffic")
result = agent.execute(plan, simulate_first=True)from spatial_agents import SpatialAgent
agent = SpatialAgent(model="gpt-4o")
# Understand a 3D scene from images
scene = agent.understand(
images=["room_view_1.jpg", "room_view_2.jpg"],
query="Where is the laptop relative to the window?"
)
print(scene.objects) # Detected objects with 3D positions
print(scene.relationships) # Spatial relationships between objects
print(scene.answer) # "The laptop is 2m left of the window, on the desk"from spatial_agents import SpatialAgent, Simulation
agent = SpatialAgent(model="claude-opus-4-5-20250514")
# Plan a complex action
action = agent.plan("Move the robotic arm to pick up the red cube")
# Simulate first to verify safety
sim = Simulation(physics=True)
outcome = sim.run(action, steps=100)
if outcome.success and outcome.collision_free:
agent.execute(action) # Safe to execute in real worldTest and compare spatial intelligence across different AI models.
from spatial_agents import Arena, Agent
arena = Arena(challenge="urban_navigation")
results = arena.compete([
Agent(model="claude-opus-4-5-20250514", name="Claude"),
Agent(model="gpt-4o", name="GPT-4o"),
Agent(model="gemini-ultra", name="Gemini"),
])
print(results.leaderboard())| Rank | Model | Navigation | Object Reasoning | Physics | 3D Understanding | Overall |
|---|---|---|---|---|---|---|
| 🥇 | Claude Opus 4.5 | 94.2 | 91.8 | 89.5 | 92.1 | 91.9 |
| 🥈 | GPT-4o | 92.1 | 90.3 | 88.7 | 89.4 | 90.1 |
| 🥉 | Gemini Ultra | 89.8 | 88.9 | 90.2 | 87.6 | 89.1 |
| 4 | DeepSeek V3 | 87.2 | 86.5 | 85.8 | 84.9 | 86.1 |
| 5 | Qwen 2.5 | 85.4 | 84.2 | 83.1 | 82.8 | 83.9 |
┌─────────────────────────────────────────────────────────────────────────────┐
│ SPATIAL AGENTS │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Perception │ │ Simulation │ │ Action │ │
│ │ │ │ │ │ │ │
│ │ • 3D Vision │ │ • Physics │ │ • Navigation │ │
│ │ • Depth │ │ • Scenarios │ │ • Manipulation │ │
│ │ • Object Det. │ │ • Prediction │ │ • Control │ │
│ │ • Scene Graph │ │ • Training │ │ • Execution │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Models │ │ Arena │ │ Benchmarks │ │
│ │ │ │ │ │ │ │
│ │ • OpenAI │ │ • Competitions │ │ • SpatialIQ │ │
│ │ • Anthropic │ │ • Challenges │ │ • NavScore │ │
│ │ • Google │ │ • Leaderboards │ │ • PhysicsBench │ │
│ │ • DeepSeek/Qwen │ │ • Evaluation │ │ • 3D-Reason │ │
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ QAI EARTH ENGINE │ │
│ │ Real-world geospatial data • Physics • 3D environments │ │
│ │ Terrain • Buildings • Sensor simulation │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
3D scene understanding, object detection, depth estimation, spatial relationship extraction, multi-view reconstruction.
Path planning, obstacle avoidance, SLAM, multi-destination routing, real-world wayfinding.
Spatial logic, physics prediction, object permanence, "what's behind X?", relative positioning.
Force understanding, trajectory prediction, stability analysis, collision detection, material properties.
Robotic manipulation, motion planning, grasping, assembly, human-robot interaction.
Real-world maps, satellite imagery, terrain analysis, urban environments, global positioning.
from spatial_agents.benchmarks import SpatialIQBenchmark
benchmark = SpatialIQBenchmark()
results = benchmark.evaluate(your_model, split="test")
print(results.to_latex())
print(results.statistical_analysis())Cite our work:
@software{spatial_agents_2025,
title={Spatial Agents: AI Agents That Understand, Simulate, and Act in the Real World},
author={Qian, Dr. and QAI Lab},
year={2025},
url={https://github.com/qai-lab/spatial-agents}
}We welcome contributions! Whether you're:
- 🔍 Adding perception — New vision models, depth estimators
- 🌐 Building simulations — Physics engines, virtual environments
- 🎯 Creating actions — Robotics integrations, control systems
- 🏟️ Designing challenges — New arena benchmarks
- 📖 Improving docs — Help others build spatial agents
Check out our Contributing Guide to get started.
git clone https://github.com/qai-lab/spatial-agents.git
cd spatial-agents
pip install -e ".[dev]"
pytest
- 💬 Discord: discord.gg/qai-lab
- 💼 LinkedIn: QAI Lab
- 🐦 X/Twitter: @qai_lab
- 📧 Issues: GitHub Issues
Built with ❤️ by Dr. Qian and QAI Lab
⭐ Star us on GitHub — it helps the project grow!
AI agents that see, think, simulate, and act in the physical world