Turn your Ray-Ban Meta glasses into a real-time AI assistant.

Your AI sees what you see. Talks back instantly. Remembers everything.
Open-source iOS app + backend runtime for agents that live in the real world — not just in chat.

_{Top 10 at the Mistral Worldwide Hackathon 2026 · Winner of Giant Ventures' "Future Unicorn" prize}

Look at something → ask a question → get an answer instantly
Walk through the world → your AI remembers what mattered
Talk naturally → no screen, no typing, no app switching

Get started in minutes

pip install portworld

Imagine walking through a city with an AI that can:

• Identify buildings, objects, and landmarks in real time
• Answer follow-up questions by voice
• Remember what you saw earlier in the day
• Act through tools, APIs, and external agents

"What is this building?" "Where should I eat nearby?" "What was the name of that church we passed earlier?"

Port:World is a runtime for AI agents in the physical world.

Instead of building chatbots, you build agents that:

• See through a camera
• Hear through a microphone
• Speak back in real time
• Remember across sessions
• Act through tools

This is not an app — it's a platform. You bring the agent logic. Port:World handles streaming, model routing, memory, and the glasses.

• AI engineers building agent systems that go beyond text
• Developers exploring wearable + AI integration
• Builders who want to ship real-world AI — not another chat wrapper

Time to first interaction: ~5 minutes

pip install portworld
portworld init
portworld doctor --target local

git clone https://github.com/portworld/PortWorld.git && cd PortWorld
cp backend/.env.example backend/.env
# Set at least one provider key — see "Minimum config" below
docker compose up --build

Verify:

curl http://127.0.0.1:8080/livez
# → {"status":"ok","service":"portworld-backend"}

open IOS/PortWorld.xcodeproj

1. Let Xcode resolve Swift Package dependencies
2. Build the PortWorld scheme
3. Enter your backend URL in Settings and validate the connection

The iOS app exposes a Siri shortcut that opens PortWorld and attempts to start an assistant session.

Available phrases:

• Start PortWorld session
• Start assistant in PortWorld
• Launch PortWorld assistant session

If onboarding is incomplete or backend or glasses readiness is blocked, Siri still opens the app but the session will not start until those requirements are met.

If your OpenClaw gateway runs on a different VM/VPS/cloud than PortWorld, use the openclaw-gateway-bridge agent skill for provider-agnostic setup and validation:

npx skills add portworld/PortWorld --skill openclaw-gateway-bridge -a codex -y

It guides secure endpoint exposure, auth wiring, and PortWorld OPENCLAW_* configuration end-to-end.

You only need one API key to start:

Vision, search, memory consolidation, and tool integrations are all off by default — enable them as you need them.

┌──────────────┐       WebSocket (audio + control)       ┌──────────────────┐
│              │ ◄──────────────────────────────────────► │                  │
│  Ray-Ban     │                                          │   FastAPI        │
│  Meta        │       HTTP (vision frames)               │   Backend        │
│  Glasses     │ ──────────────────────────────────────►  │                  │
│              │                                          │   ┌────────────┐ │
│  ↕ DAT SDK   │                                          │   │ Realtime   │ │
│              │                                          │   │ Bridge     │─┼──► OpenAI / Gemini
│  iPhone      │                                          │   ├────────────┤ │
│  (bridge)    │                                          │   │ Vision     │─┼──► Mistral / Claude / GPT-4o / ...
│              │                                          │   ├────────────┤ │
│              │                                          │   │ Memory     │ │
│              │                                          │   ├────────────┤ │
│              │                                          │   │ Tools      │─┼──► Web search, MCP, OpenClaw, ...
│              │                                          │   └────────────┘ │
└──────────────┘                                          └──────────────────┘

Glasses capture audio and camera frames via Meta's DAT SDK. iPhone bridges glasses I/O to the backend over WebSocket (audio) and HTTP (vision). Backend routes audio to a realtime AI provider, processes vision frames through pluggable analyzers, manages persistent memory, and executes tools during the conversation.

portworld providers list          # see all available providers
portworld providers show <id>     # inspect a specific provider

Port:World is designed to be extended. Here's how developers plug into the system:

Tools are async functions the AI can call mid-conversation. Register a definition + executor in the tool catalog:

registry.register(
    ToolDefinition(
        name="my_tool",
        description="Does something useful",
        parameters={"type": "object", "properties": { ... }},
    ),
    executor=my_tool_executor,
)

Implement a vision analyzer and register it in the vision factory. Your analyzer receives camera frames and returns semantic descriptions that feed into the agent's memory.

Implement the realtime bridge interface and register it in the provider registry. The bridge handles upstream audio streaming and tool dispatch for any new model API.

The backend supports Model Context Protocol (MCP) extensions. Drop a server config into the extensions system and expose new capabilities to the agent without touching core code.

Use the OpenClaw delegation layer to offload long-running or tool-heavy tasks to external agent runtimes, while Port:World stays the live conversational orchestrator.

See backend/README.md for the full API and extension reference.

docker compose up --build

portworld deploy gcp-cloud-run       --project <id> --region <region>
portworld deploy aws-ecs-fargate     --region <region>
portworld deploy azure-container-apps --subscription <sub> --resource-group <rg> --region <region>

See portworld_cli/README.md for readiness checks, log streaming, and redeployment.

Port:World is evolving from a hackathon winner into a full wearable AI platform:

• Agentic delegation — OpenClaw integration for heavy multi-step tasks
• Richer memory — identity, routines, social graph, preferences, with confidence tracking
• Passive context — ambient scene understanding even outside active conversations
• Proactive assistance — timely suggestions earned through context quality and user trust
• Siri / App Shortcuts — launch a session with your voice, no app interaction needed
• Android support — bring the same glasses-first experience to Android
• Multi-agent coordination — orchestrate multiple specialized agents in parallel

Full roadmap: docs/roadmap/AGENTIC_PERSONAL_ASSISTANT_ROADMAP.md

Port:World is in its first stable release phase. Core surfaces are release-ready.

• Stable: backend self-hosting, CLI bootstrap/deploy, iOS app with Meta glasses
• Shipping: first public PyPI + GHCR releases with v0.2.x
• Hardening: managed cloud defaults, operator docs, production security posture

• Provider API keys required — no keyless demo mode
• AWS/Azure one-click deploys use public DB access by default (tighten before production)
• Full glasses features require Meta hardware + Meta AI app
• Xcode test schemes are not yet maintained

Contributions are welcome. Please read CONTRIBUTING.md before opening a PR.

• Bug reports & features: open an issue
• Security: SECURITY.md
• Code of conduct: CODE_OF_CONDUCT.md

Built during the Mistral Worldwide Hackathon 2026 by Pierre Haas, Vassili de Rosen, and Arman Artola.

MIT — see LICENSE.