Orchestrated Reasoning for Accurate Calibrated Likelihood Estimation

ORACLE is an advanced AI-powered forecasting system that generates well-calibrated probability estimates for future events. It uses a multi-layered pipeline with multiple AI agents that research, deliberate, debate, and synthesize their estimates to produce accurate and reliable forecasts.

• 5-Layer Forecasting Pipeline: Systematic approach combining research, estimation, debate, risk analysis, and calibration
• Multi-Agent Architecture: 8 diverse AI agents with different analytical perspectives
• Real-Time Web Research: Perplexity and OpenAI deep research integration for up-to-date information
• Adversarial Debate: Agents argue opposing positions to stress-test forecasts
• Black Swan Detection: Identifies low-probability, high-impact tail risks (optional)
• Calibrated Confidence Intervals: Statistical methods for reliable uncertainty quantification
• Multiple Model Support: Choose from Claude, GPT-4, Gemini, DeepSeek, and more via OpenRouter
• Modern Web Interface: Real-time progress tracking, interactive visualizations, dark mode

• Node.js 18+
• npm or yarn
• OpenRouter API key (get one at openrouter.ai)

# Clone the repository
git clone <repository-url>
cd oracle

# Install dependencies
npm install

# Set up environment variables
cp .env.example .env.local
# Edit .env.local and add your OPENROUTER_API_KEY

# Initialize the database
npx prisma db push

# Start the development server
npm run dev

Open http://localhost:3000 to use the application.

Create a .env.local file with:

# Required
OPENROUTER_API_KEY=your_openrouter_api_key_here

# Optional
DATABASE_URL="file:./dev.db"
DEBUG=false

ORACLE processes forecasting questions through 5 specialized layers:

• Parses the forecasting question into key components
• Identifies relevant reference classes and historical precedents
• Conducts web research to gather current information
• Synthesizes a base rate estimate from historical data

8 diverse AI agents independently analyze the question:

• Base Analyst: Neutral, balanced analysis
• Skeptical Analyst: Focuses on why predictions fail
• Constructive Analyst: Focuses on theories of change
• Historical Analyst: Heavy emphasis on base rates
• Quantitative Analyst: Explicit probability calculations
• Plus 3 additional specialized perspectives

• Agents are paired to argue opposing positions
• Multi-round debates stress-test assumptions
• Convergence analysis measures agreement
• Post-debate estimates incorporate new insights

• Generates tail risk scenarios
• Researches each scenario for plausibility
• Calculates aggregate tail risk adjustments
• Identifies high-impact scenarios that could invalidate forecasts

• Applies temperature scaling for calibration
• Uses extremization based on agent agreement
• Generates final probability distribution
• Produces confidence intervals (50% and 90%)
• Creates quality metrics and methodology summary

oracle/
├── app/                    # Next.js App Router pages
│   ├── api/               # API routes
│   │   └── forecast/      # Forecast CRUD endpoints
│   ├── forecast/[id]/     # Forecast detail page
│   ├── history/           # Forecast history page
│   ├── guide/             # User guide page
│   ├── methodology/       # Methodology documentation
│   └── settings/          # User settings page
├── components/            # React components
│   ├── forecast/          # Forecast-specific components
│   ├── ui/                # Reusable UI components
│   ├── layout/            # Layout components
│   └── settings/          # Settings components
├── lib/                   # Core libraries
│   ├── oracle/            # Forecasting pipeline
│   │   ├── pipeline.ts    # Main orchestration
│   │   ├── layer1-decomposition.ts
│   │   ├── layer2-swarm.ts
│   │   ├── layer3-debate.ts
│   │   ├── layer4-blackswan.ts
│   │   ├── layer5-calibration.ts
│   │   ├── prompts.ts     # AI prompts
│   │   ├── config.ts      # Runtime configuration
│   │   ├── cache.ts       # Response caching
│   │   └── error-recovery.ts
│   ├── openrouter.ts      # OpenRouter API client
│   ├── models.ts          # Model configurations
│   ├── types.ts           # TypeScript types
│   ├── validations.ts     # Zod schemas
│   ├── db.ts              # Database client
│   └── store.ts           # Zustand store
├── prisma/                # Database schema
│   └── schema.prisma
├── hooks/                 # Custom React hooks
└── docs/                  # Documentation

POST /api/forecast
Content-Type: application/json

{
  "statusQuo": "Current state description (50-5000 chars)",
  "resolutionConditions": "How the question resolves (20-2000 chars)",
  "webResearchModel": "perplexity/sonar-reasoning-pro",
  "reasoningModel": "google/gemini-3-flash-preview",
  "enableBlackSwan": true
}

GET /api/forecast/{id}

POST /api/forecast/{id}/run

GET /api/forecast?limit=10&offset=0&status=COMPLETED

DELETE /api/forecast/{id}

See docs/API.md for complete API documentation.

The pipeline can be configured via lib/oracle/config.ts:

// Search limits
layer1MaxSearches: 24,      // Max searches in Layer 1
layer4MaxSearches: 40,      // Max searches in Layer 4

// Caching
enableWebSearchCache: true,
enableLLMCache: true,

// Timeouts
webSearchTimeoutMs: 30000,
llmCallTimeoutMs: 120000,
forecastTimeoutMs: 600000,

// Retry configuration
maxRetries: 3,
retryBaseDelayMs: 1000,

import { applyFastPreset, applyThoroughPreset, applyBalancedPreset } from '@/lib/oracle/config';

applyFastPreset();      // Faster, fewer searches
applyThoroughPreset();  // More comprehensive, slower
applyBalancedPreset();  // Default configuration

npm run dev          # Start development server
npm run build        # Build for production
npm run start        # Start production server
npm run lint         # Run ESLint
npm run db:push      # Push schema to database
npm run db:studio    # Open Prisma Studio
npm run db:generate  # Generate Prisma client

ORACLE uses SQLite by default with Prisma ORM. To use PostgreSQL:

1. Update prisma/schema.prisma:

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

2. Update .env.local:

DATABASE_URL="postgresql://user:password@localhost:5432/oracle"

3. Run migrations:

npx prisma migrate dev

• Specific, time-bound, and objectively verifiable
• Clear resolution criteria
• Sufficient context for research

Examples:

• "Will the Federal Reserve cut interest rates before June 2025?"
• "Will SpaceX successfully launch Starship to orbit by Q2 2025?"

• Vague questions without timeframes
• Subjective or opinion-based questions
• Questions that can't be verified

Examples to avoid:

• "Will the stock market go up?" (no timeframe)
• "Is AI good for society?" (subjective)

• Typical forecast time: 3-8 minutes depending on models
• Token usage: ~50,000-100,000 tokens per forecast
• Caching: Web searches and LLM calls are cached to reduce costs
• Black Swan toggle: Disable to reduce time by ~20%

"OPENROUTER_API_KEY not set"

• Ensure .env.local exists with your API key
• Restart the development server after adding the key

"Forecast stuck in RUNNING"

• Check server logs for errors
• The forecast may have timed out (10 min limit)
• Delete and retry the forecast

"Rate limit exceeded"

• OpenRouter has rate limits per API key
• Wait a few minutes before retrying
• Consider using a paid plan for higher limits

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

This project is licensed under the MIT License - see the LICENSE file for details.

• Built with Next.js and React
• AI models via OpenRouter
• UI components from shadcn/ui
• Inspired by forecasting research from Metaculus and Good Judgment Project