Skip to content

Conversation

@SamMorrowDrums
Copy link
Collaborator

⚠️ Experimental

This is an experiment exploring bitmap-based indexing for efficient tool filtering. Not intended for immediate merge - just capturing the approach for discussion.

Problem

In stateless server patterns, we rebuild tool lists per-request. With ~130 tools and multiple filter dimensions (toolsets, read-only, feature flags), filtering becomes O(n × filters).

Approach

Pre-compute bitmaps at startup, then use bitwise operations for O(1) filtering:

// Build once at startup
index := BuildToolIndex(allTools)

// Per-request: ~78ns, 0 allocations
result := index.Query(QueryConfig{
    Toolsets: []string{"repos", "issues"},
    ReadOnly: true,
})

// Materialize only when needed: ~1µs for 130 tools
tools := index.Materialize(ctx, result)

What's included

  1. bitmap.go - ToolBitmap type (256-bit, 4×uint64) with Set/And/Or/Iterate
  2. tool_index.go - Pre-computed index with Query() and Materialize()
  3. tool_variants.go - Simple override mechanism for tools with variant definitions

Benchmarks

Query (130 tools, 2 toolsets):     78ns    0 allocs
Materialize (full list):        1114ns    3 allocs
ToolOverrides.ApplyToTools:    ~18µs    ~130 allocs (condition funcs)

Key design decisions

  • 256-tool limit via fixed bitmap size (expandable if needed)
  • Lazy allocation in ApplyToTools - no allocs when no overrides match
  • Pointer returns from Materialize to avoid struct copies
  • Simple ToolOverrides instead of full VariantIndex (only 2 tools need variants)

Open questions

  • Is this level of optimization needed for the local server?
  • How does this integrate with the remote server's caching?
  • Should we expose ToolIndex publicly or keep it internal?

This implements a high-performance bitmap index for tool filtering:

- ToolBitmap: 256-bit bitmap (4 uint64s) with O(1) set operations
- ToolIndex: Pre-computed index for toolset, read-only, and feature flag filtering
- Query: Returns guaranteed tools + those needing dynamic checks
- Materialize: Only runs dynamic Enabled() checks on survivors

Performance benchmarks:
- Query (2 toolsets): 78 ns/op, 0 allocs
- Query (all toolsets): 44 ns/op, 0 allocs
- Bitmap OR/AND: <0.3 ns/op
- Full query + materialize (130 tools): 5.4 µs/op, 4 allocs

Key insight: Static filters (toolset, read-only, feature flags) can be
pre-computed as bitmaps. Dynamic Enabled() checks are only run on tools
that survive static filtering, avoiding wasteful checks on tools that
would be filtered out anyway.
Adds two new methods to ToolIndex:

- UniqueFeatureFlags(): Returns all unique feature flags used by tools
- QueryWithFeatureChecker(): Queries with automatic flag checking

Key optimization: Instead of checking the same feature flag N times
for N tools that use it, each unique flag is checked exactly once.

For example, if 50 tools use 3 unique flags, this reduces feature flag
service calls from O(50) to O(3).

Test demonstrates: 50 tools using 3 flags results in exactly 3 checks.
Change Materialize() to return []*ServerTool instead of []ServerTool.
This avoids copying the entire ServerTool struct (which includes the
JSON schema) for each tool in the result.

Benchmark improvement:
- Materialize: 6199ns → 1114ns (5.6x faster)
- Memory: 12584B → 472B (27x less)
- Allocations: 4 → 3

The returned pointers reference the cached tools in the index.
Callers must not modify them.
Provides a lightweight mechanism for tools that have different definitions
based on runtime conditions (e.g., feature flags, capabilities).

Designed for the small number of tools (~2) that need variant handling:
- ToolOverride: condition + replacement definition
- ToolOverrides: map[string]ToolOverride with Apply() and ApplyToTools()
- Lazy allocation: no allocs when no overrides match

This is intentionally simple - only 60 lines - because most tools don't
need variants and the full VariantIndex would be overkill.
@SamMorrowDrums
Copy link
Collaborator Author

Superseded by a cleaner implementation that encapsulates bitmask complexity behind a composable EnableCondition API. See upcoming PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants