Semantic KV cache reuse for LLM inference engines.

SemBlend extends exact-prefix KV caching (LMCache, vLLM prefix cache, SGLang RadixAttention) with semantic donor discovery. When a new prompt is semantically similar to a prior one but lexically different — different instruction wording, sentence ordering, or template fields — SemBlend finds and reuses the cached KV tensors, eliminating redundant prefill computation.

Without SemBlend:  vLLM + LMCache  →  0% hit  →  full prefill every request
With SemBlend:     vLLM + LMCache  →  30–88% hit  →  reuse KV from similar past requests

SemBlend runs ~8ms of in-process MiniLM embedding + cosine search on every request. On a miss it adds negligible overhead. On a hit it replaces a 2–17 second prefill with sub-second KV retrieval.

Empirical results on A10G GPU, Qwen2.5-7B-AWQ + vLLM 0.14.1 + LMCache.

Hit TTFT is consistently ~800ms regardless of context length — it's bounded by KV retrieval, not prefill.

Miss overhead grows with context length (5–212ms at ~20 donors in store). Break-even: P_hit > 5% at 8K is net-positive.

Hit rate increases with cosine similarity: 17% at 0.50–0.60 → 60% at 0.90–1.00.

Speedups measured over cold vLLM+LMCache baseline (0% hit on semantically different requests).

Multi-turn conversations naturally reuse the same prefix → near-perfect hit rates without any workload tuning.

Each sample pair uses the same document and question but different instruction phrasings, shifting all chunk boundaries so exact-prefix caching gets 0% hits.

Baseline = SGLang vanilla (FP16, RadixCache only). SemBlend achieves 4x higher overall hit rate.

SemBlend injects donor KV with RoPE position correction. Output quality impact on high-hit workloads:

PPL ratio ≤ 1.025 on most datasets; elevated ratios (≥1.1) on low-hit runs are due to miss-run variance, not KV injection.

# Core (CPU-only: numpy + rapidfuzz)
pip install semblend

# With vLLM integration
pip install semblend[vllm]

# With SGLang integration
pip install semblend[sglang]

# With sentence-transformers embedder
pip install semblend[embedder]

vLLM integrates via LMCache's KVConnectorBase_V1 — a first-class public API. No patching required.

pip install semblend[vllm] vllm lmcache

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
  --kv-transfer-config '{
    "kv_connector": "SemBlendConnectorV1",
    "kv_connector_module_path": "semblend.integration.vllm.connector_v1",
    "kv_role": "kv_both"
  }'

Configure via environment variables:

SGLang integrates via a RadixCache patch applied at startup. This is necessary because SGLang's RadixCache.match_prefix does not currently have a hook for semantic fallback lookup — a PR to SGLang is in progress to add a first-class SemanticPrefixProvider interface (analogous to LMCache PR #2803).

pip install semblend[sglang] sglang

# Option 1: CLI launcher — applies the RadixCache patch automatically
semblend-sglang --model-path Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 --port 8000

# Option 2: Programmatic — call before SGLang initializes
from semblend.integration.sglang.radix_patcher import patch_radix_cache
patch_radix_cache()
import sglang as sgl
# ... start server ...

Request → MiniLM Embed (5ms) → Cosine Search (1ms) → Align (1ms) → Inject KV
              ↓                       ↓                    ↓
         384-dim vector        Find similar donor    Match chunk boundaries
                                in donor store       via MD5 hash alignment

1. Embed: Compute a 384-dim MiniLM-L6-v2 embedding (sliding-window for long docs)
2. Search: Brute-force cosine similarity against the donor store (<1ms at 1K donors)
3. Align: MD5 chunk hashing at 256-token boundaries finds reusable KV chunks
4. Inject: Replace target token IDs with donor token IDs — LMCache/RadixCache finds cached KV

SemBlend is most effective for workloads where prompts share a large common context:

• Document Q&A / RAG: Same retrieved documents, different questions
• Summarization: Same article, different instruction phrasing
• Multi-turn dialogue: Conversation history grows across turns
• Code completion: Shared repository context across requests

SemBlend has minimal overhead on dissimilar workloads (e.g. code generation from scratch: measured 0% hit, 0.96x — 4% overhead from embedding + search).

Business Source License 1.1 (BSL-1.1). Free for non-production use including testing, development, evaluation, and academic research. Production use requires a commercial license from WorldFlow AI. Converts to Apache License 2.0 on 2030-03-16.

Contact: research@worldflowai.com