wiki-embedded

What it does

wiki-embedded turns a folder of markdown wiki pages into a knowledge base your AI agent can query as if it were an API. The compiler parses pages + [[wikilinks]], derives a thesis (the wiki's purpose lens), pre-computes answer chunks with an LLM teacher, and embeds everything with multilingual E5 — emitting one portable compiled_wiki.zip (~5 MB).

The MCP server (pip install wiki-embedded-mcp) loads that bundle and exposes 9 tools to Claude Desktop, Cursor, or Cline: semantic search, ready-made cited answers, page reads, backlinks, and crossref-graph traversal. Pre-computed chunks mean query_wiki_answer returns a source-cited answer at embedder speed — no per-query LLM call.

Extracted from the internal toolchain at 42rows.com, an AI sales-intelligence platform that ships agents grounded on customer-specific wikis. Three components in one repo: the compiler (Apify actor + Docker), the MCP server (pip + Docker), and an optional polars-runner analytics side-channel with a property-based-testing oracle.

What's different

• 01 Pre-computed answer chunks, not just passages. An offline LLM teacher synthesizes thesis-conditional "ideal answers" that cite their evidence by [[slug]]. At query time query_wiki_answer returns one ready, cited answer at embedder speed — the LLM is only in the loop for optional synthesis, not every question.
• 02 One portable bundle, MCP-native. The whole wiki compiles to a single ~5 MB compiled_wiki.zip you ship to colleagues or production; the server serves it over the Model Context Protocol so Claude Desktop / Cursor / Cline query it directly. Incremental recompiles re-embed only what changed.
• 03 Structure preserved, not shredded. One chunk per author-curated page (no token-window splitting that amplifies source bias), every YAML frontmatter field kept as filterable metadata, and [[wikilinks]] parsed into a crossref graph exposed via get_wiki_backlinks / get_wiki_graph.

Example commands

Install

Three install paths. pip is the default; Docker ships a non-root image with the MCP server bundled; clone-from-source uses uv workspaces for contributors.

# MCP server, cloud embeddings (Pinecone Inference)
pip install 'wiki-embedded-mcp[cloud]'

# …or bundle local embeddings (sentence-transformers + torch)
pip install 'wiki-embedded-mcp[all]'

# Serve a pre-compiled bundle — instant boot, no re-embedding
export PINECONE_API_KEY="..."
wiki-embedded-mcp --compiled-wiki https://.../compiled_wiki.zip

# …or point at a local markdown dir (compiles on first boot)
wiki-embedded-mcp --wiki ./my-wiki

Quick start

Wire the compiled bundle into your AI client. Drop this into claude_desktop_config.json (or Cursor / Cline), restart, and your agent queries the wiki over MCP — pre-computed cited answers, no glue code.

// claude_desktop_config.json  (or ~/.cursor/mcp.json, cline_mcp_settings.json)
{
  "mcpServers": {
    "wiki-embedded": {
      "command": "wiki-embedded-mcp",
      "args": ["--compiled-wiki", "https://.../compiled_wiki.zip"],
      "env": { "PINECONE_API_KEY": "..." }
    }
  }
}

// Restart the client, then ask your agent:
//   "use wiki-embedded to find the top prospects"
// → it calls query_wiki_answer(...) and gets a pre-computed, source-cited answer.

Benchmark

Measured on a real 1,851-page Italian sales wiki: 20 strategic queries (IT + EN), same LLM (Gemini 2.5 Flash), answers graded against author-written ground truth. Scripts and raw results live in benchmarks/ — reproduce them or run your own.

When wiki-embedded does not help. Wiki ≤ 30 pages and your agent already has filesystem access (Claude Code locally) → grep -rli is faster and cheaper. wiki-embedded earns its keep when (a) the wiki ships to consumers without filesystem access (Claude Desktop, Cursor, Cline, hosted apps), (b) queries are semantic rather than keyword-matched, or (c) the same wiki is queried many times.

What's in each chunk

Stable IDs across recompiles. Idempotent upserts. Every YAML frontmatter field on the page lands in metadata as-is — your vector DB's metadata filter can target any of them.

{
  "id":       "companies/acme-logistics#0",       // stable across recompiles
  "text":     "passage: Acme Logistics S.p.A. ...", // E5 expects this prefix
  "embedding": [0.012, -0.089, /* ... */],        // 1024-dim float, or null
  "metadata": {
    "slug":          "companies/acme-logistics",
    "title":         "Acme Logistics S.p.A.",
    "kind":          "company",                   // any frontmatter field
    "confidence":    0.88,
    "wikilinks_out": ["segments/retail-warehouse", "products/wms-suite"],
    "char_count":    2147
  }
}

Why we built this

wiki-embedded was extracted from the internal toolchain at 42rows.com, an AI sales-intelligence platform that ships agents grounded on customer-specific markdown wikis. We needed our agents to query a large wiki without feeding the whole thing to the LLM every turn — so we built a compiler that pre-computes cited answers and an MCP server that serves them. We open-sourced both. If it is useful to you, a star on GitHub helps people find it; a look at what 42rows does helps us.

FAQ