wiki42

What it does

wiki42 is an open-source Python library that compiles a folder of markdown wiki pages into RAG-ready chunks for any vector database — one chunk per page, YAML frontmatter as typed metadata, [[wikilinks]] as an outgoing edge list, multilingual E5 embeddings via Pinecone Inference or local sentence-transformers.

Output is parquet or jsonl with {id, text, embedding, metadata} dicts ready to upsert into Pinecone, FAISS, Chroma, Qdrant, Weaviate — or a plain NumPy array. The library hands you portable data; what you do with it is your call.

Extracted from the internal toolchain at 42rows.com, the AI sales-intelligence platform that ships agents grounded on customer-specific wikis. Open-sourced as the chunking step of a RAG pipeline. Not an orchestrator. Not a vector store. Not yet another MCP server.

What's different

• 01 One chunk per page, no LLM teacher. Author-curated wiki pages are the atomic unit of meaning. Token-window chunking and LLM-teacher Q&A pre-generation amplify source bias and bloat indices 10–30×. Bench B on a 1851-page Italian wiki: +71% answer quality vs filesystem grep, 0 hallucinated facts vs 2.
• 02 Frontmatter as filterable metadata. [[wikilinks]] as a graph. Every YAML field on a page becomes a filterable field on its chunk — kind=="decision" AND confidence>0.8 works in any vector DB with metadata filters. [[slug]] references are parsed into metadata.wikilinks_out, an outgoing edge list ready for graph traversal.
• 03 Bring your own vector DB. Output is parquet or jsonl. Zero lock-in. Pinecone, FAISS, Chroma, Qdrant, Weaviate, or plain NumPy — your call. Pinecone server-side embedding chunks mid-sentence and keeps vectors in Pinecone; wiki42 keeps pages intact and hands you portable dicts.

Example commands

Install

Three install paths. pip is the default; Docker ships a non-root image with the MCP server bundled; clone-from-source uses uv workspaces for contributors.

# Cloud embeddings (Pinecone Inference, ~150 MB install)
pip install wiki42

# Local embeddings (sentence-transformers, ~1.2 GB)
pip install "wiki42[local]"

# Both backends available at runtime
pip install "wiki42[all]"

# CLI is available after install
wiki42 --version
wiki42 compile ./my-wiki --out chunks.parquet

Quick start

From a markdown wiki to a populated vector index in five lines. Same shape works with FAISS, Chroma, Qdrant, Weaviate, or a plain NumPy array.

from wiki42 import compile_wiki
from pinecone import Pinecone

# 1. Compile a markdown wiki into chunks (one per page, with E5 embeddings)
result = compile_wiki("./my-wiki/")          # ~5 s on a 1000-page wiki

# 2. Upsert into your vector DB of choice
index = Pinecone(api_key="...").Index("my-wiki")
index.upsert(vectors=[
    (c["id"], c["embedding"], c["metadata"])
    for c in result.to_list()
])

# Same shape works for FAISS, Chroma, Qdrant, Weaviate, or a plain NumPy array.

Benchmark

Two benches on the same real 1,851-page Italian sales wiki. Same LLM (Gemini 2.5 Flash via Vertex AI), same context-budget cap, answers graded against author-written ground truth. Scripts and raw results live in benchmarks/ — reproduce them or run your own.

When wiki42 does not help. Wiki ≤ 30 pages and your agent already has filesystem access (Claude Code locally) → grep -rli is faster and cheaper. wiki42 earns its keep when (a) the wiki ships to consumers without filesystem access (Claude Desktop, Cursor, Cline, hosted apps), (b) queries are semantic rather than keyword-matched, or (c) the same wiki is queried many times.

What's in each chunk

Stable IDs across recompiles. Idempotent upserts. Every YAML frontmatter field on the page lands in metadata as-is — your vector DB's metadata filter can target any of them.

{
  "id":       "companies/acme-logistics#0",       // stable across recompiles
  "text":     "passage: Acme Logistics S.p.A. ...", // E5 expects this prefix
  "embedding": [0.012, -0.089, /* ... */],        // 1024-dim float, or null
  "metadata": {
    "slug":          "companies/acme-logistics",
    "title":         "Acme Logistics S.p.A.",
    "kind":          "company",                   // any frontmatter field
    "confidence":    0.88,
    "wikilinks_out": ["segments/retail-warehouse", "products/wms-suite"],
    "char_count":    2147
  }
}

Why we built this

wiki42 was extracted from the internal toolchain at 42rows.com, an AI sales-intelligence platform that ships agents grounded on customer-specific wikis. We needed a clean way to turn a markdown wiki into vectors that work in any vector DB — without lock-in, without LLM-teacher chunk inflation, without losing the frontmatter signal. We open-sourced the result. If it is useful to you, a star on GitHub helps people find it; a look at what 42rows actually does helps us.

FAQ