substrate

A folder of markdown, read as a graph

There is no database behind the wiki. There is a directory tree of .md files, a convention for linking them, and about two hundred lines of TypeScript that turn that into something a model can navigate.

The source calls this the Karpathy LLM Wiki pattern — cited twice, at src/tools/files.ts:89 and :301. In practice the pattern means: keep knowledge as plain files a human can edit, let the model read them directly instead of through a retrieval abstraction, and accept the token cost when the corpus is small enough. Everything in this section exists to make that cost negotiable.

What counts as a page

A single recursive walk defines the corpus. It is deliberately blunt: files ending in .md, absolute paths, sorted, with six directory names excluded.

src/tools/files.ts:27-45walkWiki()
// Walk wiki dir, return absolute .md paths (skips raw/, _proposed/, graphify-out/, .git)
async function walkWiki(): Promise<string[]> {
  const fs = await import('node:fs/promises');
  const out: string[] = [];
  async function walk(dir: string) {
    const entries = await fs.readdir(dir, { withFileTypes: true });
    for (const e of entries) {
      const full = `${dir}/${e.name}`;
      if (e.isDirectory()) {
        if (['.git', 'graphify-out', '_proposed', 'proposed', 'imports', 'raw'].includes(e.name)) continue;
        await walk(full);
      } else if (e.isFile() && e.name.endsWith('.md')) {
        out.push(full);
      }
    }
  }
  await walk(WIKI_DIR);
  out.sort();
  return out;
}

Every read tool in files.ts starts by calling this and re-reading files from disk. There is no cache and no watcher: the corpus is whatever is on disk at the moment of the call. For a wiki of a few dozen pages that is the right trade — correctness for free, cost paid in stat calls.

Two walkers, two definitions of the corpus

The write path keeps its own walker. src/tools/wiki_append.ts:17 excludes eight directories — the six above plus _review and auto — and it starts from <wiki>/wiki/ rather than the wiki root. So the set of pages files_neighbors can read is a strict superset of the set wiki_append will route to. A page living outside wiki/ is readable but is not an append target; asking to append to it produces a candidate list, or a new page under wiki/.

Page anatomy and link resolution

Three pure functions do all the parsing. extractTitle looks at only the first 800 characters and prefers YAML frontmatter over the first # heading, falling back to the relative path. extractSnippet strips frontmatter, skips headings, and returns the first real paragraph collapsed to a single line. extractWikilinks normalises link targets into slugs.

The knowledge-graph model: from a markdown file to a neighbourhood On the left, a markdown page with frontmatter, a heading, a first paragraph and two double-bracket wikilinks. Three extraction functions read it: extractTitle from the first 800 bytes, extractSnippet from the first non-heading paragraph, and extractWikilinks by regular expression. On the right, the resulting neighbourhood: a target node with three outgoing links, one of which resolves to no file, and two inbound references discovered by scanning every other page. 1 · a page on disk 2 · extraction 3 · the neighbourhood wiki/<slug>.md --- title: <Page Title> # <Page Title> First paragraph of prose that is not a heading … ## Updates - a durable fact [[other-page]] - another one [[missing]] (added …) first 800 B feed title detection whole file feeds link detection extractTitle(head, fallback) frontmatter title → # heading → path files.ts:51-58 extractSnippet(content, 280) strip frontmatter, skip headings files.ts:60-71 extractWikilinks(content) lowercase · spaces → hyphens anchors and pipe aliases dropped files.ts:73-82 slug resolution order 1 · exact slug match 2 · exact relative-path match 3 · either string endsWith the other else: suggestions from a 6-char prefix files_neighbors(slug) → one subgraph outgoing not yet written target page returned in full inbound — found by scanning every other page for [[slug]] or slug.md, returned as snippets whole response capped at max_bytes (default 30,000) Cost profile of one neighbourhood call reads: 1 target file + every page in the corpus (inbound scan) + each resolved outgoing target returns: ≈2–10K tokens of the relevant subgraph rather than the whole wiki — files.ts:180
The graph model. Nothing is precomputed. Every neighbourhood is derived at call time by reading the corpus and running two regular expressions over it. The edges are a property of the text, not of an index that can drift from it.

Links are text, and text is lossy

Outgoing edges are cheap: one regular expression over the target file. Inbound edges are expensive and approximate — there is no reverse index, so the tool opens every other page and tests two patterns against it.

src/tools/files.ts:75outgoing edges
const re = /\[\[([^\]|#]+)(?:[#|][^\]]*)?\]\]/g;
// capture group 1 stops at ] | or # → anchors and pipe aliases are discarded
// then: target.trim().toLowerCase().replace(/\s+/g, '-')

[[Memory System#Design|the memory system]] and [[memory system]] both normalise to the slug memory-system. A link whose slug matches no file is still returned, flagged resolved: false and rendered as “(not yet written)” — so an unwritten page is a visible hole in the graph rather than a silent omission.

src/tools/files.ts:231-234inbound edges
const incomingPatterns = [
  new RegExp(`\\[\\[${targetSlug}(?:[#|][^\\]]*)?\\]\\]`, 'i'),
  new RegExp(`\\b${targetSlug.replace(/-/g, '[- ]')}\\.md\\b`, 'i'),
];

The second pattern lets a page be found by prose that mentions its filename, with hyphens matching either a hyphen or a space. Note that the slug is interpolated into a regular expression without escaping — a page slug containing regex metacharacters would change the meaning of the pattern. Slugs produced by the write path are constrained to [a-z0-9-] (wiki_append.ts:22-24), so this only matters for files created by hand outside that path.

The cheap index

wiki_index exists so an agent never has to guess. It returns one line per page — relative path, byte size, extracted title — and the tool description states the intended cost: roughly 500 tokens for a 35-page wiki. Its input schema is empty; there is nothing to get wrong.

The four read tools over the substrate, by what they return
Tool Returns Files opened Size control
wiki_index Path, size and title of every page All, first 800 B each none needed
files_neighbors Target page in full, plus snippets of every outgoing and inbound neighbour All, in full max_bytes 1,000–500,000, default 30,000; snippet_chars 80–2,000, default 280
wiki_read_all Every page concatenated under ## <relative path> headers All, in full max_bytes 1,000–2,000,000, default 1,000,000
files_read Raw bytes of one file, local or over SSH One max_bytes 1–5,000,000, default 1,000,000

Truncation is always explicit. files_neighbors appends [... TRUNCATED at N bytes — pass larger max_bytes to see more ...]; wiki_read_all reports how many pages it did not reach. The model is told what it is missing and how to ask for it.

The corpus is small enough that the honest answer to “what is in the wiki?” is a directory listing.

the design premise of wiki_index

How material gets in

Ingestion is deliberately two-phase and the tool refuses to blur them. files_wiki_ingest writes a raw source into <wiki>/raw/ — the one directory the page walker ignores — and returns instructions rather than a page. Turning a source into curated prose is an agent's job, performed later, with the wiki's own schema document in scope.

Ingestion pipeline, from raw source to queryable graph Content or a fetched URL enters files_wiki_ingest, which sanitises the filename and writes into the raw directory. The raw directory is excluded from the page walker, so nothing is searchable yet. An agent later reads the raw file and writes curated pages into the wiki directory, either through files_write or through wiki_append, which commits to git. A separate scheduled builder converts the page set into a graph JSON file, which the three graph tools query. Two dashed feedback arrows show that the search index and the graph are rebuilt out of band. capture curation derivation query content: string pasted by the agent url: string parsed as URL, http/https only, then fetched files_wiki_ingest filename sanitised to [a-zA-Z0-9._-] — everything else becomes a hyphen <wiki>/raw/ excluded from walkWiki() not indexed, not searchable an agent, later reads the raw file with the wiki schema in scope, writes pages <wiki>/wiki/ curated pages, one per topic every write is committed to git search index BM25 + embeddings served by a local CLI graph.json built by a scheduled job on a 30-minute timer graph_query · graph_path · graph_explain read this file If the graph file is absent, the three graph tools fail closed with rebuild instructions, not an empty result — graph.ts:14-19
Ingestion. Two of the four stages are out of band — the search index and the graph file are rebuilt by processes ian-brain does not own. It reads their output and reports honestly when that output is missing.

The derived graph

The three graph_* tools do not traverse markdown. They shell out to a Python module against a pre-built graph.json, with a guard that runs first and returns a remediation message if the file is absent. That guard is the whole error-handling strategy: fail with instructions, never with an empty array that a model might read as “nothing connects these”.

graph tools — arguments and timeouts, from src/tools/graph.ts
ToolArgumentsSubcommandTimeout
graph_query question, dfs? (default breadth-first), budget? 200–20,000 output tokens, default 2,000 query <question> --graph … [--dfs] [--budget N] 60 s
graph_path from, to — node labels as they appear in the wiki path <from> <to> --graph … 30 s
graph_explain node explain <node> --graph … 30 s

The distinction that matters: files_neighbors answers “what is one hop from this page?” from the live text, while graph_query answers “what connects these two ideas?” from a snapshot that may be up to half an hour stale. Freshness and reach pull in opposite directions, and the tool descriptions say so.

No wiki page names, page contents, directory listings or file paths from the live corpus appear on this site. <slug>, <wiki> and <host> are placeholders.