--- title: "Building Karpathy's Knowledge Base — Pi SDK Sessions as RAG" date: Sun Apr 05 2026 00:00:00 GMT+0000 (Coordinated Universal Time) excerpt: "How to build a RAG system using Pi SDK agent sessions instead of a vector database. Each session is a createAgentSession() call with different tool permissions — the indexer writes, the query agent reads, the eval agent checks citations. No LangChain. No orchestration framework. Working TypeScript code." template: "technical" category: "AI Engineering" --- [Part 1](/articles/building-karpathy-knowledge-base-part-1) argued you don't need embeddings for a personal knowledge base. An LLM reading a markdown index table picks the right files better than cosine similarity — at small scale. This post is the how. And the answer is simpler than I expected: **a [Pi SDK](https://github.com/mariozechner/pi) agent session that can read files IS a retrieval system.** You don't need LangChain. You don't need a retrieval pipeline. You need an agent with tool permissions. The entire RAG architecture — indexing, retrieval, generation, evaluation — is four calls to the same function with different tool configurations. --- ## Agent sessions replace the RAG pipeline The whole system is just sessions with different tool access: ``` ┌──────────────────────────────────────┐ │ Session Tools Job │ │ ─────────── ──────────── ────────── │ │ Indexer read + write build │ │ index.md │ │ Query read-only answer │ │ Research read + write answer + │ │ save │ │ Eval read-only check │ │ citations │ └──────────────────────────────────────┘ ``` No orchestration framework. No message passing between agents. Each session is a `createAgentSession()` call with different tools. The tools define what the session can do. That's the security boundary AND the architecture. The query session can only read — it literally cannot modify your documents. The research session can read and write — it answers AND saves analyses to `wiki/outputs/`. The eval session reads query traces and checks citations. Same LLM, different permissions, completely different behaviors. This is RAG without the R being a separate system. The retrieval IS the agent reading files. The generation IS the agent answering. One session does both. --- ## The indexer: how LLM-powered file selection works The indexer session is what makes [the no-embeddings approach](/articles/building-karpathy-knowledge-base-part-1#what-i-do-instead) work. It reads every source file, writes a one-line summary and topic keywords, and produces the index table that powers file selection at query time. ```typescript const loader = new DefaultResourceLoader({ cwd: folder, agentsFilesOverride: (current) => ({ agentsFiles: [ ...current.agentsFiles, { path: ".llm-kb/AGENTS.md", content: agentsContent }, ], }), }); const { session } = await createAgentSession({ cwd: folder, resourceLoader: loader, tools: [ createReadTool(folder), createBashTool(folder), createWriteTool(folder), ], sessionManager: SessionManager.inMemory(), settingsManager: SettingsManager.inMemory({ compaction: { enabled: false }, }), }); await session.prompt( `Read each file in .llm-kb/wiki/sources/ and write index.md with a summary table.` ); session.dispose(); ``` The key pattern is `agentsFilesOverride`. We inject a dynamically generated `AGENTS.md` that tells the agent what sources exist, what libraries are available, and what format the index should be. The user's folder stays clean — no config files, no framework scaffolding. The same pattern works for query, research, and eval. Different `AGENTS.md` content, different tools, same `createAgentSession()` call. The whole RAG system is variations of one function. --- ## The ingest pipeline: scan, parse, index, watch Four things happen when you run `npx llm-kb run`: **Scan** — walk the folder, skip `.llm-kb/`, collect supported files. Pure `node:fs`. **Parse PDFs** — every PDF produces two files. Markdown for the LLM. JSON with [bounding boxes](/articles/parsing-pdfs-with-bounding-boxes) for citations. LiteParse runs locally, no cloud API. Mtime check means re-runs are instant — only new or changed files get re-parsed. **Index** — the Pi SDK session above reads sources and writes `index.md`. **Watch** — chokidar watches the folder. New PDF → parse → re-index. Two-second debounce so dropping five files triggers one batch, not five cycles. ``` llm-kb v0.0.1 Scanning ./research... Found 12 files (8 PDF, 2 XLSX, 1 DOCX, 1 TXT) 8 parsed Building index... Index built: .llm-kb/wiki/index.md Watching for new files... (Ctrl+C to stop) ``` ## Why only PDFs get pre-parsed at ingest time I started with adapters for Excel, Word, and PowerPoint. Three more dependencies, three more edge cases. Then I realized the agent already has tools. Pi SDK sessions can run bash, read files, write scripts. When someone asks about a spreadsheet, the agent writes a quick script using ExcelJS (bundled with llm-kb) and reads it live. No pre-parsing needed. PDFs are different. They need [bounding boxes](/articles/parsing-pdfs-with-bounding-boxes) extracted ahead of time. They need OCR for scanned pages (expensive — do it once). Everything else is structured text the agent handles on demand. Cutting three adapters halved the codebase. The agent with tools replaced all of them. --- ## Try it: build a knowledge base in one command ```bash npm install -g @mariozechner/pi-coding-agent pi # authenticate once npx llm-kb run ./your-documents ``` [Part 3](/articles/building-karpathy-knowledge-base-part-3) adds the query engine — two modes (query and research), and the compounding loop where every answer makes the knowledge base smarter. [GitHub →](https://github.com/satish860/llm-kb) --- **Related:** [Part 1: No Embeddings](/articles/building-karpathy-knowledge-base-part-1) · [Why I Stopped Using RAG](/articles/why-i-stopped-using-rag-for-document-processing) · [PDF Parsing With Bounding Boxes](/articles/parsing-pdfs-with-bounding-boxes) *[GitHub](https://github.com/satish860/llm-kb) · [Pi SDK](https://github.com/mariozechner/pi) · [LiteParse](https://github.com/run-llama/liteparse)*