The agent already has a memory
AI agents have remembered things for a while now. There’s CLAUDE.md for project instructions, and a memory the agent writes itself: small markdown files, one fact each. The cold-start problem is mostly solved.
But that memory holds the agent’s notes on working with me: my preferences, this repo’s shape, last week’s correction. I wanted something else: a second brain. A place where what I’m learning lives, that the agent builds with me and either of us can think with. I spent a few weeks building one. It’s called Reverie.
The idea isn’t mine
The shape comes from Andrej Karpathy’s “LLM Wiki” pattern: no vector database, nothing crammed into the context window. Give the agent a wiki and let it read and write that wiki the way a person would. The agent maintains the graph.
I already keep my own notes in Obsidian, so a wiki is something I understand. So is the way it rots when nobody tends it. The question was never “can the agent store notes.” It was “will it keep them good.”
Memory has types
One flat folder of markdown worked badly. Everything was the same kind of thing, so nothing had a clear job. The fix was borrowing structure from human memory. Episodic memory is a journal, one entry per session. Semantic memory is durable topic pages, abstracted from any one day. Working memory is a single Now page. There’s a self-model, and a procedural routine the agent runs each time it wakes.
What ties it together is consolidation. Every so often the agent reads back its raw journal entries and promotes what’s worth keeping into the topic pages. The rest fades. That’s what a brain does when you sleep, and it’s the part most memory setups skip.
Reading is the easy part
I drop a source into the vault and the agent ingests it. It writes a page summarising the source, then ripples the new facts into the topic pages it already keeps. Reading a paper is easy. Threading it into everything you already know is the tedious part, the part I never did by hand.
When a source contradicts an existing note, the agent doesn’t overwrite it. It files the contradiction as a proposal for me to settle. And every claim carries a tag: observed, inferred, or speculated. Without it, a guess written on Monday gets mistaken for a fact by Friday.
It runs while I’m not there
The agent doesn’t only run when I talk to it. Three times a day a scheduler wakes it for a session. It re-reads its notes, consolidates the journal, and picks up a thread it previously left open. If something’s worth telling me, it leaves a briefing in the form of a new mardown file in an ‘inbox’.
These sessions also forget. They drop an orphan nothing links to, or let a journal entry fade once its one good idea has been promoted. A memory that can’t forget isn’t a memory. It’s a log.
Agreement isn’t truth
The agent is good at synthesis. It finds where sources agree and writes that down. But agreement isn’t truth. A number gets copied into a note, a later note cites it, a third cites the second, and soon it reads like a fact nobody checked.
So one practice exists only to check. I start it by hand. It takes a well-used note, finds a claim the vault has quietly treated as settled, and tests it against primary sources. The actual paper, not the blog post about it. The first time I ran it, it caught a garbled number the wiki had repeated for weeks.
Where it is
I’ve stripped the parts specific to me and put it on GitHub: timkeller/reverie. A schema, a template vault, the scheduler scripts.
One warning: an unattended agent runs without permission prompts; the runner uses --dangerously-skip-permissions. Point it at a notes vault, not at anything you’d mind it editing on its own.
