I asked my own system when I did my first internship. It answered instantly, in a confident full sentence, and it was wrong by three years.
The right date was sitting in my own mail, in an email I had written years later. The system pulled 2003 from the wrong document instead of the correct 2006 date and dressed it up as fact. The code behind that answer is still in the project, switched off, with the failure recorded in a comment above it.
I’ve spent thirteen years building enterprise software. That work teaches a simple rule: exact questions get answered by the system of record, not by whoever sounds most confident. I forgot that rule as soon as an AI was involved.
One thing to say up front: I didn’t hand-write most of this code. I directed AI coding agents, reviewed what came back, tested it against my own data, and decided what stayed. AI did much of the implementation. I remained responsible for the evidence and the decisions.
What I was trying to build
I have years of mail and documents on disk. Finding a file by name was never the problem. The questions I struggle with are less tidy: when did I last deal with this, what did I decide, which of these three copies is the real one?
Desktop search is poor at all of that.
I also wasn’t willing to upload my mail to somebody else’s cloud. Mail is among the most personal data most of us own. So I set one hard rule before any code existed: the documents, search index, and model would all run on hardware I control. Nothing in the archive would leave it.
The result is a small private web app I call SecondBrain. It holds about sixteen thousand documents, most of them email, split into roughly ninety thousand searchable pieces in one database. Search and Ask sit on top.
The path I switched off
One answer path tried to extract personal facts while a question was being answered. It retrieved text that looked relevant, handed that text to the model, and accepted the result too easily. That is how an internship in 2006 became an internship in 2003.
Retrieval and generation are useful for questions such as “summarize this letter.” They are the wrong tools for an exact date when a stored value or confirmed record can answer directly. Telling the model to be more careful would not repair weak evidence.
So Ask has three routes. Exact values already stored in database columns, such as dates, numbers, and counts, return through a structured lookup without a language model. Long-lived facts about education, work, and identity can come from an evidence-backed table, but only after human confirmation. Open questions, the “what happened with this?” kind, go through retrieval and the model.
I still type one normal sentence. The system works out what kind of question it is and keeps deterministic answers away from generation.
Facts need proof
The facts-about-me table was the part I trusted least, so it got the strictest rule in the system.
Facts are proposed only from documents that could plausibly be authoritative. Newsletters and job alerts do not enter that pass. Every proposed fact must carry a quote, and the code checks whether the first 60 characters of that quote appear in the source text, ignoring case. No match, no candidate.
That check proves only that the model copied supporting text. It does not prove that the proposed fact is correct. The system currently holds 179 candidates, and I have not human-confirmed any of them yet. None count as verified.
I’d rather the system say “I don’t know” than tell me a confident fiction about my own life.
Testing instead of vibes
For a while quality control was simple: change something, run two or three searches, and decide it felt better.
That wasn’t testing. It was mood.
Now I keep a fixed set of 23 questions from my archive, each paired with the document I know is correct, and run the set after retrieval changes. The numbers move as the archive grows and gets cleaned, so I treat them as directional rather than proof that the system is good.
The reranking code also keeps a minimum candidate pool of 12. Twelve is a floor, not a cap; the calculation and its reason are recorded beside the code. Details like that are why the test set exists. It catches changes that sound sensible but move known answers in the wrong direction.
A slow laptop turned out to be useful
All of this runs on an ordinary 16 GB laptop without a serious GPU. Structured lookups return in milliseconds. Hybrid search takes a few seconds because the query is embedded locally before the two search paths are combined. A generated answer takes one to three minutes.
That slowness forced a useful question: which steps actually need a language model?
Far fewer than I assumed.
Stored values do not. Human-confirmed facts do not. The slow path is for questions that need retrieved evidence turned into prose.
Things that didn’t survive
The most useful document in this project may be the do-not-build list.
An AI-generated knowledge graph of the whole archive sounded impressive. The research and operating cost did not justify it, so it went on the list and was never built. On-the-fly fact extraction did get built. It produced the wrong internship date and is currently disabled in the code.
The system also keeps both keyword search and search by meaning. Meaning-based search helps when I remember an idea but not the exact words. Keyword search remains better for names and reference numbers, where the literal text is the point. The interface still has a keyword-only mode, while the vector leg adds useful recall when literal matching is not enough.
What I’d keep
If I started again, I’d build the boring half first: clean data coming in, a database I can inspect when an answer looks wrong, a repeatable test, and a clear rule for what counts as evidence. The chat box is the easy last five percent, and it is the first thing everyone wants to build.
Three rules survived the work. If a database can answer the question, let the database answer it. A fact without proof is not a fact. “I don’t know” is a valid answer; a visible gap beats a convincing invention.
They apply to how the system was built, too. AI agents wrote most of the code. The test set, evidence rules, and my review kept me responsible for what the system was allowed to call true. That is the same evidence-first habit I bring to enterprise platform work.
Better OCR for the genuinely ugly scanned attachments is next on the list. Most of the remaining work still looks like data work.