One engineer, many AI agents, 60+ apps.

Now: a native AI engine in Rust

Rebuilding the core inference engine in Rust, over a pinned llama.cpp

Why

The apps' AI server ran its inference loop through several layers. I wanted one fast, owned engine that every app can switch to without code changes.

Built

A Rust server (tokio, axum) over pinned, checksum-verified llama.cpp builds for CPU, Vulkan and CUDA: request scheduling, chat templates, embeddings, resumable model downloads with SHA-256 checks, a supervisor that restarts the engine, and an OpenAI-compatible API. Around it: an admin API and web console, licence-gated HTTPS, a Windows service, metrics and a Linux container image.

Results

Same laptop, same session, small models (1.5B and 3B), against my existing AI server: about 1.5× faster generation, first token in 45 ms instead of 163 ms, 2.2–3.1× more throughput with four clients at once, and a cold start answering in 0.8 s instead of 7.4 s. It passes 30 of 30 API conformance checks, and existing apps connect to it with no code change (5 of 5 end-to-end tests).

Trade-offs

It uses more memory while a model is loaded, and it has no audio, vision or governance features yet. I measured those costs too, and decided to go ahead for text models.

What I built

A cross-platform suite of local-AI apps (.NET 10, TypeScript, Rust)

Product

60+ apps — translation, document Q&A, chat and agents, transcription, image and video tools, email, accounting — that run AI on the user's own device. 14 are published on the Microsoft Store, Apple App Store and Google Play, in 16 languages.

Platform

360 shared libraries and 214 test projects: one engine layer over several local AI runtimes (llama.cpp, ONNX Runtime, Microsoft Foundry Local), agents and retrieval (Microsoft Agent Framework, MCP, hybrid search), licensing across three app stores, consent-gated telemetry, and back ends on Cloudflare Workers.

My role

Sole engineer and architect. I set the architecture and the rules, direct the agents, review and test what they produce, and handle releases to every store.

My own tool for running a team of agents

AI Coding Coach — a desktop app I built to manage Claude Code and Codex sessions side by side

What it does

Runs Claude Code and Codex sessions in embedded terminals, browses every past session across my machines, keeps a crash-safe prompt editor and history, tracks each tool's subscription usage with a forecast, and backs conversations up to Git.

Agents check agents

One click hands a session to a different agent for an independent review. It receives my original prompts word for word and the working tree as the source of truth, not the first agent's summary of what it did.

Coaching

Measures my own habits — prompts per task, interruptions, clarification rate, tool errors — and can rewrite a draft prompt using a small local model, falling back to the original if anything fails.

Lessons from it

An automatic "continue" feature once kept an agent going for days. I found it from the transcript: 382 injected messages, a median of 6 seconds apart. I added a hard budget of 25 unattended turns. Long prompts were also arriving cut off in the CLI; I reproduced it against the live tool and fixed it by sending prompts as one bracketed paste.

How I work with the agents

Agents are fast; the system around them is what makes the output reliable

Rules

A project instruction file every agent reads first: the architecture, the conventions, and rules written after real incidents (for example, "always wipe the publish folder before packaging" after a release shipped a stale library).

Memory

About 300 short notes of decisions, gotchas and open items that carry across sessions, so an agent starting fresh knows what was decided last week and why.

Parallel work

Up to 17 Claude Code agents at once across three laptops, on a shared code tree. Each agent stages and commits only the files it changed, so parallel work doesn't collide.

Shared parts

When two agents solved the same problem differently, I merged the result into one shared library and wrote a rule that every app must use it.

How I check what the agents build

I treat AI output like a strong but new team member's pull request

Guard tests

46 "drift-gate" tests fail the build when an agent re-invents a shared component or breaks a convention — for example, writing its own model-selection logic instead of the shared one.

Output scans

A gate scans the built app for strings that must never ship, such as another brand's name in an app published under a different company.

Real runs

I run apps and real data, not only unit tests: small AI models are tested on real documents and their failures become rules or tests.

Incident rules

Store-submission and packaging checklists grew from real rejections and crashes; agents must read them before any release.

A small example: this website

Built with Claude Code from my notes in about a day

Built

A career profile in Markdown, résumés written as HTML and printed to PDF by script, this static site on Cloudflare Pages, and a contact form protected by Cloudflare Turnstile that sends mail through Amazon SES.

Caught

After one change, my check of the PDFs' text showed an item I had removed was still there. The cause: an agent's edit to the build script used variable names that PowerShell treats as identical, so new PDFs went to the wrong folder. I found it, fixed it and rebuilt everything.

Lesson

Verify the output, not the claim that the work is done.

Try one

AI PDF Reader — ask questions about a PDF and get answers with page references, on your own PC

Open on the Microsoft Store

The code is private. I am happy to show the codebase, the agent rules and the test gates in a screen-share.