AI writes, code decides.
My Magic UW is open source under the MIT licence. This page shows how it is built, what we measured, how we measured it, and what we have not measured yet. Every figure comes from the repository's own benchmark record.
A desktop app with the database on your laptop.
An Electron app. The main process owns the app's own UW session; a utility-process
worker owns everything that touches your data. Your coursework stays in a local SQLite
store (node:sqlite).
Code decides
Dates, ids, permissions and quotes are exact facts, so code computes and checks them. A model's output never authorizes anything.
Jev judges
Jev is a typed judgment gateway: bounded questions code can't settle (for example, what kind of assignment this is), cached by input so the same question isn't asked twice.
Your AI is the agent
The student's own Claude Code runs as a CLI with a warm session pool. It drives the app through the app's own tools over the local database: read tools, page navigation and "open in Canvas". Harness skills are fixed cheapest paths.
One checked call per pack
A study pack is one model call whose output code checks: every stored quote verbatim, one correct answer per question. A failed check is replaced or dropped, not shown.
What we measured, and what we didn't.
As of 2026-09-27. Timings are from one machine (Windows 11, Intel Core Ultra 9 275HX, Node 24.14.1) and haven't been reproduced on a second. Missed targets are reported alongside met ones. A "p95" from fewer than 20 runs is the maximum of those runs.
Backend at 5,000 synthetic resources
| Measured | Before | After | Target | Result |
|---|---|---|---|---|
| Ingest throughput | 78 resources/s | 1,431 resources/s | ≥750 | met |
| Search p50 / p95 (n=120) | 56.0 / 197.4 ms | 3.3 / 4.8 ms | ≤5 / ≤15 ms | met |
| Database size per 1,000 resources | 15.6 MB | 10.3 MB | ≤11 MB | met |
| Question recall@5, 52 planted questions | 0.0 | 1.00 (52/52) | ≥0.90 | met |
| Correct "not found", 26 unanswerable | — | 0.96 | ≥0.80 | met |
| Purge ("Delete local data") | 8.7 s | 0.48 s, 0 rows left | ≤1 s | met |
| Zero-change re-sync growth | — | +1.48% | ≤1% | missed |
| Ingest slope, last batch vs first | — | 2.97× | ≤2× | missed |
Harness pnpm magic:perf --suite baseline; before 91e39fa, after f365af7, same machine. Planted questions were written by the builder. Source: docs/benchmarks.md §2.1.
Sync, search and routing
| Measured | How | Before | After |
|---|---|---|---|
| Canvas first sync, 5 synthetic courses | replay, 150 ms latency model, median of 3 | 115 requests, 5.7 s | 65 requests, 2.1 s |
| File acquisition, 300 synthetic files | replay, Files tab hidden, 60 ms latency | 0/300 files with text | 285/300 (15 scans marked for OCR) |
| MCP search at 5,000 resources | PR #19, n not stated | p50 6.3 s | about 0.28 s |
Agenda due verb, 306 assignments | PR #24, n not stated | 25–41 s | p50 / p95 576 / 580 ms |
| Command handled by code (0 tokens) | test, n=24 | — | p50 1.1 ms, p95 3.2 ms, 0 billed calls |
| AI fallback after a code miss vs AI only | 24 interleaved pairs, fake CLI at 800 ms | AI only p95 820 ms | fallback p95 819 ms (no added latency; gated in CI) |
Fresh claude -p vs warm-session follow-up | build record §5.5 | 5.8–7.4 s, 11.3k fixed tokens | 1.7–2.2 s, 2.8k fixed tokens |
| Jev calls, 100-assignment synthetic course | PR #14, code decides from submission types | 100 | 50 |
Source: docs/benchmarks.md §2.2–§2.9. Live first sync after these changes: not measured (65 s on the operator's account before them; target ≤10 s).
Quality: grounded answers and study items
| Measured | How | Before | After |
|---|---|---|---|
| Sentences contradicting their own cited quote shown to the student | live, student's Claude Code (claude-sonnet-5), 20 synthetic scenarios × 3; every sentence read by us | — | 0/77 shown (1/80 written, caught) |
| Correct sentences wrongly replaced by their quote | same live run | 15/60 answers (first fix) | 2/60 answers (final fix) |
| Planted item defects caught | offline item eval, code metrics; 4 synthetic courses + local MIT OCW | 43/62 (0.69), missed | 58/59 (0.98) |
| Correct items wrongly dropped | same | 5/198 | 0/196 |
| Stored quotes verbatim | same, code-checked | 216/216 | 217/217 |
| Repeat pack request at 0 tokens | same | 38/38 | 38/38 |
| Teaching text changed by name scrubbing; personal canaries leaked | 742 texts incl. 730 OCW pages, 5-student roster | — | 0 characters; 0/14 canaries |
Sources: docs/break-card.md (live, Wilson 95% intervals there); docs/benchmarks.md §3.1
(branch feat/item-eval at 3172c2a, tested in isolation; offline items come
from a rule-based stand-in writer, so they measure the checks, not a model's writing) and §2.10
(branch feat/privacy-hardening, tested in isolation).
Against NotebookLM, ChatGPT or Claude Projects, and Quizlet
| Comparison | Status |
|---|---|
| Grounded answer quality vs NotebookLM (Gemini Notebook) | not measured |
| Grounded answer quality vs ChatGPT or Claude Projects | not measured |
| Flashcard and quiz quality vs Quizlet | not measured |
| Time to first useful answer vs any of them | not measured |
| Item quality with a real model; grounded-ask citation pass rate; prompt-injection rate | not measured |
The head-to-head protocol was fixed on 2026-09-26 before any competitor run; none has run yet, so we claim no win. What the docs do state from public sources: NotebookLM's citations are direct quotes that aren't independently checked, and it needs each source added by hand (docs/notes/competitive-comparison.md).
How we measure and report.
Fixed before the run
- Thresholds are pre-registered; changing one after a run is a new, dated registration.
- Case files are hashed (
evals/freeze.ts); a changed, missing or extra file fails verification. - Before and after run the same harness on the same machine.
Checked, not graded by vibes
- Metrics are code-checked where there's one right answer: verbatim quotes, one correct option, arithmetic re-done by code.
- Comparisons use blind judges: system names shuffled, badges stripped, both orders rated. A model never judges its own family.
- Rates are reported as k/n with Wilson 95% intervals.
Case sets
- Synthetic courses and a synthetic university for Canvas replays.
- Public MIT OpenCourseWare (6.006, 6.042J, 18.05), read locally and never committed.
- Live figures from one student's own account are aggregates only, no course content.
Cost, latency and receipts
- One warm-up run is discarded; percentiles are nearest-rank over the raw samples.
- Adopting a change needs a 3× effect or a structural change, because run-to-run noise was about ±20%.
- Token and request budgets are CI gates. Every AI request leaves a receipt of what was sent.
Cheaper, checkable and private by default.
Study time costs 0 tokens
Packs are built once and cached by content; reviewing them is code. A repeat request with unchanged content made no model call in 38 of 38 cases.
One checked call per pack
No tool loops inside the app: one call, one schema, then code checks every quote, id and date before anything is stored.
Quotes you can verify
A grounded sentence cites a passage from your own materials, and code checks that the sentence says what the quote says.
Your data stays yours
A local store, consent per provider, and a receipt per request. "Delete local data" leaves no content row behind.
Clone it and build it.
Node 24 and pnpm 10.29.2. Tests and the performance harness use synthetic data and never touch a real account.
git clone https://github.com/benverhaalen/magic-uw cd magic-uw pnpm install pnpm dev pnpm test pnpm magic:perf --suite baseline