My AI Wrote the Bug, and Its Test Said It Was Correct
What a local open-weight model found when it audited an AI-built app before release.
- Written by
- Naresh Kumar
- Published
- Reading time
- 4 min
- Topic
- Tech & AI
I’m building Jot, an iPhone and Mac time tracker I use to bill consulting clients. I built it with a coding agent, in this case Claude Opus 5.5 in Claude Code, following the agentic and software engineering principles I bring to every project: a written spec, recorded decisions, small changes, and tests for the logic that handles money and sync. In one evening we added per-client letterheads and kept past entries at the billing rate they were logged with, all with tests passing.

Jot's timer on iPhone.
That speed worried me. Before shipping something that puts money on documents I send to clients, I wanted a reviewer that hadn’t written the code.
The setup
I ran mlx-community/Qwen3.8-27B-8bit, a dense 27B model quantized to 8 bits, through oMLX on my MacBook Pro M5 Max (128 GB). Its brief was to act as an independent senior engineer auditing the app before it ships to paying users, and to report ranked findings with file and line, quoted code, a failure scenario and a fix. It saw only the source and the README, not our earlier review notes, so it couldn’t simply agree with us.
The model on Hugging Face: Qwen3.8-27B converted for Apple silicon, 29.5 GB at 8 bits.
There were two passes: core logic (billing, sync, migrations and their tests), then the apps (iPhone, Mac, widgets, PDF export, privacy, accessibility).
| Metric | Value |
|---|---|
| Input read | ~135,000 tokens |
| Speed | ~285 tok/s reading, ~5.9 tok/s writing (two passes sharing one GPU) |
| Time | About 73 minutes, both passes in parallel |
| Memory | ~29 GB of 128 GB |
| Cost | $0, and the code never left the laptop |

oMLX after the two passes: two requests, 134,823 tokens read.
It never wrote its report: it spent its whole 24,000-token budget per pass thinking. Its saved reasoning ended in a ranked list of 29 candidate issues, so nothing was lost.
What it found
Each candidate was checked against the code. Several described the same problem from different screens, so the 29 came down to 22 distinct issues: 7 real and new, 5 already on record from the earlier review or the decision log, 5 wrong, and 5 minor or unproven that I didn’t act on.
The headline bug: when a client’s currency changed mid-week, the total added the amounts together, so $100 + €200 showed as “300.” The AI had written a test for exactly this case earlier that evening, and the test asserted 300. It passed.
The other six:
- Deleting or saving an entry could fail silently on iPhone and Mac, and so could the Mac’s Start and Stop.
- A sync merge dropped the client’s chosen letterhead.
- A migration could create duplicate letterhead ids.
- A stored time zone was never used.
- CSV rows past midnight had no end date.
- The clock’s second hand still bounced when Reduce Motion was on.
The false alarms mattered too. The model confidently said to remove an iOS background mode, which would have silently broken iCloud sync.
All seven real issues were fixed that evening. The five in the shared billing, sync and export code got automated tests; the two on-screen fixes have none yet.
Takeaways for vibe coders
1. Passing tests only mean the code does what the author believed. When one AI writes both the code and the test, a wrong belief gets written down twice, and the test still passes. Read the assertions in your money-related tests yourself.
2. Have something that didn’t write the code review it. A 27B model on a laptop found bugs the far stronger model that wrote the code had missed. It didn’t need to be smarter; it just didn’t know what the author intended.
3. Treat AI review output as leads, not verdicts. About one issue in three was real and new. Check each against the code before acting on it, especially the confident ones that tell you to delete something.
4. Focus review on the code where mistakes are expensive. Billing, migrations, sync, error handling and anything a customer receives deserve an independent review before every release. A local model makes that free and private, but at ~6 tokens per second it’s a release check, not something to run on every commit.
Vibe coding gets you to “it works” fast. Shipping still takes the fundamentals: tests that assert the right answer, a reviewer who didn’t write the code, and errors you can see. AI can help with all of them, but only if you make them part of the process.
