← Blog
Section 1 of 3
  1. The setup
  2. What it found
  3. Takeaways for vibe coders

← Blog

My AI Wrote the Bug, and Its Test Said It Was Correct

What a local open-weight model found when it audited an AI-built app before release.

Written by
Naresh Kumar
Published
Reading time
4 min
Topic
Tech & AI

I’m building Jot, an iPhone and Mac time tracker I use to bill consulting clients. I built it with a coding agent, in this case Claude Opus 5.5 in Claude Code, following the agentic and software engineering principles I bring to every project: a written spec, recorded decisions, small changes, and tests for the logic that handles money and sync. In one evening we added per-client letterheads and kept past entries at the billing rate they were logged with, all with tests passing.

Jot’s timer on iPhone: 00 hours and 11 minutes in large digits, 38 seconds, started 11:15 PM, with Pause and Stop buttons

Jot's timer on iPhone.

That speed worried me. Before shipping something that puts money on documents I send to clients, I wanted a reviewer that hadn’t written the code.

The setup

I ran mlx-community/Qwen3.8-27B-8bit, a dense 27B model quantized to 8 bits, through oMLX on my MacBook Pro M5 Max (128 GB). Its brief was to act as an independent senior engineer auditing the app before it ships to paying users, and to report ranked findings with file and line, quoted code, a failure scenario and a fix. It saw only the source and the README, not our earlier review notes, so it couldn’t simply agree with us.

The Hugging Face page for mlx-community/Qwen3.8-27B-8bit: an MLX conversion of Qwen3.8-27B, 27B parameters, 8-bit, Apache 2.0 licence, 29.5 GB

The model on Hugging Face: Qwen3.8-27B converted for Apple silicon, 29.5 GB at 8 bits.

There were two passes: core logic (billing, sync, migrations and their tests), then the apps (iPhone, Mac, widgets, PDF export, privacy, accessibility).

Metric Value
Input read ~135,000 tokens
Speed ~285 tok/s reading, ~5.9 tok/s writing (two passes sharing one GPU)
Time About 73 minutes, both passes in parallel
Memory ~29 GB of 128 GB
Cost $0, and the code never left the laptop

The oMLX dashboard on my Mac: 2 requests, 134,823 prompt tokens, average speed 285.4 tokens per second for prompt processing and 5.9 for generation

oMLX after the two passes: two requests, 134,823 tokens read.

It never wrote its report: it spent its whole 24,000-token budget per pass thinking. Its saved reasoning ended in a ranked list of 29 candidate issues, so nothing was lost.

What it found

Each candidate was checked against the code. Several described the same problem from different screens, so the 29 came down to 22 distinct issues: 7 real and new, 5 already on record from the earlier review or the decision log, 5 wrong, and 5 minor or unproven that I didn’t act on.

The headline bug: when a client’s currency changed mid-week, the total added the amounts together, so $100 + €200 showed as “300.” The AI had written a test for exactly this case earlier that evening, and the test asserted 300. It passed.

The currency bug, before and after One client, one week: 100 dollars logged in US dollars, then 200 euros after the client switched to euros. Before the fix, the total showed 300, and a test asserted 300. After the fix, each currency has its own total: 100 dollars and 200 euros, side by side. One client, one week $100 logged in USD, then €200 after a switch to EUR Before Total: 300 The test asserted 300, so it passed After the fix $100 · €200 Each currency keeps its own total
The headline bug, simplified. The fix keeps a separate total for each currency and lists them side by side.

The other six:

  • Deleting or saving an entry could fail silently on iPhone and Mac, and so could the Mac’s Start and Stop.
  • A sync merge dropped the client’s chosen letterhead.
  • A migration could create duplicate letterhead ids.
  • A stored time zone was never used.
  • CSV rows past midnight had no end date.
  • The clock’s second hand still bounced when Reduce Motion was on.

The false alarms mattered too. The model confidently said to remove an iOS background mode, which would have silently broken iCloud sync.

All seven real issues were fixed that evening. The five in the shared billing, sync and export code got automated tests; the two on-screen fixes have none yet.

Takeaways for vibe coders

1. Passing tests only mean the code does what the author believed. When one AI writes both the code and the test, a wrong belief gets written down twice, and the test still passes. Read the assertions in your money-related tests yourself.

2. Have something that didn’t write the code review it. A 27B model on a laptop found bugs the far stronger model that wrote the code had missed. It didn’t need to be smarter; it just didn’t know what the author intended.

3. Treat AI review output as leads, not verdicts. About one issue in three was real and new. Check each against the code before acting on it, especially the confident ones that tell you to delete something.

4. Focus review on the code where mistakes are expensive. Billing, migrations, sync, error handling and anything a customer receives deserve an independent review before every release. A local model makes that free and private, but at ~6 tokens per second it’s a release check, not something to run on every commit.

Vibe coding gets you to “it works” fast. Shipping still takes the fundamentals: tests that assert the right answer, a reviewer who didn’t write the code, and errors you can see. AI can help with all of them, but only if you make them part of the process.