A 0.6B model on a phone reached 80% of Claude Sonnet. Here is how I measured it.

Heed turns a messy spoken note into a clean, sorted task list, entirely on the phone. This is how I built the model, and, more importantly, how I convinced myself the numbers are real.

The problem

People think out loud. A note sounds like "umm okay so after church I need to call Kwabena's mum about his boots, before Tuesday's training… oh and Kojo said my phone will be ready Monday at four, so I have to go pick that up… and yeah, buy like twenty cones, the old ones are all ruined". What the user wants back is three items: a call, a pickup on Monday at four, and something to buy. Nothing invented and no detail lost.

The hard part was the constraint, not the task. Heed is private by design: the app has no internet permission, so a cloud model is not an option. The model had to fit on an ordinary Android phone, run on the CPU in a few seconds, use about a gigabyte of memory, and read the output of a speech model that writes everything in lowercase with no punctuation. In practice that means a model under one billion parameters.

So the real question was: how close can a model that small get to a frontier model on this one narrow job? Answering it honestly turned out to be most of the work.

The result

On a sealed test set of 617 notes, 126 of them real notes from me and five friends, the fine-tuned Qwen3-0.6B running as an int8 file on a OnePlus phone scored 75.6 judged task F1. Claude Sonnet 5.5 scored 94.8 on the same notes. That makes Heed 80% of Sonnet, and 84% on the real notes alone.

I want to be precise about what this does and does not say. Heed beats GPT-5.4 mini by 20 points, and the interval is nowhere near zero. Claude Haiku 5.5 and Gemini 3.5 Flash are still better, by 10 and 16 points. On the 126 real notes alone, which are shorter and easier, GPT-5.4 mini catches up and the two are level. A 0.6B model running offline is not a frontier model. It gets surprisingly close on one narrow job, at no cost per note and with no data leaving the phone.

How it was measured

This is the section I would read first if I were reviewing this work. A number like "80% of Sonnet" is only as good as the test behind it.

1. A sealed test, opened once

Early on I set aside 617 notes: 126 real ones, plus synthetic and long notes in fixed shares. They were never used for training, never used to pick an epoch or a seed, and never even copied to the machine that trained the models. The test was run exactly once, on the final model, after I had written down in advance how each result would be worded. Everything I tuned, I tuned on a separate validation set of 451 notes. That set flatters the chosen model by about a point, which is why the headline uses the test.

2. Why not word overlap

My first metric compared the model's tasks with an answer key using word overlap. It is cheap and repeatable, but it has two flaws that matter here. It counts a different but correct wording as an error: "Phone Kwabena's mum about the boots" against "Call Kwabena's mother about his boots". And because the key was written by Sonnet, it rewards models that write like Sonnet. Comparing frontier models that phrase things their own way would be unfair. Word overlap is fine for comparing two versions of my own model during development. It is not fine for a headline.

3. The metric: judged task F1

For the comparison, a judge reads the note and a list of tasks, and labels every task:

• good: means what the speaker meant; they would keep it as is.
• fixable: the right task, but it needs an edit (a lost time, a wrong name, misleading wording).
• wrong: invented, meaning changed, a remark turned into a task, or a repeat.

It also lists what the speaker clearly meant that no task covers: the missed tasks. Then:

precision = good / (good + fixable + wrong)
recall    = good / (good + fixable + missed)
F1        = 2 · precision · recall / (precision + recall)

A fixable task counts against both precision and recall. That is strict on purpose. A task that needs an edit still costs the user an edit, so it should not count as a success. A small example: "buy milk and eggs, call the dentist friday, and i wonder if we should repaint the fence". If the model lists "Buy milk", "Buy eggs", "Call the dentist" (Friday lost) and the fence as an idea, it scores 3/4 for both precision and recall, so F1 is 75. Keep "Friday" and it is 100.

4. An LLM judge, and how I tested it before trusting it

Judging about 7,000 tasks by hand was not realistic, so the judge is an LLM: Gemini 3.1 Pro. I chose a different model family from the one that wrote the labels, so it would not favour Claude-style lists. It sees only the note and an anonymous list. It never sees the answer key, never knows which system wrote the list, and the calls run in a shuffled order.

An LLM judge is a measuring instrument, so I tested it like one before using it. Each check had a pass mark written down before the test:

The human check is the one I care about most. I rated 50 items blind, without seeing the judge's verdicts, and the judge agreed with me at Cohen's κ 0.84 (92% raw agreement). Earlier in the project I checked the simpler scorer the same way: 351 task pairs, κ 0.95 between my links and the automatic ones. I also measured anchoring: showing the judge the answer key moved its "good" rate by only 1.7 points.

A second judge from another family. Any claim that names a competitor also had to hold under a second judge, GPT-5.4, on 100 notes. The order came out the same: Sonnet, Flash, Haiku, Heed, GPT-5.4 mini. Two details make this check stronger. GPT-5.4 rated its smaller sibling about 12 points higher than Gemini did, which is a possible family bias that works against my claim. Heed still came out ahead, +10.3 [+3.3, +17.2]. And under that judge Heed reached 86% of Sonnet, so 80% is the more conservative of the two numbers.

5. A fair fight

The cloud models got the full 4,105-character labelling rulebook in their prompt; Heed got none, because it learned the rules from data. They ran as served, at temperature 0 with default reasoning settings. All answers went through the same parser. GPT-5.4 mini might do better with more reasoning effort, and I say so wherever the number appears.

What the errors look like

On real notes, Heed gets the whole list perfect on 52% of notes, against 83% for Sonnet. It stays quiet correctly on 39 of 42 notes with nothing to do. The weakest bucket is Watch: 15% of Watch items land in the wrong bucket, partly because "watch" is also a verb.

The data

I had a few hundred real notes and needed thousands, so most of the training data is synthetic. Every note was labelled the same way: Claude Sonnet 5.5 at temperature 0, under one frozen labelling prompt, so the model learns one consistent style.

Two choices mattered more than the rest:

• Train on what the phone actually sees. The speech model writes lowercase text with no punctuation. A model trained only on tidy text lost almost 6 points on that input. Training every note twice, once as written and once in speech style, cut the loss to 0.2.
• Write data for the errors you have. After reading every mistake on the validation set, I generated 1,696 notes aimed at the failure types I found: people correcting themselves mid-sentence ("make that six"), details that must be kept, rambling, feelings that are not tasks. Wrong items fell by about a fifth, and real-note quality rose by 5.5 points [+1.4, +10.3]. The overall gain, though, missed the pass mark I had set for it (+1.4, interval −0.2 to +2.9), and I recorded it as a miss rather than moving the goalposts.

Choosing the model

I fine-tuned six small models on the same data with the same recipe and counted the corrections a user would have to make per note. Bigger was better, smoothly. A learning curve on the same data showed quality still improving at 100% of the notes. Data, not size, was the limit, which is why most of the later effort went into data.

Qwen3.5-0.8B was slightly better than Qwen3-0.6B. I still shipped the 0.6B. Google's official LiteRT-LM converter supported Qwen3 but not Qwen3.5's hybrid attention layers. The 0.8B ran slowly on a phone CPU, its file was 27% larger, and the quality gap was small. On a phone, the model you can ship beats the model that scores best.

I also checked whether a small model could do the job with good prompting and no training. It could not. Prompted Qwen3-0.6B needed 3.6 corrections per note and got 3% of notes fully right. Untrained small models were worse than a TF-IDF baseline at picking the bucket. Fine-tuning was not optional.

Training

Full fine-tuning or LoRA

I used both, for different jobs. The early bucket and reminder classifiers were trained with LoRA (rank 16, α 32, on every linear layer), because they were trained on a laptop GPU with 4 GB of memory, where memory was the binding constraint. For the task writer I used full fine-tuning. A 0.6B model trains fully in 17 GB on a single rented RTX 4090, for about $2, so LoRA's main advantage, memory, was not needed. For scale, rank-16 LoRA on every linear layer of Qwen3-0.6B would train about 10 million parameters, under 2% of the model.

To be clear about the evidence: I did not run a head-to-head comparison of LoRA and full fine-tuning for the writer, so I make no claim about which gives better quality here. That comparison would be the next obvious ablation.

Choosing the epoch: loss is the wrong signal

In every run, validation loss was lowest after the first or second epoch and then rose, while the task score kept improving.

Rising loss here means the model gets more confident on the tokens it gets wrong, not that its answers get worse. What I care about is the answers, so epochs are chosen by task score. The same run also showed that training longer did not help: eight epochs plateaued at the level of four.

Seeds and noise

The same recipe on two seeds gave results 1.5 to 3 points apart. So a single-run difference under about 4 points is not a result, and every recipe decision used paired comparisons and at least two seeds. It sounds obvious, but it killed several ideas that looked good on one run.

The most important decisions

What did not work

• More synthetic notes in my own style. 1,109 notes written to sound like me brought no measurable gain. The difference was smaller than the seed noise.
• Telling the model about the user. Adding past items and a job description fixed 41% of work-versus-personal mix-ups, and up to 70% after retraining with it. But it hurt reminders, and the retrained model got worse for users with no history. I dropped it.
• Training longer. Eight epochs were no better than four.
• Starting from the previous model instead of the base. No gain.
• A shorter "lines" output format instead of JSON. About 30% fewer tokens, but 2 points worse.

On the phone

The model runs through LiteRT-LM on the CPU with greedy decoding. The median is 5.1 seconds per note (p95 12.7 s). Time is almost entirely decoding: about 1.35 s plus 17.6 ms per character of answer. The whole app uses about 1.2 GB with the model loaded, and only one model (speech, tasks or search) is in memory at a time. Since the runtime has no grammar-constrained decoding, a small guard stops the answer when the JSON closes or a task starts repeating. 614 of 617 answers on the test were valid JSON.

Limitations

• The labels come from one model, Claude Sonnet 5.5, so "% of Sonnet" partly measures how closely Heed copies Sonnet. That is exactly why the judge never sees the key.
• The real notes come from six people. Accents, other speakers and other languages are not measured. The app is English only.
• Long, chatty notes are the weak spot: about 70 F1 on 40 to 99 word notes.
• Custom buckets the model has never seen are not tested at scale.
• The judge was validated against one human on 50 items, and checked by a second LLM. That is not a large human study.