I tested open-weight AI models on real product work

I started with the very product-person thought of, "Should this be a whole page on my website?"

Which is how I knew I was already in trouble.

There is a specific kind of danger in building things with AI, which is that every half-formed curiosity can become infrastructure if you let it. A benchmark suite. A living leaderboard. A dashboard. A beautiful little public artifact with filters and charts and maybe a place in the nav.

Reader, I was absolutely about to make a page.

Then I did the annoying responsible thing and ran the experiment first.

the question I actually cared about

I have been using expensive my Claude Code subscription (or a Claude model via my work subscription) for so much of my work. Meaning, I've been leaning heavily on frontier models. Meeting notes. Product writing. Competitive research. The weird glue tasks that sit between "thinking" and "making something useful."

Given that I run local agents, local models, and enjoy testing open-weight models, I was really curious about how much of this work actually needs the fancy model?

How much of this work actually needs the fancy model?

Not the demo work. Not the "write me a haiku about Kubernetes" work. Real PM work. The stuff where the output becomes a decision, a Slack update, a release note, or the beginning of a strategy conversation.

So I built a tiny benchmark around three tasks I actually care about:

1. Turn a messy meeting transcript into a debrief.
2. Turn technical pull requests into customer-facing release notes.
3. Turn messy pricing notes into a comparison table with confidence notes.

The obvious benchmark question is, "Can the model get the answer right?"

The better question is, "Can the model admit when the answer is not there?"

the contestants

I tested three cheap cloud models through OpenRouter:

• MiniMax M3
• Qwen 3.7 Max
• GLM 5.2

I also tested my local Qwen3 30B model running on the Mac Studio knowing it would struggle but I was curious nonetheless.

The whole OpenRouter run cost about eleven cents.

ELEVEN cents.

I have spent more than that emotionally deciding whether to buy a coffee. The robots are not the expensive part of this workflow. My judgment is.

task one: the meeting debrief

The first task was the one I thought would feel the most obviously useful. I gave the models a long, messy transcript from a sanitized platform maintenance discussion and asked for a structured debrief: summary, decisions, action items, open questions.

This is exactly the kind of thing people want AI to do.

All three cloud models completed it. None of them nailed it.

MiniMax and cloud Qwen both scored 64 out of 100. GLM scored 57. The local Qwen did not complete the task at all. It closed the connection on the giant transcript prompt, then recovered afterward on a tiny health check.

That local failure is useful information. It does not mean the model is bad. It means the workflow is wrong. A 92,000-character transcript shoved into a local model as one giant request is not a workflow. I suspected this would be too much stress for the local model and testing it was the way to confirm.

The cloud models had a different problem. They could read the meeting. They could summarize the vibe. They could produce something that looked like a debrief.

But under the strict rubric, they missed key decisions, invented some certainty, and sometimes treated "this was discussed" as "this was decided."

In product work, "we discussed this" and "we decided this" are different species. One is a bird in the room. The other is a bird you have agreed to feed.

If an AI turns every bird into a pet, you now have a very weird roadmap. Probably even worse than the one you keep getting pushed to create by stakeholders.

task two: release notes

The release notes task was almost boring.

I gave the models five merged pull requests from an open-source repo and asked for customer-facing release notes. The models had to include the user-facing changes, skip the internal test fix, and avoid making engineering gibberish sound like a product launch.

They all did well.

MiniMax and GLM scored 98. Cloud Qwen scored 96. Local Qwen scored 90.

The local model was fast and mostly right, but it duplicated the same variable-only deployment fix under two different categories. Not catastrophic. Just the kind of thing a human editor would clean up in ten seconds.

This was the first hint that a big stand alone benchmark page on the website might be overkill.

If every model can do the task, the benchmark is not telling me much. It is still useful as a sanity check, but not as a story.

Release notes, at least in this small test, are now table stakes.

That is not nothing. It is actually very useful. But useful is not the same as interesting.

task three: pricing comparison

The pricing task is where the floorboards started creaking.

I gave the models messy pricing notes for four fitness tracker brands and asked for a structured comparison table. The prompt explicitly told them to include confidence notes and to say when something was not found, not confirmed, or only available from a secondary source.

This was the real test.

Not extraction. Honesty. A sign of a strong model is when it can be honest. It's why you constantly get "let me be real" type of responses from Claude and Chat.

MiniMax scored 94. GLM and cloud Qwen both scored 82. Local Qwen scored 60.

The difference was not whether the models could make a table. They could all make a table. The difference was whether they could keep the uncertainty attached to the right cells.

MiniMax did the best job preserving the gaps. It noticed when pricing came from press coverage instead of an official page. It kept "not confirmed" separate from "not found." It treated missing information as missing information, which is apparently now a personality trait I admire in software.

GLM and cloud Qwen were mostly fine, but they blurred the edges. "Not confirmed" became "not found." Trial-only became no free tier. The shape was right, but the labels lost precision.

Local Qwen was the most interesting failure. It completed the table, but then it started bringing in outside claims and assumptions that were not in the prompt. It filled uncertainty with confidence. Which is the exact thing I was trying to test. Smaller models want to do the right thing even if it means making it up. You know, like toddlers.

I thought I was testing cheap models.

I was actually testing task boundaries.

Release notes were easy enough that almost anything decent could do them. Meeting debriefs were hard because long context plus human ambiguity is a nasty combination. Pricing comparisons were revealing because the model had to keep two things in its head at once: the answer and the evidence for the answer.

That is the part I care about now. Not "can it write?" Most models can write something that looks okay. The bar has moved. Writing polish is no longer the scarce thing.

The scarce thing is knowing when the model is quietly sanding off uncertainty.

Because that is where product work gets expensive. Not in the bad sentence. In the almost-right summary. In the action item that was implied but not assigned. In the pricing table that looks complete because the model hated an empty cell.

I hate empty cells too. This is why I am vulnerable to the machine's nonsense.

what I would trust today

I would use the cheap cloud models for first drafts of release notes.

I would use them for pricing comparisons if the prompt forces confidence notes and a human checks every ambiguous cell.

I would use them for meeting debriefs as a starting point, not as the source of truth. I'll stick with my work approved frontier model for that job.

I would use local Qwen for smaller structured tasks where speed and privacy matter.

I would not shove a giant transcript into the local model and expect magic. If I want that workflow locally, I need chunking, intermediate notes, and probably a better scoring pass. Annoying, but fair.

The point is not that cheap models are secretly just as good as expensive ones.

The point is more useful than that.

Some real product work is now cheap enough to run casually. Some of it still needs the expensive model. Some of it does not need a better model at all. It needs a better workflow, ie, harness.

That is less satisfying than a leaderboard. So, no, I am not building the giant benchmark page yet. I am writing this down first.

The page can wait until the experiment earns it.