I tested a fancier AI model on product work

Fugu is a pufferfish. It's also a new model by Sakana. So a model named after a poison fish, from a company whose name just means "fish" in Japanese, with a genuinely interesting approach to AI and a million-token context window. That definitely caught my attention.

I recently wrote about testing cheap open-weight models on real product work, and now this model is out, so it felt like an opportunity to run the same tests. I ran the same three tasks. Same meeting debrief. Same release notes. Same messy pricing table with confidence notes.

I'm fascinated by AI models, but I'm also actively working on not having vendor lock-in.

Most of my real work has been leaning on frontier models from one or two companies, and that's a comfortable place to be right up until it isn't. Prices move. Terms move. The model you tuned your whole workflow around gets deprecated on a Tuesday. The more the field fills up with real contenders, the more leverage the rest of us have to not get locked in.

So when a new player shows up with a genuinely different idea, I want to see it work before I have an opinion about it. Not the marketing version. The "does it do my actual job" version.

what makes Fugu interesting

Fugu is not really one model. Under the hood it's a learned orchestration system, a router that takes your request and quietly fans it out to a little team of sub-agents, then assembles the answer. So when you call it, you're not poking one brain. You're hiring a small crew that argues among itself and hands you the result.

I love this as an idea. It's the difference between asking one very smart person a question and asking a well-run team. The team can be slower and more expensive. But a good team checks each other's work, and that turns out to matter for exactly the kind of task where a single model gets overconfident.

Fugu isn't cheap. It ran the same three tasks for seventy-four cents. By itself. That's roughly twenty times the cost per task of capable open-weight models, and very close to Claude Opus pricing. This is because every one of those little sub-agents is burning tokens behind the scenes that you pay for.

how it actually did

It was very, very good.

On the meeting debrief it scored 94. On the release notes, 95. On the pricing table, a clean 100. Average of 96 across the three.

For contrast, here's where the cheap models landed on the same suite:

The thing I care about is not whether a model can write a table, but whether it can admit when the answer isn't there. The pricing task was the honesty test. The one where a weaker model fills the empty cell with a confident guess because it hates a blank space as much as I do.

Fugu got a perfect score on that one. It kept "not confirmed" separate from "not found." It flagged when a price came from press coverage instead of an official page. It hedged on the cells where the data was thin and committed on the cells where it wasn't.

The expensive part of product work isn't bad grammar. It's the almost-right summary that reads as finished. Fugu's little internal crew seems to catch each other before they sand the uncertainty off, which is a convincing argument for the orchestration idea.

so should you switch

MiniMax scored an 85 on this suite for a rounding error of a penny. It matched Fugu on release notes and came within a few points on pricing. For a huge amount of normal product work, the cheap model already clears the bar, and the expensive crew-in-a-trench-coat is buying you a handful of points on the genuinely hard tasks.

So I read this two ways at once. One, the field is getting crowded with real contenders, and a crowded field is the best protection against vendor lock-in. More good options means none of them owns me. Two, "good enough" keeps getting cheaper.

Fugu is going on my watch list. The architecture is novel enough that I want to see where it goes, and I'd reach for it on a high-stakes task where being calibrated is worth real money. For everything else, the eleven-cent crowd is still doing my job.