Treat Your Product Experiments Like a Lab Notebook

Recently I came across a framework that was built to help AI coding agents stop fooling themselves. The AI part, of course, is interesting, but the fact that it forces every experiment into something like a lab notebook is what made me consider mapping this to product work.

Determining which metrics to follow is something everyone in product needs to do. The issue is when we do multiple things at once, it becomes difficult to determine what variable made a metric move in the direction we wanted. Pricing changes, so does the onboarding flow, and maybe some UI elements. Doing that all at once? Well now you can't tell what lever worked.

With this framework, every time AI tries something, it writes down four things in the same place. The hypothesis (what I think will happen). The change (what I actually did). The evidence (what happened). And the lesson (what I now know). Failures don't just get shrugged off. They get written down as rules. Taking the time to do this means future team members won't waste a week re-running a dead end somebody already ran in March.

There's one more piece of this framework that I particularly liked. Even when an experiment looks like a win, it doesn't get to count yet. It gets re-tested against fresh data the model hasn't seen, to make sure the win is real and not just the system learning to game its own scoreboard. One of the co-authors put it this way: if the metric isn't trustworthy, it'll just optimize toward an untrustworthy result faster.

A loop is not the same as progress.

four things product teams can take

One

Only use one isolated change per experiment. If you move the price and the flow and the copy at the same time, you have learned nothing except that the bundle did something. Change one thing so you can actually point at the cause when someone asks.

Two

Write your failures down, not just your wins. We are great at celebrating the thing that worked and weirdly silent about the thing that didn't. But "we tried discounting and it cratered margin without moving signups" is durable knowledge. It saves the next person, who might be you in six months, from re-litigating it from scratch. Failures written down as constraints are how a team stops going in circles.

Three

Do not trust the first number you see. The metric on your dev dashboard is not the truth. It's a proxy for the truth, and proxies drift, and teams quietly start optimizing for the proxy. Gate your wins on something independent. A held-out cohort, a fresh week of data, a check the experiment couldn't have trained itself to pass. A bad metric doesn't save you time. It just helps you sprint toward the wrong answer.

Four

Keep the record somewhere. Arbor calls its notebook a tree, big bets near the trunk and small refinements out on the leaves. When everything you've tried lives in a scroll-back nobody can find, you lose the map of where you've already been. You don't need their exact structure. You need a place that holds the shape of what you've explored, so the team can see the whole tree instead of the last three messages. This doesn't require a fancy system. This could just be now a shared doc with four columns, hypothesis, change, result, lesson, and a rule that a win doesn't get to be a win until you've checked it against data it hasn't seen yet.

The pull to change five things and ship is strong, especially when a number is staring at you and someone upstairs wants it to move. But moving one lever and knowing what you learned will get you further than the sprint ever did. Slower, sure. But it compounds. The other thing just loops.

And remember a loop is not the same as progress.