Bot Fraud, Manual Detection, and Why the System Worked (This Time)

When you launch a new product you get a bit obsessive about numbers. Okay, maybe you are always obsessive about numbers, let's not judge me. I checked Stripe on a Sunday afternoon about 72 hours post launch. We were past the launch chaos, past the "oh god is this working" moment, into the weird quiet of "okay now what."

I saw a surge of signups since Friday. Excitement rushed through me, then dread.

Looking closely I started to see a pattern.

I looked at the name on the account, the email, the credit card. Checked our Slack. Michael was definitely just one person. But the second account was also named Michael. Different email, same credit card number.

Then the third one. Same pattern.

By the time I'd scrolled through the first 15, I wasn't thinking anymore. I was just checking. Same card. Different variations on the same disposable email domains. Timestamps two minutes apart like clockwork. It had the rhythm of a script.

I sent a message to the team and gave a heads up.

By Monday morning, another wave hit. By Monday evening, we were seeing an explosion of accounts across four separate bursts, different hours, different email patterns, different timing strategies.

I did what I always do when I don't understand something: I looked at the logs.

The LiteLLM logs were weird. Every account had the same user agent: "OpenAJS." And when they actually made requests (which was rare), it was the same thing every time. Eight tokens, consistent output, no variation. They weren't building anything. They were checking what was there.

Not a bulk usage run, not someone trying to scrape the model cheap. Just reconnaissance. They wanted to know: what's our API? How does it work? What's it connected to? After they answered those questions, they basically... stopped.

The second wave had a slightly different signature. Same email pattern, but this time they actually tried a few synthesized calls before the keys were revoked. Third and fourth waves, the same rhythm. Methodical. Testing. Adapting.

What grinds me gears is that we caught this manually.

I was checking Stripe because I get obsessive after a launch. If I hadn't thought to pull that report on a whim, we'd have taken another 48 hours minimum to notice. The detection method was "Lauren is a paranoid person who obsessively monitors things and happened to be in the right place at the right time."

That's not a system. That's luck.

The response was solid, though. We pushed email verification before you can hit the API. We're blocking common disposable email patterns and we enforcement reCAPTCHA 0.7 at signup. That changes the economics from "script a few hundred accounts" to "actually hire humans to solve CAPTCHAs," which is a meaningful friction increase.

We could do more. It's just how much infrastructure do you build to stop bots from stealing $200 a month?

The answer changes when you're at $200K a month. It changes again at $2M. Right now, the Keycloak fixes give us breathing room. We're not going to pretend we have a battle-hardened system when we're still in the free-tier stage.

Bot abuse of free tiers isn't an edge case. It's a scaling problem. Every company offering compute on a free tier hits this eventually. The question isn't whether it happens. It's when, how hard, and whether you have a playbook.

Detection matters more than perfection. We didn't have a perfect system. We still don't. What we have is visibility. That's what actually caught this. Experience and just someone noticing something was off.

Next time, I want that to happen because the system caught it, not because I happened to be paranoid at the right moment.