What I Learned from a $2,000 Pen Test

This is a short story about how I caught hackers timing their attacks around my daily routine.

For a few nights in a row, I woke up to alerts about a huge spike in sign-ups. By the time I started troubleshooting, the activity had stopped. I activated CAPTCHA and moved on.

When the pattern repeated yet again, I decided to do a deep dive into the data.

What I found was that the hackers were...

creating thousands of accounts
adding a valid payment method to each account
running a single very expensive LLM call (2-3 USD)

This would let the first request go through, then trigger a charge to their payment method. The payment method gets rejected, but the request has already been processed.

NOTE

The reason the first request goes through is that we deposit a little bit of money into the account when it's created. A nominal amount that's enough to play with the API. However, if a payment method is added, we allow people to go into overdraft, which is why their expensive LLM call goes through.

Anyway, using this method, they would get away with about a thousand dollars' worth of credits every night, which kept them interested in the service.

But what caught my attention wasn't the money – it was the timing. The attacks coincided with my sleep cycle.

Coincidentally, that day I decided to take a break and disconnect from my computer early. Just 30 minutes after I shut down my computer, I got the first notification.

I logged in to check, and it stopped.

Went to play some games and ... 30 minutes later, I got the second notification.

That's when it clicked – the timing of the attacks wasn't random. They were checking my Discord status to see if I was online.

Discord status

Sure enough, I confirmed this by setting myself as offline on Discord, and the attacks popped right back up.

Over the next few days, I used this insight to mess with the hackers and to use them as my personal pen testers.

I didn't want to remove free credits for everyone, so I began experimenting with different ways to deter future attackers.

I would make a change, then go 'offline'. I'd watch them troubleshoot their automations until they figured out a workaround. Then I'd go back 'online'. The attacks would resume, and the cycle would repeat.

I mostly forgot about the entire incident until they came back to try their luck again the following week. But despite a few nights of alerts about tripwires getting triggered, they never managed to get more than the few cents we deposited into new accounts – not enough of an incentive to keep trying.

In the end, it was the cat-and-mouse game that made the whole experience worth it. I got free pen testing; they got a few dollars.

Card testing vulnerability

I wasn't surprised about the overdraft feature being abused. This was something we were aware of and treated as a conscious trade-off between convenience and risk of abuse.

The bigger issue was that this made me realize that a malicious actor could abuse our system for card testing. That's a widespread problem and one that will get your Stripe account flagged. When researching this problem, I didn't find many effective solutions, so I wanted to dedicate part of this blog post to sharing what I learned.

Here's what I tried and how it held up:

Method	Effectiveness
Device fingerprinting	Ineffective. Fingerprints are great for detecting legitimate returning users (e.g. to bypass CAPTCHA), but because they are easy to fake, they are not effective at detecting malicious actors.
IP address blocking	Ineffective. Residential proxies are cheap and easy to get.
CAPTCHA	Mild deterrent. Ineffective. Many existing solutions to bypass CAPTCHA.
OTP	Mild deterrent. Ineffective. Many existing solutions to bypass OTP.
JA4	Somewhat effective. JA4 is a TLS fingerprinting method that identifies clients based on how they negotiate TLS connections. Of all data points that we collect, JA4 is the most stable identifier.
ALTCHA	Somewhat effective. ALTCHA is a proof-of-work challenge that requires the client to solve a computational puzzle before submitting a request. When combined with prior methods, can slow down the attacks enough to deter the attacker.
Rate limiting	Somewhat effective. Slows down the attacks, but may hurt legitimate users.

At the end of the day, each method is individually bypassable – the game is making the combination expensive enough that the attacker moves on.

Oh, and set your Discord status to offline.