独立开发者月报：涨价三倍销量、对抗爬虫与托管选型

New here?

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers.

Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall.

Highlights

• July’s book sales tripled the already-strong sales I saw in June.
• I’m fighting the world’s dumbest scraper bot.
• I’m using AI to make a multiplayer browser game.

Goal grades

At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals:

Pitch to 5 podcasts to talk about Refactoring English

• Result: Pitched to only one podcast
• Grade: D

I spent a lot of time working on a pitch to Talking Postgres, but then I realized the host has no contact information except for LinkedIn. So, I sent her a LinkedIn message but never heard back. I was a guest on The TMPDIR Podcast, though they invited me even before I pitched to them, so I can’t count that one.

Attract 30k unique readers to the Refactoring English website

• Result: The site had 23.8k unique readers
• Grade: B

I was secretly thinking of this goal as “get one post on the front page of Hacker News,” but I did and still didn’t reach 30k readers.

Wrap up early access, and declare the 1.0 release of my book

• Result: Still not at 1.0 release
• Grade: D

I ended up spending more time than I expected responding to user feedback. My feedback app is working, but it also generates new work that’s hard to predict.

Refactoring English metrics

June was the second best month of sales for my book since the Kickstarter, but then July’s sales tripled June’s.

The main reason for the jump in sales was that I ended the early access discount. I announced on July 13th that early access pricing would end on July 20th, so the price would increase from $30 to $49.

On the last day of the sale, I published a blog post called “Why I Stopped ‘Creating Content,’”, which reached the front page of Hacker News. And then it was the top post of the day on bubbles.town, a Hacker News-style site that’s more indie and less tech-centric.

There was a huge spike in sales on the last day of the sale, with over $1k in sales on that day alone.

When I bumped the price to $49, sales quickly plummeted, which I expected. I plan to experiment more with pricing after I get to the book’s official 1.0 release.

I’m fighting the world’s dumbest scraper bot

I published my blog post on Monday and got a big jump in visitors. Then, Tuesday, there was another huge surge:

I saw the big jump in visitors and thought some popular blogger linked to me, and then I realized it had to be bots.

All the visitors were going to the Hacker News Popularity Contest. It’s happened before where a blogger talks about their rank in the contest and links to my tool, and I see a big jump in visitors, but never like this, and never such a sustained wave of visitors.

When I saw the second spike, I thought, “Wow, I’m on a roll this week!” I was telling my wife about it when I had a startling realization:

I got another fifty thousand visitors today! I’m not even sure where they’re coming from. My guess is someone tweeted it.

Hmm, actually, if someone tweeted it, I’d see Twitter as the referrer.

Oh… It’s bots.

And I checked the logs and saw that all of the requests had the exact same browser user agent:

That’s Chrome 125, a browser version that’s over two years old. The browser version made it easy to identify the bots, as it’s unlikely that any human users who read my blog still use a browser that old.

The scraper seemed to just repeatedly load the root contest page and then click every link over and over again, so I tried getting sneaky with it. I put a rewrite script on Bunny that checked the user agent and returned a fake response if it was the scraper. So instead of generating a page with 3,000+ links, it would generate a page with only three links.

For whatever reason, my fake responses didn’t work. I suspect the scraper had already added the full set of URLs to a database, and so hiding them from the homepage didn’t do anything.

I tried rate-limiting, but the lowest rate limit Bunny supports is 1 KB/s, and most of the app’s data is in files that are only a few KB each. The scraper was rotating around 100ish different IP blocks, so I couldn’t throttle by IP.

Finally, I just blocked by IP range. I vibecoded a tool that scraped my Bunny logs for the specific user agent and collected all the IPs associated with the attack. That worked, but then the attack started up again a week later from new IPs and a new, slightly more recent user agent, so I just re-ran my script and updated my list of IPs to block, and that seems to be working.

The weirdest thing about the attack is that the bots don’t care about being blocked. They just keep hammering the server anyway. I’d expect them to say, “Oh, no use wasting compute and bandwidth on requests that have 100% been blocked at the TCP level for the past two weeks,” but they don’t mind apparently.

Even after I block the attackers’ IPs, they just keep hammering away.

I reached out to Netlify support the day the attack began, but they were useless. I had to wait a week for each response. I think the first response was AI-generated because it told me to modify settings that didn’t exist. And then the second response seemed more human, but it basically said, “It looks like you solved this problem in the two weeks it took for me to respond, so nothing left for me to do!” Fortunately, they did refund me the $55 in overage fees after I asked.

Where can I host a static site?

I host all my static sites with Netlify, and they’ve been getting progressively worse, but their complete indifference to the scraper bot attack has inspired me to find a vendor that will handle scraper bots more proactively.

The obvious answer is “Cloudflare,” but I’m alarmed at how much of the Internet’s infrastructure has centralized around Cloudflare, so I don’t want to centralize it further.

I also considered just hosting on a VPS or a VPS + Bunny as a CDN, but I don’t want my site to go offline the day I’m on the front page of Hacker News because my VPS crashes or I misconfigure caching on Bunny. I want a solution where I just pay someone else to keep my site online.

• Surge
• Pros
• Focused exclusively on static hosting, which is exactly what I want
• Unlimited bandwidth, so they assume the cost of scraper bot attacks
• Cons
• Run by a single person (I think), so increased outage risks
• All management is through their terminal app. There’s no web app
• It doesn’t look like they support multi-factor authentication, though it looks like they’re working on it
• The upload process unconditionally uploads every file rather than an rsync-like sync of only the changed files, which is a pain for my large sites that only change incrementally
• statichost
• Pros
• Run by a single person, so customer service is responsive and comprehensive
• Focused mainly on static hosting without extra complexity
• Cons
• Run by a single person, so increased outage risks
• The upload process unconditionally uploads every file rather than an rsync-like sync of only the changed files, which is a pain for my large sites that only change incrementally
• Bot scraper protection is not included
• Bundles together site builds and hosting, but I only want hosting
• A big selling point is being EU-centric, but I’m in the US
• Vercel
• Pros
• Claims to prevent DDoS / scraper bots
• I think they support rsync-style uploads
• Cons