Inspired by this post commentary by Kev Quirk, I decided to look into my own logfiles of the past months and see how much bot visits I’m getting.
Some findings:
- No, bots are not eating my blog, two thirds of all requests seem human.
- But boy, there are a lot of bots around nowadays!
- Surprisingly, Googlebot isn’t the hungriest bot. It’s OpenAI/ChatGPT!
- Strong contender: Ahrefsbot – they’re really nosy! I can see why you would pay for their service, though, given that browsers generally don’t send referer headers anymore.
- Bytespider, operated by TikTok’s parent company ByteDance, also seems really into Amiga, demoscene, and Teletext content.
- Of course, all of this can only take bots into account that play nice and identify themselves as bots.
The top user agents of the past months are:
| # | Hits | Percentage | Identified as |
|---|---|---|---|
| 1 | 316,944 | 66.3% | User agent looking like a human-operated browser |
| 2 | 22,622 | 4.7% | openai.com |
| 3 | 21,574 | 4.5% | GoogleBot |
| 4 | 15,075 | 3.2% | AhrefsBot |
| 5 | 11,407 | 2.4% | Scrapy |
| 6 | 9,140 | 1.9% | Bytespider |
| 7 | 6,474 | 1.4% | bingbot |
| 8 | 5,815 | 1.2% | babbar.tech |
| 9 | 5,105 | 1.1% | ClaudeBot |
| 10 | 4,895 | 1.0% | Amazonbot |
| 11 | 4,298 | 0.9% | facebook.com |
| 12 | 4,185 | 0.9% | Applebot |
| 13 | 3,666 | 0.8% | PetalBot |
| 14 | 3,532 | 0.7% | SemrushBot |
| 15 | 3,449 | 0.7% | ImagesiftBot |
| 16 | 2,564 | 0.5% | MJ12bot |
| 17 | 2,401 | 0.5% | DuckDuckBot |
| 18 | 2,134 | 0.4% | BLEXBot |
| 19 | 1,961 | 0.4% | Empty (“-”) |
| 20 | 1,428 | 0.3% | YandexBot |
| 21 | 1,118 | 0.2% | curl |
| 22 | 1,100 | 0.2% | Dalvik |
| 23 | 1,058 | 0.2% | VelenPublicWebCrawler |
| 24 | 1,038 | 0.2% | Timpibot |
| 25 | 1,001 | 0.2% | serpstatbot |
| 26 | 970 | 0.2% | AwarioBot |
| 27 | 967 | 0.2% | SeznamBot |
| 28 | 963 | 0.2% | DotBot |
| 29 | 881 | 0.2% | DataForSeoBot |
| 30 | 870 | 0.2% | coccocbot |
The bot-vs-human ratio roughly ends up like this:
| Percentage of all requests | Category |
|---|---|
| 66.3% | Probably human |
| 33.7% | Bot |
So… go humans, I guess? That is, unless we filter out all file requests (CSS, images, other media). Then we get:
| Percentage of page requests | Category |
|---|---|
| 50.8% | Bot |
| 49.2% | Probably human |
In any case it was lovely to see some Amiga users visiting with IBrowse on AmigaOS 3.2 and 3.9! And I guess the “PSP (PlayStation Portable)” user agent is a nerdy joke? :)
