Getting unwanted request from AWS IP with bytespider

Question

Hi Team,

In my Ec2 instance, they have many request from AWS IP address with different series range and its access log its showing bytespider attack. Could you please tell me the solution for how to fix this issue. For your clarification, Please check below log.

47.128.53.162 - - [01/Apr/2024:03:12:11 +0530] "GET /images/mirusports_no_image.png HTTP/2.0" 200 5343 "https://www.mirusports.com/product/7a20aea4" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"
47.128.53.162 - - [01/Apr/2024:03:12:11 +0530] "GET /images/products/vicky/7a20aea4/default/t/1667970674227.jpeg HTTP/2.0" 200 2005 "https://www.mirusports.com/product/7a20aea4" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"
47.128.53.162 - - [01/Apr/2024:03:12:13 +0530] "GET /images/products/vicky/7a20aea4/default/1663836295.jpg HTTP/2.0" 200 98304 "https://www.mirusports.com/product/7a20aea4" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"
47.128.53.162 - - [01/Apr/2024:03:12:13 +0530] "GET /images/products/vicky/7a20aea4/default/t/1667970665886.jpeg HTTP/2.0" 499 0 "https://www.mirusports.com/product/7a20aea4" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"
47.128.53.162 - - [01/Apr/2024:03:12:13 +0530] "GET /images/products/vicky/7a20aea4/default/t/1667970652224.jpeg HTTP/2.0" 499 0 "https://www.mirusports.com/product/7a20aea4" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"
47.128.53.162 - - [01/Apr/2024:03:12:13 +0530] "GET /images/products/vicky/7a20aea4/default/t/1663836295.jpg HTTP/2.0" 499 0 "https://www.mirusports.com/product/7a20aea4" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"
47.128.53.162 - - [01/Apr/2024:03:12:13 +0530] "GET /images/products/nivia/handball/balls/3d306949/default/l/1651473237213.png HTTP/2.0" 499 0 "https://www.mirusports.com/product/7a20aea4" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"
47.128.27.247 - - [01/Apr/2024:03:12:15 +0530] "GET /product/14518590 HTTP/2.0" 200 4703 "-" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"
47.128.115.66 - - [01/Apr/2024:03:12:19 +0530] "GET /product/1cafec68 HTTP/2.0" 200 5001 "-" "Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36" "-"

Please check and let me know.

Answer

Hello.

If you are using Apache as your web server, you can deny access from Bytespider by configuring the following settings in .htaccess.  
The following settings were actually set on a blog site that I manage, but so far there is no access from Bytespider.  
```
SetEnvIf User-Agent "Bytespider" bothersome_bot
BrowserMatchNoCase "Bytespider" bothersome_bot

RequireAll Granted
Require not env bothersome_bot

```

If you are using Nginx, the following URL settings may be helpful.  
https://gist.github.com/hans2103/733b8eef30e89c759335017863bd721d

Since user agents can be spoofed, it may not be possible to block them with this setting alone in the future.  
If blocking becomes difficult due to Apache settings, etc., I think it would be a good idea to introduce WAF.

Getting unwanted request from AWS IP with bytespider

Relevant content