Amazon

[14Steve14]

Have a read of this page. It is on of many ways to stop bad bots. https://perishablepress.com/blackhole-bad-bots/ They also do the 7g firewall.

Its up to you how you stop them but there are many ways. You have to trial and error test things to see what works and what doesnt. We find with the 7g firewall most things are stopped but we also have a htaccess blacklist, bad blocker where we can enter IP addresses when we find bad ones. We also have an updated robots.txt file of which there is lots of info online.

It is something that needs keeping on top of.

[burt]

There are two different problems here, so you need to understand what you perceive the problem to be and what it is that you want to do;

1. The sessions table is getting massive
- add the user-agent to the spiders.txt file
This is a Phoenix solution.

And/Or

2. Your hosting might have a limit of how much your site can use.
- ban the IP ranges from your site entirely
This is usually a hosting company solution, done through your hosting control panel.
There is a firewall solution in the Addons area that may help you.

#1 will allow the Spider to access your site, but will keep the sessions table from getting filled up.
#2 will stop the Spider accessing your site, as a by-product of that, your sessions table will stop getting filled up.

[Xpajun]

There are two different problems here, so you need to understand what you perceive the problem to be and what it is that you want to do;

1. The sessions table is getting massive
- add the user-agent to the spiders.txt file
This is a Phoenix solution.

And/Or

2. Your hosting might have a limit of how much your site can use.
- ban the IP ranges from your site entirely
This is usually a hosting company solution, done through your hosting control panel.
There is a firewall solution in the Addons area that may help you.

#1 will allow the Spider to access your site, but will keep the sessions table from getting filled up.
#2 will stop the Spider accessing your site, as a by-product of that, your sessions table will stop getting filled up.

The big problem is Gary @burt is the fact that these robots are totally ignoring the spiders.txt file and deny in .htaccess
The one that started this -Amazon - is using any of the ip addresses between 3.128.00 and 3.225.225.225 I've blocked them all with the CDIR 3.128.00/8 - I'm lucky enough to have VPS so have access to alter settings as needed including firewall

In addition Facebook also have some really bad bots as well ignoring Deny and the spiders.txt as well. I'll post a list of them once I have them

[ecartz]

these robots are totally ignoring the spiders.txt file and deny in .htaccess

They can't ignore the spiders.txt file or Deny in .htaccess. Both those things are on your server and do not operate with consent from the other side. If they aren't working, then there's something wrong. For example, the user-agent may be incorrect in the spiders.txt file. Or the server may not be configured to allow Deny in .htaccess.

They can ignore the robots.txt file. That just has instructions for the robot. The robot can ignore it. But that's just further evidence that this isn't actually Amazon.

Note that the user-agent is something that they have to tell you. It doesn't need to be correct, but they have to say something. Also, all the spiders.txt file does is it blocks them from creating sessions. Your initial problem was the session table filling. It addresses that problem directly by preventing your server from creating sessions when it matches. Requires admin > Sessions > Block Spider Sessions to be true.

The firewall is better than Deny in .htaccess, so if that isn't working, I wouldn't sweat it. Just do it in the firewall.

You can report malicious actions to AWS: https://support.aws.amazon.com/#/contacts/report-abuse

You probably should not start from the perspective that this is an Amazon bot. Instead say things like "A bot from IP 3.152.5.7 identifying itself as user-agent Amazonbot blah blah is effectively DOSsing my site, ignoring the robots.txt file. Here's the logs:" and include logs from the incident. Note that you should use the actual IP and user-agent, not "blah blah" or .152.5.7 (unless that is the actual IP). You should provide specific IPs, not ranges.

[bonbec]

@Xpajun, I have the impression that you are confusing amazonbot (for amazon.com amazon.fr etc...) and AWS (Amazon Web Service) which is a company in the Amazon group and which is a host with hundreds of thousands of servers.

amazonbot respects robot.txt, I have noticed it on my sites.

aws hosts both very good sites and malicious sites that you can block with .htaccess and report (by providing the corresponding logs) to abuse@amazonaws.com