Facebook Crawler

[ReneH4]

For the past months I have noticed the Facebook crawler hitting my site with calls.

I have blocked the crawler with IPv6, but removed it july 19th.
Yesterday I received a message from my provider that CPU had gone up to an inacceptable level, causde by the crawler.

Anyone else who has noticed this?

The blockingcode that I use now is:

Code: Select all

 RewriteEngine On
# Block Facebook bot
RewriteCond %{HTTP_USER_AGENT} ^facebookexternalhit/1\.1  [NC]
RewriteRule ^ - [F,L]

[burt]

There has been some chitchat in the Pro forum about Amazon Bot eating lots of CPU. The `amazon bots` are not from amazon, but are hosted on amazon AWS (so it looks like it is Amazon).

There has been no chitchat about Facebook so far as I am aware.

[ReneH4]

This is the bandwidth graph of my site. Look at the arrows where the problems started..... and ended...

[zipurman]

This is why I wrote the Phoenix Firewall

app.php/addons/paid_addon/phoenix_firewall/

Seems to be happening a lot lately.

[radhavallabh]

There has been some chitchat in the Pro forum about Amazon Bot eating lots of CPU. The `amazon bots` are not from amazon, but are hosted on amazon AWS (so it looks like it is Amazon).

There has been no chitchat about Facebook so far as I am aware.

Hi dear;
any idea on how to go about blocking these amazon AWS hosted bots, as banning IP's has no effect on them.
Any help input shall be deeply appreciated dear;
Thank you in advance;
Very Warm Regds.
radhavallabh

[Alice777]

I also have noticed the Facebook DOS attack. I assume they are scraping sites for marketing purposes. At any rate, I've blocked AS32934 FACEBOOK using Cloudflare free plan. I've also blocked all Amazon cloud ASN's and other cloud providers to mitigate these constant pests.

[Yahalimu]

Probably go scrape data for their Llama AI. I blocked it via htaccess.

[ReneH4]

Never thought of that, you're probably correct!

Probably go scrape data for their Llama AI. I blocked it via htaccess.