Page 1 of 1
Facebook Crawler
Posted: Thu Jul 25, 2024 7:06 am
by ReneH4
For the past months I have noticed the Facebook crawler hitting my site with calls.
I have blocked the crawler with IPv6, but removed it july 19th.
Yesterday I received a message from my provider that CPU had gone up to an inacceptable level, causde by the crawler.
Anyone else who has noticed this?
The blockingcode that I use now is:
Code: Select all
RewriteEngine On
# Block Facebook bot
RewriteCond %{HTTP_USER_AGENT} ^facebookexternalhit/1\.1 [NC]
RewriteRule ^ - [F,L]
Re: Facebook Crawler
Posted: Thu Jul 25, 2024 8:27 am
by burt
There has been some chitchat in the Pro forum about Amazon Bot eating lots of CPU. The `amazon bots` are not from amazon, but are hosted on amazon AWS (so it looks like it is Amazon).
There has been no chitchat about Facebook so far as I am aware.
Re: Facebook Crawler
Posted: Thu Jul 25, 2024 11:28 am
by ReneH4
This is the bandwidth graph of my site. Look at the arrows where the problems started..... and ended...
Re: Facebook Crawler
Posted: Thu Jul 25, 2024 4:09 pm
by zipurman
This is why I wrote the Phoenix Firewall
app.php/addons/paid_addon/phoenix_firewall/
Seems to be happening a lot lately.
Re: Facebook Crawler
Posted: Fri Aug 02, 2024 11:28 am
by radhavallabh
burt wrote: ↑Thu Jul 25, 2024 8:27 am
There has been some chitchat in the Pro forum about Amazon Bot eating lots of CPU. The `amazon bots` are not from amazon, but are hosted on amazon AWS (so it looks like it is Amazon).
There has been no chitchat about Facebook so far as I am aware.
Hi dear;
any idea on how to go about blocking these amazon AWS hosted bots, as banning IP's has no effect on them.
Any help input shall be deeply appreciated dear;
Thank you in advance;
Very Warm Regds.
radhavallabh
Re: Facebook Crawler
Posted: Sat Aug 03, 2024 5:41 pm
by Alice777
I also have noticed the Facebook DOS attack. I assume they are scraping sites for marketing purposes. At any rate, I've blocked AS32934 FACEBOOK using Cloudflare free plan. I've also blocked all Amazon cloud ASN's and other cloud providers to mitigate these constant pests.
Re: Facebook Crawler
Posted: Sun Aug 04, 2024 9:31 pm
by Yahalimu
Probably go scrape data for their Llama AI. I blocked it via htaccess.
Re: Facebook Crawler
Posted: Mon Aug 05, 2024 6:03 am
by ReneH4
Never thought of that, you're probably correct!
Yahalimu wrote: ↑Sun Aug 04, 2024 9:31 pm
Probably go scrape data for their Llama AI. I blocked it via htaccess.