Page 1 of 1
PerplexityBot/1.0
Posted: Sun Aug 31, 2025 7:53 am
by radhavallabh
Hi dear;
PerplexityBot/1.0 seems to be scraping the site and causing max connection issues, I added it to the spiders.txt list but still seems to do the same .
any ideas on why is it still scraping content??
Thank you in advance,
Help shall be deeply appreciated ;
Warm Regds./
radhavallabh
Re: PerplexityBot/1.0
Posted: Sun Aug 31, 2025 7:56 am
by burt
Spider.txt does not stop a Bot, it simply stops the Bot from getting a session.
To stop a Bot, you need to use Robots.txt, see
https://datadome.co/bots/perplexitybot/
Re: PerplexityBot/1.0
Posted: Sun Aug 31, 2025 7:58 am
by burt
Re: PerplexityBot/1.0
Posted: Sun Aug 31, 2025 1:52 pm
by ecartz
Perplexity may not be scraping the site. It also works on a user operated basis. E.g. a person asks "What's good on radhavallabh's site?" and it takes a look.
CloudFront's free tier might help with Perplexity if the suggestions from the AI that Burt posted don't work.
Re: PerplexityBot/1.0
Posted: Mon Sep 01, 2025 4:52 am
by radhavallabh
ecartz wrote: ↑Sun Aug 31, 2025 1:52 pm
Perplexity may not be scraping the site. It also works on a user operated basis. E.g. a person asks "What's good on radhavallabh's site?" and it takes a look.
CloudFront's free tier might help with Perplexity if the suggestions from the AI that Burt posted don't work.
Understood dear thank you so much for the valuable advice!
Very Warm Regds./
radhavallabh