Page 1 of 1
Robots.txt Hits in Logs
Posted: Thu Jan 06, 2022 1:13 pm
by gabbysplaza
Hello,
I just wanted to know if a robots.txt file is needed for shop installation as I see a lot of hits for /robots.txt in my logs but I haven't one. I don't even know if one is needed or what it does. Please can you advise if one is needed and what should be in it?
Re: Robots.txt Hits in Logs
Posted: Thu Jan 06, 2022 1:38 pm
by heatherbell
gabbysplaza wrote: ↑Thu Jan 06, 2022 1:13 pm
Hello,
I just wanted to know if a robots.txt file is needed for shop installation as I see a lot of hits for /robots.txt in my logs but I haven't one. I don't even know if one is needed or what it does. Please can you advise if one is needed and what should be in it?
It is not required but you can find out a lot more by internet searching, for example, "is robots.txt needed".
Re: Robots.txt Hits in Logs
Posted: Thu Jan 06, 2022 1:41 pm
by gabbysplaza
heatherbell wrote: ↑Thu Jan 06, 2022 1:38 pm
gabbysplaza wrote: ↑Thu Jan 06, 2022 1:13 pm
Hello,
I just wanted to know if a robots.txt file is needed for shop installation as I see a lot of hits for /robots.txt in my logs but I haven't one. I don't even know if one is needed or what it does. Please can you advise if one is needed and what should be in it?
It is not required but you can find out a lot more by internet searching, for example, "is robots.txt needed".
Thank you
@heatherbell
I'd started looking at that already and whilst many websites use it, in particular sites that run wordpress, if you say it's not required by Phoenix then I'd just ignore it. I don't even know why I'd be getting requests for the file in first instance.
Re: Robots.txt Hits in Logs
Posted: Thu Jan 06, 2022 1:47 pm
by heatherbell
gabbysplaza wrote: ↑Thu Jan 06, 2022 1:41 pm
I'd started looking at that already and whilst many websites use it, in particular sites that run wordpress, if you say it's not required by Phoenix then I'd just ignore it. I don't even know why I'd be getting requests for the file in first instance.
You'll see that internet search results will give differing opinions as to its usefulness.
Unless you have specific instructions to put in the file (which will sometimes be ignored by bots in any case) you don't require one.
You obviously have search bots looking for it in case you do have one.