robots.text or noindex

[14Steve14]

On my 1.0.7.11 site I use a robots.txt text file to try to control the bots as best as that file can. I notice that in the later versions of Phoenix there is a ht_robot_noindex.php headertag module.

Can this file be used on older versions if the robots.txt file is removed.

I only ask as I was reading on google search console that its better to noindex that to use robots.txt.

What would be the worse thing that could happen if I tried.

[ecartz]

1. They do different things. Robots.txt gives directions to robots as to how crawl your site. The noindex tells search indexes not to include a page. If you want to not include a page in search indexes, you should use noindex. But if you want to control how robots crawl the site other than not indexing a page, you should use robots.txt.
2. You can use both at once. You don't have to remove the robots.txt in order to use noindex.
3. ht_robot_noindex.php was in Gold and osC (not new in Phoenix): https://github.com/osCommerce/oscommerc ... oindex.php
4. You can only use header tag modules on pages in the cart. They won't work on image files and other static content.

[burt]

That noindex module will show something like;

Code: Select all

<meta name="robots" content="noindex,follow" />

in the underlying code (source code) on pages that you set. Once you have set those pages, visit a few of your important pages that you have NOT set to noindex, eg index.php / product_info.php

That line of underlying code should NOT appear on pages that you have NOT set it to show on. If it does show on any pages that you have NOT set it to show on, there will be complications down the line (eg google deindexing important pages).

Hope that all makes sense ?

[14Steve14]

That noindex module will show something like;

Code: Select all

<meta name="robots" content="noindex,follow" />

in the underlying code (source code) on pages that you set. Once you have set those pages, visit a few of your important pages that you have NOT set to noindex, eg index.php / product_info.php

That line of underlying code should NOT appear on pages that you have NOT set it to show on. If it does show on any pages that you have NOT set it to show on, there will be complications down the line (eg google deindexing important pages).

Hope that all makes sense ?

Cheers Gary. That will give me something to do tomorrow.

[lecarlb]

That noindex module will show something like;

Code: Select all

<meta name="robots" content="noindex,follow" />

in the underlying code (source code) on pages that you set. Once you have set those pages, visit a few of your important pages that you have NOT set to noindex, eg index.php / product_info.php

That line of underlying code should NOT appear on pages that you have NOT set it to show on. If it does show on any pages that you have NOT set it to show on, there will be complications down the line (eg google deindexing important pages).

Hope that all makes sense ?

Cheers Gary. That will give me something to do tomorrow.

@burt I have a question. For pages that are NOT checked but still receive a noindex from Google, what could be the cause? And what would be a fix? Here's an example:

Code: Select all

https://mysite.com/cuba-jungle-snake-by-fragluxe-eau-de-parfum-spray-115-oz-for-women-fragluxe-fragluxe-403004-55938-pr-4994.html

That's a review page. When I clicked on the link it took me to the login page. I'm guessing it has something to do with guess writing/viewing reviews.

Thanks.

[burt]

@burt I have a question. For pages that are NOT checked but still receive a noindex from Google, what could be the cause? And what would be a fix? Here's an example:

Code: Select all

https://mysite.com/cuba-jungle-snake-by-fragluxe-eau-de-parfum-spray-115-oz-for-women-fragluxe-fragluxe-403004-55938-pr-4994.html

That's a review page. When I clicked on the link it took me to the login page. I'm guessing it has something to do with guess writing/viewing reviews.

I don't know. It does sound like the redirect to "login" would be the issue as if that is happening it does mean that Google would be redirected to login as well (therefore would never see the page as google would not be able to actually login). That would be the first thing to solve.

After that is solved, if the "no index" is still happening;

Phoenix doesn't have Review pages, so if it is an addon, it could be a problem in the addon.
or
It could be a problem in the URL rewriting addon (very unlikely, but you never know).

In any case, the first thing to solve would be that redirect to login. It's likely to be a line of code near the top of the review page, depending on Phoenix version it might look like this;

Code: Select all

  if (!tep_session_is_registered('customer_id')) {
    $navigation->set_snapshot();
    tep_redirect(tep_href_link('login.php', '', 'SSL'));
  }

Or if a more modern version, something like this;

Code: Select all

$OSCOM_Hooks->register_pipeline('loginRequired');

It should be fairly obvious, and very close to the top of the code in the page.