Scraping bots stuck in cookie_usage page

[Pierre_P]

Lately we are under a huge scraping bot attack all over the world.
The cookie_usage.php is doing fine but i do not want our server to allocate unnecessary resources to this bots.
I did modify my whos online page to show the bots name as well and legitimate bots are behaving correctly.

For the bad bots - can we create a basic text only page (no other resource loading)?
What could be the impact to doing this?

[BrockleyJohn]

Have you got buy buttons enabled in your product lists?

If you have bots that you've included in spiders.txt they won't have a session, so if they follow a link with action=buy_now they get redirected to cookie_usage.php

It would save site resources only to display buy buttons if there's a session established (as it's pointless otherwise). There'll be less crawl from all bots including legitimate ones.
You can do this by overriding product_card in your active template.

Afterthought: it will also focus your google crawl budget on more useful urls

[Pierre_P]

@BrockleyJohn

I do not mind the rogue bots stuck in cookie_usage page, google, bing and so on listed in robots file is doing its job.

So then for product_card we can change to likely:

Code: Select all

<?php
    if (Session::is_started() && count($buttons) > 0) {
        ?>
        <div class="card-footer">
            <div class="d-flex justify-content-between"><?= implode(PHP_EOL, $buttons) ?></div>
        </div>
        <?php
    }
?>

Every scraper and bot adds queries the database, pull up images and what else adding server load.
Last week our our hosting company blocked all international traffic i guess server wide - i see it is still ongoing - this severely affected our google services for example.
I'm being frustrated with them(well known hosting company) started realizing there is a serious issue actually.
I found Force Cookie Use was set to false, changes this to true and now them bots is sent to cookie page.

This prompt me to start acting and be more stringent on how bots crawl our site.
The robots.txt is the updated version but we are getting pounded hard.

I also started blocking with htaccess several ip's in the millions range countries that will never buy from our store - china, Vietnam, Russia and few more.

So making changes to the product_card will help a bit but not relieve the complicated issue.

Next i guess will be a WAF approach or go cloudflare

[burt]

viewtopic.php?f=10&t=2986
&
viewtopic.php?f=43&t=3263
&
viewtopic.php?f=39&t=3252

[ArtcoInc]

I did modify my whos online page to show the bots name as well and legitimate bots are behaving correctly.

Can you please share this mod?

Thanks!

M

[Pierre_P]

@ArtcoInc
Pm sent