FUAI

[MyGamesShop]

With the advent of AI training my site is inundated with bad bot crawlers.
Thousands of sessions maxing out Mysql server delaying responses. Maxing CPU etc

Is there a plugin after visiting 50 pages or so (reasonable viewing) or multiple sessions for the same IP that could throw up a 'Are you Human' checkbox.
With whitelist for good bots, Google etc.

Or just stuff them up with a genetic product page information touting how good the store is, so they only Learn that and regurgitate it to the world? (Making the site unattractive to scrape)

[burt]

Difficult. The only thing I can think of would be to force the "customer" to login once the session reaches X pages, which would mean these bots get stuck at login.php

You don't want good bots (eg google) to stop working though. So it's difficult.

You could just ban unwanted IP addresses?

[heatherbell]

Internet search "How to Block Bad Bots and Spiders using .htaccess" gives a variety of methods

[MyGamesShop]

Thank-you, I do host myself so I am capable of blocking IP's thou IP-tables.

So let me explain why IP-blocking and .htaccess are not the solution.

Bad bots create sessions sometimes 1 for each product visited and also a new 'Guest' viewable in the 'whos' online'
My session log was 532mb in my SQL db. 'whos online showed 1400 Guests. That's a long SQL or .htaccess lookup time for each bot visiting 1 product in a store with lots of product, bringing the site to its knees, effecting real customers log-in experience..

So IP blocking is also not the answer as there are Four Billion unique IP4 address to ban, even blocking by IP range lets say for Chinese hackers using Tencent or Digital Ocean Cloud it starts to also become a big lookup table that takes away time to connect, delaying real customers. Lets not even get into IPv6.

Remember each bot is reconnecting for each product so that lookup table is being absolutely smashed.

So while I'm not asking anybody to do anything here, but just keep it in mind.
With a plug-in as suggested, Phoenix would be a very unattractive product for trawlers, scrapers, crawlers and AI.
And we may be the first to have it.


Just saying.

[MyGamesShop]

Yay!! I have found just the thing right here....mostly
https://perishablepress.com/blackhole-b ... e-download

However...
I need a Header and Footer module so I don't have to hack the core of my system.
Or could you tell me where to place the two lines of code...

Pleeeeeeease......Somebody make this a module so we can all enjoy

[14Steve14]

We have been using their 7g Firewall for years now with no problems. Never needed the black hole thing but if you feel you need it why not contact one of the certified developers and have it coded for you.

[burt]

I'd be slightly reticent to use such a system as you might accidentally lock out (eg) the google-bot. If you lock that out, you might find your sales drop substantially.

If you've factored that in and are ready to realise that there might be unintended side-effects from using such a system, it would take perhaps an hour of coding/testing to make this system into a no-core change addon.

It's more or less unlikely that addons just get made anymore, I think that's overall a good thing as it stops the addon area getting full up with crud addons. Usually good addons appear because someone needed it, they had the skill to code it and so it got made, or someone needed it (and paid some coffee/beer for it to be coded). After this, the owner of the code sometimes adds it to the addons area (sometimes for free download, sometimes for paid).

[zipurman]

I am planning on building a phoenix Firewall similar to WordFence https://www.wordfence.com/

Likely not for a few months, but its on my list.

[MyGamesShop]

Thanks for the response Burt and Zip.

Yes Burt I read the downloaded php code and it looks clean and allows those good bots so they don't get blocked.

Also used the test procedure and it works on my server.
Visit this site twice... https://nsgames.com.au/blackhole/

which is the page a bot would follow blindly into if the header and footer were on my pages if it ignores robots.txt a regular user would ..never.. see the links in the header/footer.

BTW its just a test so you can still see the site (I need those modules for the trap)

Although I'm not happy with the block message, I'm sure there would be a way to unblock the Contact_us page (which would be blocked if I had the modules) so a blocked IP if ..ever.. got trapped would not need a proxy and be a link to it there so a real user could send an email request.

I looked at the Phoenix tutorial to add a module for a footer kindly supplied in the Phoenix forums.
But it seems to need a header and footer module and I'm not a coder, I'm sure I can work it out over time.
As it seems relatively easy, I just don't have time and I'm old, and a little confused where/how to put



I will remove your bans after a bit, until I can work out the footer/header code to automate it, then ..bad Luck. you will need to send an email..

Ta moochly Mark

This is the bit I can't do: or maybe I can, I don't know.

Step 4: Include the Blackhole script by adding the following line to the top of your pages (e.g., header.php):

<?php include(realpath(getenv('DOCUMENT_ROOT')) . '/blackhole/index.php'); ?>

The Blackhole script checks the bot’s IP address against the blacklist data file. If a match is found, the request is blocked with a customizable message.

Step 5: Add a hidden link to the /blackhole/ directory in the footer of your site’s web pages (replace “Your Site Name” with the name of your site):

<a rel="nofollow" style="display:none" href="https://example.com/blackhole/" title="Do NOT follow this link or you will be banned from the site!">Your Site Name</a>

This is the hidden trigger link that bad bots will follow. It’s currently hidden with CSS, so 99.999% of visitors won’t ever see it. Alternately, to hide the link from users without relying on CSS, replace the anchor text with a transparent 1-pixel GIF image. For example:

<a rel="nofollow" style="display:none" href="http://example.com/blackhole/" title="Do NOT follow this link or you will be banned from the site!"><img src="/images/1px.gif" alt=""></a>

Remember to edit the link href value and the image src to match the correct locations on your server.

[MyGamesShop]

I'm not associated with this website, I'm not promoting it but being free and open so I like it.
I also added the ht-access code in this post to my sever running Phoenix with no dramas.

https://perishablepress.com/8g-firewall/
https://perishablepress.com/8g-firewall ... e-attacks/

Some google protection for your users
https://perishablepress.com/opt-out-goo ... via-apache

I'm not sure if this helped but anyone under attack it might help.

Use the 7g wirewall as some of the code mucks up images etc. (updated 8.01.2025)