This forum does not appear in google search results...

[SJC]

I tried this on my pc, work laptop, my iphone and got same results.
(google chrome, Brave, Internet Explorer & Safari browser).

When for example searching "tb42 patrol4x4.co.za" on google.
List of results will show links to patrol4x4.com, 4x4community.co.za etc etc, but never list a link to patrol4x4.co.za.
Did anyone else test this?

So if a new or wannabe patrol owner or anyone for that matter needs some patrol related info, google results will not include a link to this forum.

Maybe this is why the forum is so quite lately?

[SJC]

According to chat-gpt the problem is caused by incorrect configured robots.txt. (chat gpt is not always right, so just throwing it out there...)

robots.JPG (81.79 KiB) Viewed 1305 times

[SJC]

forum admins?

[iandvl]

Fanus - this is on my list of things to do. Along with upgrading the PHP and phpBB and a whole bunch of other things. The forum is actually on Google, but I've got a misconfiguration on the apache daemon which means that it is on IP and not DNS name.

I've also had my hands full with some personal issues (refer my earlier post). Not an excuse, I know.

This forum has also been under a series of DoS attacks, which has resulted in me blacklisting half of the world (exaggerating) - the most recent of which was on Saturday. But until that relents a little, I do not want to make serious configuration changes. Rest assured - it will be sorted soon.

[iandvl]

Right. As I said - I've not had a lot of time on my hands recently.

One of the things I did a month or two ago when we first got hit by DoS conditions was add the following to the site's robots.txt file.

User-agent: *
Disallow: /

This file is used to instruct robots / bots - ie: the automated scripts that crawl the Internet for search results, amongst other things, about what they are allowed to crawl and what they are not allowed to crawl. As such, we've not got "well-mannered" bots indexing the site at the moment. Sadly, in the last few months, the site has had regular issues with availability as various chinese-based AI learning machines do not adhere to the instructions in that specific file. Which results in them consuming vast amounts of resources (bandwidth / CPU / RAM). I get that AI probably wants to learn as much as it can about these awesome vehicles. But they should respect the instructions too. :)

I've now become pretty aggressive these days, and when I gather the bots are consuming too many resources, I blacklist whole IP ranges. So I guess I can revoke the changes to that file now. But I'll do it when I have time - which will be over the weekend. :)

On a side note: if you're in China, and patrol4x4.co.za does not work for you there, you will also probably be able to infer why... :)

My blacklisting currently looks like so:

<RequireAll>
Require all granted
Require not ip 3.0.0.0/9 47.76.0.0/16 52.0.0.0/10 52.64.0.0/12 18.128.0.0/9 18.64.0.0/10 18.32.0.0/11 47.80.0.0/13 47.74.0.0/15 47.76.0.0/14 47.74.0.0/9
</RequireAll>

Edit: And we've actually got more gurest traffic going on from the 47.x.x.x range as we speak. As well as various IPv6 ranges.

[iandvl]

I've fixed any and all config issues that were around. And I've added something to try and mitigate some of the issues we've had. I've also removed the robots.txt file I put up last time in an attempt to stop stuff. So Google results should start propogating again.

However, there are various ways of preventing these sort of attacks (unless you're behing cloudflare or some other service and don't need to worry about it) is possible, but a little tricksy to debug.

In short: I've got a change in place. I'm monitoring stuff. But until I get the config sorted properly, there may be some false positives. If people have issues accessing the forum, please let me know.

[iandvl]

Right. So far everything seems good with the changes. And Google has started indexing the site again.

[SJC]

thanks Ian, we are back...hopefully we can now get more (good) traffic.

[Sweetlips]

HI Ian

It might not be stated enough but what you are doing is greatly appreciated.