Web Spiders!

Mycroft2152

Senior Member
Under the Quick Links tab there is a "Who's line" buttton. It was very interesting to see that today Nov 11 at noon, that of the 22 people online, 8 were Gooogle and Yahoo web spiders!

Quite an eye opener.
 

MORA99

Senior Member
its a script that collects info for the search engines and what not.
spam-spiders is a script that looks for email adresses to spam :(
 

slurp

Senior Member
I've had the google sipider/bot nearly break a website downloading almost 5Gb from a 37Kb php script over two days.... I'd added a calendar to a website and google set dozens of bots on the recurring links that dissappeared off into future dates and links with no content :(

I'm just a little more careful about where I let them roam now ;)

I regularly find these 'bots on my own forum, it's good for the users and people finding the site or articles of interest but I guess they're also what drives the spammers to target the site.... Grrr!

regards,
Colin
 

Michael 2727

Senior Member
One or two lines of HTML code at the top of your page
will tell them they are not welcome, if you don't want
them clogging up your site.
They are supposed to check every link and see that
they are correct, in fact if you have a lot of dead
links they are supposed to dump the page and don't
include it in their listing.

In some cases they can be good as your info turns up
in their Engine when searched, thats how you get listed.
You can have the opposite effect if you don't follow
common rules when designing your pages.

I just hope they start removing/replacing links from
the old forum, as they show up as dead since the
new format shifted to here. Can't be good for surfers
if they think every second Picaxe link will be dead.

PS: Just a quick search on Google and it
seems the links to the old Forum have gone. With
some 8,000+ Threads they will have their work cut out
for them setting up the new links.
 
Last edited:
Top