|
Warning about new msn search
|
View this Thread in Original format
| loconet |
Because we can't have any geek threads, here is another intersting one.
Many of you may know about the new attempt by Microsoft at a search engine (http://beta.search.msn.com) to compete. Besides the fact that it's obviously a ripoff of this other popular search engine with some eye candy added in, this thing is truly irritating me.
In the last few hours my personal site has recieved a massive surge in retarded traffic from its indexing bots, similar to a drunk man knocking the wrong apartment door over and over again. It's getting raped. Check this out..
code:
date filename ip
| 2004-11-09 14:25:37 | permit-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 10:32:15 | cdp-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 06:25:07 | sync.X-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 06:19:18 | permit-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 02:51:34 | cdp-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 02:46:07 | cdp-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 02:35:36 | MultiplyWithMFC-1.0_src.zip | 66.249.64.199 |
| 2004-11-09 00:55:05 | sync.X-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 00:12:03 | permit-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 00:10:57 | permit-1.0.tar.gz | 207.46.98.33 |
| 2004-11-09 00:05:21 | sync.X-1.0.tar.gz | 207.46.98.33 |
| 2004-11-08 21:03:10 | permit-1.0.tar.gz | 207.46.98.33 |
Note that 207.46.98.33 is registered to Microsoft so lets assume it's the msn bot. Notice that the damn thing blindly keeps on downloading the same files over and over again. The requests are just a few minutes from each other. I don't mind content getting indexed but this thing is almost a Denial of Service. The other ip address is from google's bot who is more polite.
These files are not the only ones getting hit, also lots of images. I already recieved a msg from the Sysadmin to chill on the bandwith usage, and at work we've blocked the bot from our servers that hosts more popular sites. So everyone out there who hosts images, large files, etc be careful with this stupid bot.. tweak your robots.txt or this pest might ruin your month's transfer cap. ;)
From the Faq..
| quote: |
MSNBot is crawling a site too often
When MSNBot is crawling your site, it generally does not try to access your site more often than once every few seconds. If MSNBot determines that your site has a slow connection, it automatically adjusts the frequency. To specify a minimum frequency (in seconds), use the crawl delay parameter in the robots.txt file:
user-agent: msnbot
crawl-delay: 120
|
How about being polite with the damn indexing by default? .. BLAH! |
|
|
| dallastar |
dear me oh my!!!
strange shizzles goin' on here i guess... i just
HATE pop ups---god damn things!
and rip off's
:rolleyes: |
|
|
| dEsidEL |
wells if everyone continues using Google's search u can permanently block the MSN bot and not hav to worry ..
let's hope to that ! ;)
|
|
|
| rabbitjoker |
you're totally awesome, but your anti-MS drabble is really bizarre dood.
you should go get a picket sign or something or join the anti-MS club. |
|
|
| loconet |
| quote: | Originally posted by rabbitjoker
you're totally awesome, but your anti-MS drabble is really bizarre dood.
|
but with a reason no? :p |
|
|
| cereal3825 |
| quote: | Originally posted by loconet
These files are not the only ones getting hit, also lots of images. I already recieved a caution from my the Sysadmin to chill on the bandwith usage, and at work we've blocked the bot from our servers that hosts more popular sites. So everyone out there who hosts images, large files, etc be careful with this stupid bot.. tweak your robots.txt or this pest might ruin your month's bandwidth cap. ;)
From the Faq..
How about being polite with the damn indexing by default? .. BLAH! |
code:
while(1) {
for(int i=0;i < sizeof(internet);i++)
{
wget -r -l 1024 internet
if(crawldelay>5)
usleep(1);
}
}
Well based on their OS and other software, I am sure it's a bug, problem is they dont have the customers debugging their spider like they do their OS
- 'Another program has crashed would you like to send a report to Microsoft'
steve |
|
|
| rabbitjoker |
MSNBot hits my sites 2 to 3 times more often then GoogleBot.
Their are -reasonable- explanations as to why, but ok - phear if it makes you happy.
Mind you - pounding your tz and gzip stuff isn't very nice. If you took away the tz and gzip pound - there would be no issue. Drop a filetype exclusion into the mix and problem solved. |
|
|
| DigiNut |
| RJ, do you work for Microsoft in some capacity? :p |
|
|
| malek |
dude are you serious?
as if googlebot is better, 17% of the traffic on my site is due to their stupid bot!!!!! thats almost a thousand page view a day...
who's paying for the bandwidth, me:rolleyes:
if you wanna bitch, bitch at the google bot too :toothless |
|
|
| rabbitjoker |
| I do not work for MS or Google. |
|
|
| cereal3825 |
| quote: | Originally posted by malek
dude are you serious?
as if googlebot is better, 17% of the traffic on my site is due to their stupid bot!!!!! thats almost a thousand page view a day...
|
you must not get alot of traffic ;)
jk
Steve |
|
|
| malek |
| quote: | Originally posted by cereal3825
you must not get alot of traffic ;)
jk
Steve |
haaaahahaah
:whip: :whip: :whip: |
|
|
|
|
