|
Warning about new msn search (pg. 2)
|
View this Thread in Original format
| Durafei |
| quote: | Originally posted by malek
dude are you serious?
as if googlebot is better, 17% of the traffic on my site is due to their stupid bot!!!!! thats almost a thousand page view a day...
who's paying for the bandwidth, me:rolleyes:
if you wanna bitch, bitch at the google bot too :toothless |
So Google bot requests 1000 pages from your site EVERY day ? |
|
|
| loconet |
| quote: | Originally posted by rabbitjoker
Their are -reasonable- explanations as to why
|
to get hit every few minutes? what explanations are those?
| quote: |
If you took away the tz and gzip pound - there would be no issue. Drop a filetype exclusion into the mix and problem solved. |
Why should I have to do extra work because of their "ways" of implementing this thing? And yes it would have been an issue if I didn't check the logs today. .. how many people check the logs everyday? Leave it for a few days and if you have big enough files, you're going to pay for it. |
|
|
| malek |
| quote: | Originally posted by Durafei
So Google bot requests 1000 pages from your site EVERY day ? |
yes, let me get more details. |
|
|
| Durafei |
As much as I don't want to defend MS, you have to remember that this is still a beta test(in particular they are probably testing their crawler right now) and I'm sure those multiple crawls are going to be gone once they release the final product.
It's a pity that they chose to use the whole web as their test input, but if they want to compete with Google they must use whatever means necessary. |
|
|
| Durafei |
MALEK... wait a second, wait a second..
You are using AdSense right ?? Well, you have to rememember that Google has to preprocess your pages to display those targeted ads you see on your pages. It probably crawls your site so often to ensure that you continue getting fresh and relevant advertisements(this is my opinion only, i don't actually know exactly how AdSense works) |
|
|
| rabbitjoker |
| Hey loco - I seriously suggest you revise your robots.txt setup. I highly suspect that that is the cause of your issue. |
|
|
| malek |
Top 10 Visitor
Google Bot 66.249.66.7 10368
Google Bot 64.233.173.139 9528
Google Bot 64.233.173.146 8156
Google Bot 66.249.66.17 7269
Google Bot 64.233.173.217 5785
Google Bot 64.68.83.140 5770
Google Bot 66.249.65.243 5132
Google Bot 66.249.66.165 4721
Google Bot 66.249.66.80 3954
Google Bot 64.68.81.76 3947
the far right number is the page views, thats in almost 4 months!!!
but i'll rectify, google isn't the only one to blame, inktomi/hotbot is pretty agressive too :mad: |
|
|
| dEsidEL |
lol.. i found the Reaper bot in Quake 1 to be one of the most aggressive .. :toothless :whip:
|
|
|
| Durafei |
| quote: | Originally posted by malek
the far right number is the page views, thats in almost 4 months!!!
|
| quote: |
You are using AdSense right ?? Well, you have to rememember that Google has to preprocess your pages to display those targeted ads you see on your pages. It probably crawls your site so often to ensure that you continue getting fresh and relevant advertisements(this is my opinion only, i don't actually know exactly how AdSense works)
|
|
|
|
| loconet |
| quote: | Originally posted by rabbitjoker
Hey loco - I seriously suggest you revise your robots.txt setup. I highly suspect that that is the cause of your issue. |
Yah i was tweaking it, but no I didn't even have a robots.txt before. |
|
|
| rabbitjoker |
The robots.txt is all exclusionary, so.... categorize the gz, tar and other large datafiles into seperate directories and exclude them in the robots.txt
OR
You can use meta-tag exclusion on the specific pages where the links to those files are. No moving of files required (just note that the meta tag exclusion will impact other hrefs within the document as well).
----
The both do the same thing - all the major bots support both types. |
|
|
| ShadoWolf |
| The internet will implode due to spybots. |
|
|
|
|
