, hg- > Shanghai dragon crown
through the web site we can directly log server response code page which we see, what normal. In general the status code returned is 200 words normal, 404 words, that page problem.
3, 123.125.68.*ip section of the Shanghai love spiders (investigation of spiders)
4, 117.28.255.*ip section of the Shanghai spiders love (fake spider)
third, test page state is normal or not
2, through the web log analysis tools to watch, the more suitable for beginners to use
2, 123.125.71.*ip section of the Shanghai spiders love (inferior article catch spiders)
from the web log, we can come to our site spider crawling number that directly, the more that the spider crawling more friendly to our site.
through the web log analysis tools can directly view the site which page has been a spider crawling.
through the above we can count on our site to see the spider crawling directly, but the number of crawling inside there is a fake spider, so we need to make sure what is the real spider through customer IP, which is a fake. This can refer to "understand the web log analysis, web site more secure" there is a graphic tutorial, how to distinguish between true and false spider, here is not introduced.
1, by watching the web log code for viewing, the analysis of master
Through the above three points to
second, IP spider type customer glance.
fourth, on the site of the search engine friendly degree
website optimization, he slowly to focus on Web log. As the first thing they work every day of the web log analysis, to the company every day is the first time of yesterday’s Web log analysis. Maybe some webmaster also does not analyze the web log, this can refer to before the crown Xiaobian published "smatters Web log analysis, website security", but there are few think web log is a waste of time, every day looking at what is the role of the data. Here the crown Xiaobian think at least we can see four points.
Since he engaged in Shanghai dragon
1, 220.181.108.*ip section of the Shanghai spiders (love right spiders)
how to determine whether there is a spider crawl over
, the first to determine whether there is a spider crawl over