Spiders and Indexation: Insights on How Spiders Index and TrustRank
I've been reading and researching on search engine spiders and the indexes lately, which has led me to some insights on spidering/indexing, that I'd like to share.
First, many people only consider being indexed in terms of quantity: that is, how many of their pages are currently accounted for in the search engines. There's another very important point to know about indexation, and that is the frequency with which your website is crawled.
I ran an SEO experiment once to try and optimize for the toughest words on the Internet: home, about, and click here. They're the toughest because almost every website around competes for them, since most people use those words on their websites. The reason I bring this up is to highlight what I noticed in the search results. Those sites with constantly fresh content (i.e. those getting spidered something like every hour) were ranking highest, and the ranks often changed according to who had updated most recently/been crawled most recently. It should come as no surprise that news giants such as the New York Times ranked for these words.
Second, over at the Link Building blog, there's an interesting insight into Google's TrustRank. TrustRank, for those who don't know, is a parallel factor in Google's algorithms to PageRank. You can't achieve any success at SEO if your TrustRank factor is in the basement.
At any rate, what came out of the LB blog's analysis was that a thorough indexation was a sign of being trusted. Furthermore, the speed with which Google's spiders (Googlebot et al.) index your site is also an indicator of TrustRank. This is important knowledge, because Google doesn't publicly disclose what a site's TrustRank is, as it does with PageRank. In short, it allows you to monitor the results of your SEO efforts.
There are a few conclusion to be drawn here. Monitoring the depth of indexation isn't sufficient, especially in competitive keyword markets. The frequency of spidering is as important as the depth. That's the main lesson. The second lesson is that depth of indexation is important beyond simply counting how many searches you might rank for. Having 100 pages that could rank for 100 various searches is nice (assuming you only have 100 pages on your site). But if all your pages are indexed, it also shows your website is trusted.