What is a Search Spider?

Simply stated a Search Spider is a Computer Program. Most computers have a software program you can use to find files on your computer. The program you use to do this is a basic search function. Search Engines collect data from all over the web. The program Search Engines use is a much more complex program that looks for information on the internet. Not an actual spider but the program acts like it "owns the web" looking for whatever is out there to take a snap shot of it, (cache) so it can keep track of what is developing on a site over time, to analyze it for content. It looks for links pointing at other websites, like the strands of a spider web that connect the internet.

The Search Engine needs to determine if the website has any relevant information on it. Just because it is built doesn't mean humans will find it. The Search Engine Spider gathers information from millions of sites constantly. The Search Engine then takes the data presented by the Spider and applies a formula (algorithm) to the information.

The Formula is a mathematic equation that has (in the case of Google over two hundred) individual elements, that determine a value of the website. That value is referred to as a Page or Trust Rank. Websites that are determined to be relevant and have established a good history with the Search Engine will be presented at the beginning of the Search Results.


Resistance is Futile

The computer storage space required by the major Search Engines is staggering because they don't just take a snap shot of an individual website, they store the information from that website and assimilate it into it's data base. It then compares several "snap shots" of that same website over time to establish a trust value to the site. Multiply that with all the other sites on the internet and you can begin to understand the challenge.

Couple that with then retrieving those sites, ranking them in importance and relevance for a given search anytime a user like you wants to do a search and you begin to understand the complexity of the task. Since a Search Engine is only good as the results it produces for a search, you can see why some websites don't even get stored in the data base.

The old commercial where as soon as a website went live on the internet, customers began finding it and flooding it with orders was pure fiction. This is not "Field Of Dreams," where, "If you build it, they will come." You have to establish your site as worthy to be included in the game.

Search Engines zealously guard their system for sorting information. Search Engine Optimizers have to study the Search Engines to determine what is relevant to the algorithm to learn how to best present a website to be determined worthy of high search results. It is a full time operation because Search Engines are constantly reworking their algorithm elements to keep their information accurate. If you don't have that kind of time you need a SEO Pro to do it for you.