Search Engine Submission: Is It Still Necessary?

When the Internet was young, not so long ago, there were lots of search engines and the conventional wisdom was that one of the first things that you needed to do was to submit your site to all of those search engines. But what about today when a handful of search engines dominate the scene; is search engine submission still important or even necessary at all?

A short answer to both of these questions is a qualified "no." The major search engines such as Google, MSN, Yahoo and Ask.com (Ask Jeeves) all employ robotic programs "bots" which scour the Internet looking for data (sites) to add to the search engine indices. Whenever they visit a web page they look at the links on that page and follow them. If they follow a link to a new site, that site is then included in their data base. In addition, the bots periodically revisit all the sites that they know about.

If you are curious or want to know how often the bots are visiting your web pages, then look in your statistics information (provided by many hosting firms) or look in your site's traffic logs. Yahoo's bot is called Inktomi Slurp, Google's is Googlebot , MSN uses the MSNbot and Ask.com's bot is called Askjeeves .

Suppose you have just constructed a new site. How do you make sure that it gets picked up by these "bots"? The best thing to do is to find a site that is already included in the search engines and get a link from that site to yours. Your web designer or even hosting company may wish to link to you. Perhaps one of your friends or colleagues has a website in operation, ask them for a link. Perhaps you have an existing site; then by all means link to your new site.

During the infancy of the Internet search engine submission companies used to warn clients that they had better sign up for a monthly submission program because search engines sometimes "lost" sites or would not find your new material. In the same way that a single submission to search engines is hardly necessary, monthly submissions or submissions of individual urls on a particular website are no longer necessary.

Due to the revisiting patterns of the robots you don't have to worry about resubmitting your site. If you have some new pages that you want to be included in the search engine, then just make sure that you have easy to find text links to those new pages. A site map page with text links to all of your pages is the best way to ensure that the bots will follow and index all of your pages.

It is rare that a site gets dropped from a search engine, though it can happen if your site violates the terms of service of the search engine, or is down for a prolonged time due to server problems. If you are not sure whether your site is in a particular search engine you can find out by making a search: site: www.yoursite.com . If nothing turns up then your site is not yet in that search engine, or if it had been in that search engine, it is no longer there.

Similarly, if you are not sure how many of your pages have been included in each search engine, make the search: site:www.yoursite.com and the number of results will give you an idea of how many pages are included in that particular search engine.

Despite all that I have said, I have come across sites that had been in existence for a while and still were not "found" by the major search engines. In such cases the best thing to do is to make a few manual submissions at these locations:

http://www.google.com/addurl
http://search.yahoo.com/info/submit.html (and click on the Submit your site for free)
http://search.msn.com/docs/submit.aspx

In the rare case where you suspect that your site may have been dropped from a major search engine such as Google, due to a violation of the search engine's terms of service the best thing to do is to fix what is wrong and make a re-inclusion request. Google has a special url for this: http://www.google.com/support/webmasters/bin/answer.py?answer=35843

So, in summary, don't worry about search engine submission to the main search sites, the busy bots are doing most of the work.