Remember Me
forgot your password?

8 Steps To Getting Your Web Site Crawled

In order to analyze the content on your web site and figure out which pages to display (and in which order) for certain search terms, the search engines (SEs) use a program called a web crawler. You might have also heard these referred to as spiders or web-bots.

Every SE has its own web crawler program and its own formula for ranking pages. However, while page factors might be ranked differently by different engines, the idea behind the web crawlers is basically the same.

Getting your site crawled often is a good indicator that the SE finds your site important. And since important sites get better rankings, getting your site crawled frequently is definitely a good thing.

While there is no way you can make the SE crawl your site, there are a few things you can to do encourage the spiders to visit your site more often:

1. First and foremost, keep your content up to date. Add unique content often and on a regular basis. And be sure to ping Google each time you update your site.

2. Get as many backlinks as possible from other relevant sites that are crawled regularly. There are several tools available that will track and measure the crawl rate of other websites.

3. Use a unique title tag for each page. Also, while not as important as the title, each page should also have its own keywords and description tag. This will help the spiders "know" that each page is a unique and individual entity.

4. Keep your pages as small as possible. The spider will not spend an unlimited amount of time at your web site. If it's held up by your huge images, monstrous PDFs or even excessive text, it will simply abandon the page and move on.

5. Pay close attention to your internal links and avoid any duplicate content issues. If you have multiple pages with the same content (often used for testing and/or tracking purposes) be sure to use your robots text file to keep the SEs away.

6. Create a sitemap for the content you want crawled and make sure your content is compliant.

7. Monitor your web server. Obviously, your server needs to be up and running when the spiders come a-crawling. There are many commercial providers who will monitor your server and notify you when there is a problem.

In addition, ensure that your server handles all of your error pages correctly. Refer to the Google Webmaster Tools report of unreached pages for an idea of what errors are being returned.

8. And while we're discussing Google Webmaster Tools, don't forget to use it to monitor your crawl rate, adjust it if necessary, test and track and see what works best for your site.

Time to implement: As with all SE optimization, working toward an optimal crawl rate is not a one-time job. You will need to add content approximately three times a week and update your site map as needed. Monitoring your web server is an ongoing task.

Karen Scharf

Karen Scharf is an Indianapolis marketing consultant who works with small business owners and entrepreneurs. She offers several whitepapers, free reports and checklists, including her FREE Can-Spam checklist and FREE email pre-flight checklist to ensure your emails get delivered, get opened and get read. Download your copies at http://www.ModernImage.com.

Rate this Article: 4 / 5 stars - 2 vote(s)
Print Email Re-Publish

Add new Comment



Captcha
0
1. Ruma (12:53, 16.10.2008)
Quality sprinkling thoughts. Good

  • Latest SEO Articles
  • More from Karen Scharf

SEO vs PPC

By: creative | 28/12/2009
You Need Them Both We hope you’ve heard about “Paid Search” or PPC (pay per click) if your business relies on Web traffic. It’s one of the key ways to draw traffic to your site, and it works wonders when applied with SEO (search engine optimization). In other words, if you’re still using SEO as your only marketing avenue, you need to balance your marketing approach because your competitors may already have.

Freelance content writers Vs Professional content writing firm

By: Kamesh Goud | 28/12/2009
One should be careful while hiring a freelance content writer or a professional content writing firm as the content written by a freelancer may be duplicate of a copyrighted content. A professional firm always makes sure that content developed is unique and does not violate any copyright law.

Make the holidays more fun with a hot tub for the whole family!

By: johnmak | 28/12/2009
This holiday season; why not give your family a cozy and relaxing present under the tree. Hot tubs are a great addition to any home and are the perfect gift for the entire family. Hot tubs can offer your family a wide range of health benefits, muscle relaxation, stress relief and social activity.

Link Building Service- Its Vital Aspect

By: Bhargav Bavarva | 28/12/2009
For anyone who has considered going through the process of search engine optimization, it is more likely than not, that you have heard the word “backlink”. However, if you are new to the idea of SEO, the question which probably popped into your mind is- what is a backlink and why is a link building service needed?

SEO Techniques Every Website Owner Needs to Know

By: Alice Ault | 28/12/2009
It is certainly no secret that in order for an internet marketing venture to succeed much less thrive it will need a great deal of traffic flowing to its website.

The Top Three Reasons Why Visitors Leave Your Site without Buying a Product or Filling out a Form

By: Aaron Garcia | 28/12/2009
It is the frustration of every website owner. You look at your analytics at the end of the week and are delighted to see that your traffic has increased, but you have not seen the same increase in sales or leads.

Pay Per Click versus Search Engine Optimization: Behind All the Hype

By: Peter Dulay | 28/12/2009
Each online personality in the vast online world seeks the achievement of the similar dream: A boundless stream of preferred visitors on their websites,admiring their works, reading their contents and putting their advertisements and offerings into account.

Think smart, only pay if people actually click on your links!!

By: johnmak | 28/12/2009
Pay per Click or just PPC, is a part of Search engine optimization. As the name suggests, advertisers pay their host if and only if their ad is clicked. It is a very effective internet advertising model used on many websites.

Why Won't My Site Show Up in Google?

By: Karen Scharf | 07/10/2009 | SEO
We often get inquiries from prospective clients who would like us to build them a new website. They believe that their current website must be “bad” because it's not listed in the search engines. They're convinced that a new website will automatically ensure a top ten Google rank.

There Is No Such Thing as a Duplicate Content Penalty

By: Karen Scharf | 07/10/2009 | SEO
First, I want you to understand that I am NOT giving you license to steal other people’s content. There ARE repercussions to using duplicate content, but it is not a penalty per se.

Email Marketing Tests to Get You Started

By: Karen Scharf | 07/10/2009 | Internet Marketing
If you’re utilizing email marketing, you should be utilizing email testing. But it can seem complex and costly, and many small business owners aren’t quite sure where to start. If you’d like to use email testing, but aren’t sure where to start, here are 4 of the easiest tests to get you going:

Save Money With Geo-Targeting

By: Karen Scharf | 22/09/2009 | Internet Marketing
If you're a local business on a tight budget, you can save money on your advertising budget by targeting only specific locations with your AdWords campaigns.

Landing Page Optimization Tips

By: Karen Scharf | 22/09/2009 | Internet Marketing
When we hear the words Landing Page we generally think of PPC campaigns or some other form of internet advertising. But I like to use landing pages for all of my off-line advertising also. However, you’re using it, a landing page is a web page, typically hidden from the site’s menu bar and other navigation – meaning the only way you can get to the page is to type the URL directly into the browser.

Boring Subject Lines Outpull Hype

By: Karen Scharf | 18/09/2009 | Small Business
In email marketing, the subject line is often seen as the single most important element. Afterall, if your subject line isn't persuasive, no one will open your email and your message won't get read. The subject line is generally equated with the headline of a display ad.

Measuring Your Social Media ROI

By: Karen Scharf | 16/09/2009 | Small Business
If you’ve added social media to your marketing mix, you’re probably asking yourself, “Is it worth all the time and effort involved?” That might seem like a difficult question to answer at first, but there are definite steps you can take to measure the return on your social media investments.

Submit Your Articles Free: Signup
Article Categories




Use of this web site constitutes acceptance of the Terms Of Use and Privacy Policy | User published content is licensed under a Creative Commons License.
Copyright © 2005-2008 Free Articles by ArticlesBase.com, All rights reserved. (3.24, 6, w3)