Karen Scharf is an Indianapolis marketing consultant who works with small business owners and entrepreneurs. She offers several whitepapers, free reports and checklists, including her FREE Can-Spam checklist and FREE email pre-flight checklist to ensure your emails get delivered, get opened and get read. Download your copies at http://www.ModernImage.com.
In order to analyze the content on your web site and figure out which pages to display (and in which order) for certain search terms, the search engines (SEs) use a program called a web crawler. You might have also heard these referred to as spiders or web-bots.
Every SE has its own web crawler program and its own formula for ranking pages. However, while page factors might be ranked differently by different engines, the idea behind the web crawlers is basically the same.
Getting your site crawled often is a good indicator that the SE finds your site important. And since important sites get better rankings, getting your site crawled frequently is definitely a good thing.
While there is no way you can make the SE crawl your site, there are a few things you can to do encourage the spiders to visit your site more often:
1. First and foremost, keep your content up to date. Add unique content often and on a regular basis. And be sure to ping Google each time you update your site.
2. Get as many backlinks as possible from other relevant sites that are crawled regularly. There are several tools available that will track and measure the crawl rate of other websites.
3. Use a unique title tag for each page. Also, while not as important as the title, each page should also have its own keywords and description tag. This will help the spiders "know" that each page is a unique and individual entity.
4. Keep your pages as small as possible. The spider will not spend an unlimited amount of time at your web site. If it's held up by your huge images, monstrous PDFs or even excessive text, it will simply abandon the page and move on.
5. Pay close attention to your internal links and avoid any duplicate content issues. If you have multiple pages with the same content (often used for testing and/or tracking purposes) be sure to use your robots text file to keep the SEs away.
6. Create a sitemap for the content you want crawled and make sure your content is compliant.
7. Monitor your web server. Obviously, your server needs to be up and running when the spiders come a-crawling. There are many commercial providers who will monitor your server and notify you when there is a problem.
In addition, ensure that your server handles all of your error pages correctly. Refer to the Google Webmaster Tools report of unreached pages for an idea of what errors are being returned.
8. And while we're discussing Google Webmaster Tools, don't forget to use it to monitor your crawl rate, adjust it if necessary, test and track and see what works best for your site.
Time to implement: As with all SE optimization, working toward an optimal crawl rate is not a one-time job. You will need to add content approximately three times a week and update your site map as needed. Monitoring your web server is an ongoing task.
- Related Articles
- Related Q&A
- 4PSEO Search Engine Optimization (SEO) Company from India:
- Search Marketing: Turning your Website Outside-in to Increase Sales
- SEO Turning Visitors Into Sales
- Boost your Sales with SEO
- Boost your Sales with SEO
- Boost your Sales with SEO
- Want to Hire a Professional SEO Company?
- SEO Traffic Spider | Strategies That Speak Results in SEO




Search Engines to rival humans?
By: byerwills | 06/07/2009If you are looking for a professional agency that can boost sales and enquiries to your site contact us and we will strategically place you above the best in your industry.
Why SEO Copywriting is so Important
By: abhishek | 06/07/2009It is essential to recognize that all content is a form of communication, and the quality of it depends not only on what is communicated and how, but also to whom it is communicated to. When it come to textual content, good quality SEO copywriting is highly essential. People have been condition to read since they began attending school and webmasters who do not understand this usually fail. A webmaster that creates written content with the help of good SEO copywriting is directly engaged in a on
Simple Steps to Your Success in Affiliate Marketing
By: abhishek | 06/07/2009The affiliate marketing business with the internet is much easier nowadays than it was before, when people use the simple technology of the communication such as telephones, newspapers, radio or TV advertisement. These days people can work from home having £Millions business very easy and using just their personal computer with internet. Of course you have to have some computing skills to surf in the internet but we are not about it.
Some essentials to obtain higher ranks in search engines
By: Editor123 | 06/07/2009SEO or search engine optimization refers to the collective techniques used to optimize a website for a search engine. The internet has facilitated a platform for small and local business to reach out to the entire globe and thus, expand their business.
Your Business Can’t Survive Without Search Engine Optimization
By: johnmak | 06/07/2009Search engine optimization is the technique which assists to entice visitors into a website. To get the websites’ desired traffic from search engines, they are designed according to search engine preferences. This assists you in better ranking of the websites as the search engine spiders can easily crawl through them. Search engines are the most powerful medium to get the website the desired ranking and the traffic. Hence, no website can avoid the importance of search engine optimization service
Effective SEO Services to Make Your Online Business Successful
By: johnmak | 06/07/2009Once you have decided to have a website for your business, you will need to make sure that people will easily find it. Usually, people use search engines to find what they are looking for online, so you will need to rank highly in these. This is not as easy as it seems, but there are some effective SEO services that you can adopt to make better ranking on the search engines.
Effective SEO Services to Make Your Online Business Successful
By: johnmak | 06/07/2009Once you have decided to have a website for your business, you will need to make sure that people will easily find it. Usually, people use search engines to find what they are looking for online, so you will need to rank highly in these. This is not as easy as it seems, but there are some effective SEO services that you can adopt to make better ranking on the search engines.
Write Effective PPC Ads - Keep These 6 Tips In Mind
By: Karen Scharf | 14/10/2008 | ECommerceGoogle's recent changes to AdWords make it easier for novices to get started with pay per click advertising. But if you've never created a PPC ad before, the task can seem a little intimidating.
8 Steps To Getting Your Web Site Crawled
By: Karen Scharf | 09/10/2008 | SEOEvery SE has its own web crawler program and its own formula for ranking pages. However, while page factors might be ranked differently by different engines, the idea behind the web crawlers is basically the same. Getting your site crawled often is a good indicator that the SE finds your site important.
Proven Formula For Setting Your 2009 Income Goals
By: Karen Scharf | 06/10/2008 | ECommerceI just finished up my business plan for 2009 (and if you haven't updated your 2009 plan yet, I urge you to get started soon!) and I'm working with several clients to update their plans also. One of the roadblocks many of clients run into is the difficulty of setting concrete goals and objectives.
8 Copywriting Tips The Pros Use
By: Karen Scharf | 09/09/2008 | Direct MailMy most popular service lately has been copywriting, both for online web pages and offline sales letters. Whether you're a service provider or a retailer, a solidly constructed sales letter (or better yet, a three-letter series) can work for you...
How To Market In A Bad Economy
By: Karen Scharf | 05/09/2008 | MarketingDoes this sound like you...the economy is going south, and you're starting to panic. You're worried about the financial health of your small business. You think about slashing your prices to stimulate your sales. You assume you need to cut back on your...
Automatic Income For A Down Economy
By: Karen Scharf | 29/08/2008 | MarketingI don't know about you, but I get a little annoyed by the big-name pundits who declare "I refuse to participate in the bad economy." Sure, it's a great philosophy, but for the small business owner who sees his income shriveling away, who has to lay off long-time staff members...
3 Simple Steps for Gathering Effective Testimonials
By: Karen Scharf | 26/08/2008 | BrandingI've recently suffered a computer crash, and one of the folders that wasn't backed up was one that contained all my testimonials. So I'm now on a mad-scramble to try to acquire new testimonials to use...
Your 5 Day Marketing Plan
By: Karen Scharf | 21/08/2008 | MarketingUnfortunately, since many small business owners don't consider marketing as a core business activity...it's often the task that gets set aside when time is crunched. But by taking just a few minutes to plan ahead, your future marketing activities can go much smoother; you'll be able to complete your marketing activities quickly and easily.