Search engine spiders crawl pages following hyperlinks on other pages. But certain things may serve as hindrance to crawling of search engine spiders.It is important to know and understand them so that one can modify he website accordingly for giving better access to spiders.
A link that is too complex to follow or is situated in quite depth from the hyperlink is called a bump.
This serves as speed breaker for the spider in a similar fashion to traffic on the road.
Website may contain sections of data that cannot be accessed at all by the spiders is called wall.
Both of these would block indexing of certain sections of your website.
There may be some parts of your site that you do not want the search engines to know.
But if you are in business, you would want to expose maximum of your content and indexed.
These are few bumps and walls that you must avoid to do that.
Bumps
* Dynamic URLs with >2 dynamic parameters would be a bump. These kind of URLs are reluctantly crawled as complex URLs often result in errors. An example of complex URL would be http://www.yoursite.com/page.php?id=3&PF=567&user=premium&name=%Arun%
* Hundreds of outgoing links from a page. In such cases spiders may not follow each link.
* Pages buried in depth of linking structure. Generally, the spiders will not visit a website link that requires more than 3 clicks from home page until that page has high number of incoming links.
* Frames are a bump to crawling.
Walls
* Pages requiring a login. This is generally case untill you put special request for robots to visit.
* Documents blocked by robot.txt file
* Links having nofollow tag.
* Pages that require drop down menu to access them.
* Documents that can only be accessed by a search box.
* Pages that can be accessed only after filling and submitting the form. For example a sign up form.
Fewer bumps and walls your site has, smoother would be its indexing.
A rule of thumb to avoid unwanted bumps and walls is to ensure that the content of your site is easily accessible. Most of the spiders would start from your home page. To get indexed, the page must be accessible from home page.
Got a Question? Ask.
Ask the community a question about this article:
Frequently Asked Questions
Game reload
By: Robyne | 18-08-2008
I brought the "Walls of Jericho " game from Circuit city a few years ago, my computer crashed and I need to reload, but the game is no longer listed on Circuit City's site. It is also not listed on the Boonty game site. How can I reload my game?
Need watch band link No.9520-2
By: Mel | 16-08-2008
need watch band link No.9520-2
Splotchy areas on painted wall
By: Kedoba | 16-08-2008
I painted my bathroom with a semi gloss paint and the finished result is not uniform. there are areas that are lighter and dull where the rest of the area is shiny and smooth. I used Kilz primer prior to painting. I am not pleased. what can I do to fix the splotchy areas or what did I do wrong to start with. HELP
signed: paint job went to pot!
Spider
By: gram | 15-08-2008
i live in northeastern pa. and i found a spider in the back yard. it's about an inch long with a bulbous abdomen tan in color with black markings on it could you tell me what kind it is and if its piosonous
Blogosphere raw data
By: wolkomir | 11-08-2008
Looking for the latest posts on the blogger or wordpress platforms ( could be other platform as well ) , any idea how can i use a relevant service that aggregates the latest posts published in all the environment ?
( for instance anything in the last 24h )
Need the raw data for analysis, so web services like technorati are less relevant here.
Thanks
My baby has a bent penis and always hard
By: traz | 07-08-2008
my baby has a bent penis and always hard
Q&A Powered by:
Latest SEO Articles
Search Engine Marketing - Do Your Business Online by PPC
By: Darwin Sheild | 15/10/2008
Search engine marketing is an online business technique. Search engine marketing will help with the web site's ranking positions in search engines and create more traffic to the web site and converting the visitors in sales.
What You Know Before Going for Search Engine Optimization
By: pntglobal | 15/10/2008
If you are owner of a web site, you should think for Search Engine Optimization. It is time consuming job and need patience but in the long run Search Engine Optimization is the most beneficial thing for your web site. It gives life to your web site on cyber net and keeps your web site working for you.
The Importance of Seo
By: gurujitwo | 15/10/2008
Concentrate on the internal and analysis areas of the optimization for Google. I will review 3 essential areas. Let’s start:
Why do you need to use Google Sitemaps.
If your site has hundreds of pages, then a compressed version of the file can be created. You are required to create an account with Google in order to take advantage of this feature. Two things to consider before uploading the sitemap file are to verify that your robots.txt allows Google to craw the site for this to work in the
Web Analytics Technologies
By: pntglobal | 15/10/2008
There are two main technological approaches to collecting web analytics data. The first method, log file analysis, reads the logfiles in which the web server records all its transactions. The second method, page tagging, uses JavaScript on each page to notify a third-party server when a page is rendered by a web browser.
How Can You Start Your Own Video Blog?
By: gurujithree | 15/10/2008
Video Blogs have an appealing affect to the people who see it and who sympathize with its content. They are a phenomena, and here to stay. Mainstream entertainment is wired with video blogging. Here, in Blog world, you are the director of your own video. Commonly referred to and known as vlogs (short term for video blog), contents usually contain simply scenes of everyday life.
SEO and Website Design Methods to Follow
By: WebmasterM | 15/10/2008
SEO is all about receiving the sites with top privileges on the major search engines with the core intension to strike the definite consumers. You can publicize the website by utilizing different tackles, procedures and search engine optimization approaches.
A Few Things to Notice if You Want Convertible and Stable Web Traffic
By: Jefery Smiles | 15/10/2008
In most of the cases when working with SEO service providers, to be competitive they will offer very attractive service charges and to build up fast satisfaction from the clients every “kind” of internet marketing techniques are going to be used.
Developing Recommendations for Optimization
By: Deepti Bansal | 15/10/2008
By focusing on the search terms your customers actually use, search engine optimization (SEO) helps ensure your website attracts customers, not just browsers.And increasing business while keeping costs low is the aim of every marketing strategy.
More from Arun Pal Singh
How to Make Other People Help You Succeed?
By: Arun Pal Singh | 28/04/2008 | Stress Management
The easiest and best road to success is found in making a difference in the lives of those around us. The goods that we do for others, will come back to us - usually many times over.
How to Cope Up With Learning Curve?
By: Arun Pal Singh | 26/04/2008 | Entrepreneurship
Most of the failures in new endeavors occur at initial stage. They occur when person involved is not even fully equipped with how to do knowledge. In both cases the key element of success is missing-learning.
What are Different Types of Autoresponder?
By: Arun Pal Singh | 25/04/2008 | Online Business
Autoresponder as a computer program that automatically answers e-mail sent to it. Earliest autoresponders were created within mail transfer agents. These were kind of bounce messages when they were not able to deliver the mail.
Arun Pal Singh is an Internet Marketing Consultant and Web Author Who Runs Website Http://www.homeforprofits.com to Help Online Entrepreneurs Wi
By: Arun Pal Singh | 15/04/2008 | Online Business
Newsletter can be an excellent tool for building trust and long term relationship. Building trust and relationship means getting life time customers who would buy from you with pleasure for times to come.
Do not Trick Your Customers With False Headlines
By: Arun Pal Singh | 14/04/2008 | Online Business
To increase the number of emails opened, many people try to trick their subscribers by providing catchy but misleading headlines. Their motive is to get the subscriber open and read the mail. But then there is a difference in motivation and interest. If somebody clicks your mail because he is interested in it, you would have higher response rate with more mails being opened.
Track Your Ads for Better Performance
By: Arun Pal Singh | 11/04/2008 | PPC Advertising
An integral part of any advertising campaign is knowing which ads bring you the most visitors. You might have multiple ads running and want to judge the best performing ads or you might just want to evaluate your running ad for improvement purposes.
How to Create an Ad That Performs?
By: Arun Pal Singh | 09/04/2008 | Online Business
For an advertising campaign to be effective it needs a lot of planning and hard work. Though we do have resources and websites where we can put our advertisement and we have new technologies which we can use to create our advertisement, we must understand that it is content and quality of the advertisement that would bring the results.
How to Use Testimonial Effectively to Boost Your Business
By: Arun Pal Singh | 08/04/2008 | Online Business
Testimonials are positive reviews that people tell about your product. Testimonials carry tremendous weight. Because, other people confirm that you are as good as you say you are. When done rightly, testimonials can tremendously increase your conversion rates.
Internet 
About the Author:
