You Can't Trick Google

Google is tough to trick. What if you wanted to trick it? Well, even if you could trick it, this would probably be unwise, because your site could end up in the "Trash Pile" list. This largely frees webmasters from spending time on tricks and page hacks designed to get higher rankings.

There are several reasons your pages may not show up, including the way they are generated, their availability, and content.

According to information at Google's site, they limit the number of dynamically generated pages that they index at any site. Site crawlers can generate a tremendous amount of site traffic in a short amount of time. Google limits the number of dynamic pages it requests to avoid monopolizing a server's resources.

Frames can also cause problems. This is because it's difficult for an automated system to know what should be indexed: the frame, the individual panes, the entire page or the alternate content. If your site uses frames, it's very important that your frames include "noframes" content. This is alternate content for browsers that can't handle frames. It is also used by search engines.

If your site doesn't provide an internal path to all of your pages, search robots may not find them. These pages may also not be indexed if they are only linked to by pages that are not in the index for other reasons, such as being dynamically generated. This problem can be avoided by including an index page that provides direct links to pages that have convoluted navigation paths.

Pages may also be left out if they are not available when search crawlers visit. Many sites do periodic maintenance at off hours, or cycle servers during the night. If your site is not fully redundant, it may be unavailable at these times, which can result in your site being left out of the index. This will normally be corrected the next time the crawler visits your site, but should be avoided because it makes it difficult for users to find your site.

In rare cases, page content can cause your pages to be left out. If Google identifies elements on your pages that it thinks are inappropriate for general audiences, the pages will get filtered out. The content will be indexed, but may not show up in search results, depending on the searcher's SafeSearch settings.

Pages are also filtered out if they do not meet Google's "quality standards". Google doesn't detail what its quality standards are, but things like page cloaking, invisible text, or other techniques designed to fool search engines can cause pages to be excluded. Pages are also occasionally excluded for copyright reasons, or because the page content is inflammatory. Fortunately, it's rare for any of these problems to affect typical sites.