Ten Things You Should Know About Document Indexing

It's document indexing that makes the tremendous speeds of document retrievals possible. As you may have noticed, Internet search engines retrieve documents relevant to your specific query from among billions of documents on the Web in less than a second. This would have been simply impossible if they had to search through all the billions in response to each query.

1. Search engines use what is called an inverted list index that lists the documents against each word, instead of words in each document. In response to a query, the engines look up the query words in their index and then list the documents against those words.

2. Typically there will be hundreds of documents, if not thousands, against each word. It then becomes necessary to rank the documents in order of relevance to the query. Relevance is determined by using certain rules set by the engine, and typically involves more than the density of the particular query words in each document.

3. The major search engines do what is known as full-text indexing, i.e. they check all the words in the document’s content, and list it against each of these words (except perhaps too common words like ‘the’).

4. Not all indexing is full-text indexing. Full text indexes tend to be huge, requiring much storage space on their own. Indexing by document meta tags take up much less space. The meta tags provide information about the document that helps retrieve it. For example, a brief note about the content of the document, its date of creation/modification and the author name might be attached as meta tags with each document.

5. Meta tag indexing requires that the user has an idea of what the tags contain so that the person can query using these values. This is typically achieved by having standard practices for describing document contents and document naming. Often, drop-down selection boxes of such descriptions and names are used for manually tagging the document so that different users will use the same terms for similar documents.

6. Indexing is mainly used with unstructured documents, such as correspondence, reports, articles and so on. Structured documents such as transaction records are typically stored in databases, and have unique IDs for each document. Database queries can then bring up the right document in little time (instead of the many documents brought up by search queries).

7. Computer systems typically add certain meta information automatically to each document they create or modify. The date of creation and document author name are examples of such automatically added data. Other data such as document content description can be manually added by the user, or added using such devices as standard-description barcode cards.

8. Indexing can be specialized as when scientific documents are indexed using scientific notation rather than standard words. The key issue is ease of subsequent retrieval. Searchers for scientific documents, for example, will typically find it easier to retrieve documents using the specialized notations.

9. When paper documents are scanned into digital images, they cannot be indexed as such. Instead, the images need to be processed further using such tools as OCR (Optical Character Recognition) software to convert the images of text characters into standard, machine readable ASCII or Unicode characters.

10. Document indexing is not the only way to facilitate their subsequent retrieval. A hierarchical directory structure with meaningfully named folders and subfolders, and proper classification of documents and their storage in relevant subfolders, can enable quick browsing to the correct folder and retrieval. Where necessary, this can be combined with folder-level indexing and search.

Without the facility of indexing the thousands of documents using, say a desktop search facility, businesses might find that retrieving unstructured documents is a tough, and often simply impossible, task. Indexing, full text or meta tag based, changes the situation dramatically making it possible to retrieve even a particular e-mail comparatively quickly. Indexing is thus a powerful business tool.