27 Oct 2007 19:46:10 UTC
- Development release
- Distribution: KinoSearch
- Source (raw)
- Browse (raw)
- How to Contribute
- Issues (5)
- Testers (21 / 5 / 2)
- KwaliteeBus factor: 0
- License: perl_5
- Activity24 month
- Download (460.82KB)
- MetaCPAN Explorer
- Subscribe to distribution
- This version
- Latest versionCREAMYG Marvin Humphreyand 1 contributors
- Marvin Humphrey <marvin at rectangular dot com>
KinoSearch::Docs::IRTheory - Crash course in information retrieval.
Just enough Information Retrieval theory to find your way around KinoSearch.
KinoSearch uses some terminology from the field of information retrieval which may be unfamiliar to many users. "Document" and "term" mean pretty much what you'd expect them to, but others such as "posting" and "inverted index" need a formal introduction:
document - An atomic unit of retrieval.
term - An attribute which describes a document.
posting - One term indexing one document.
term list - The complete list of terms which describe a document.
posting list - The complete list of documents which a term indexes.
inverted index - A data structure which maps from terms to documents.
Since KinoSearch is a practical implementation of IR theory, it loads these abstract, distilled definitions down with useful traits. For instance, a "term" need not be associated with a field name; it doesn't even need to be made out of text. However, it is convenient to define it that way when trying to build a text search engine library.
Similarly, a "posting" in its most rarefied form is simply a term-document pairing; in KinoSearch, the class KinoSearch::Posting::MatchPosting fills this role. However, by associating additional information with a posting like the number of times the term occurs in the document, we can turn it into a ScorePosting, making it possible to rank documents by relevance rather than just list documents which happen to match in no particular order.
KinoSearch uses a variant of the well-established "Term Frequency / Inverse Document Frequency" weighting scheme. A thorough treatment of TF/IDF is too ambitious for our present purposes, but in a nutshell, it means that...
in a search for
skate park, documents which score well for the comparatively rare term
skatewill rank higher than documents which score well for the common term
a 10-word text which has one occurrence each of both
parkwill rank higher than a 1000-word text which also contains one occurrence of each.
A web search for "tf idf" will turn up many excellent explanations of the algorithm.
Copyright 2007 Marvin Humphrey
See KinoSearch version 0.20.