-
-
27 Feb 2018 08:40:23 UTC
- Distribution: Lucy
- Source (raw)
- Browse (raw)
- Changes
- Homepage
- How to Contribute
- Clone repository
- Issues
- Testers (1251 / 3 / 7)
- Kwalitee
Bus factor: 1- License: apache_2_0
- Perl: v5.8.3
- Activity
24 month- Tools
- Download (1.12MB)
- MetaCPAN Explorer
- Permissions
- Subscribe to distribution
- Permalinks
- This version
- Latest version
and 1 contributors- The Apache Lucy Project <dev at lucy dot apache dot org>
- Dependencies
- Clownfish
- and possibly others
- Reverse dependencies
- CPAN Testers List
- Dependency graph
NAME
Lucy::Docs::IRTheory - Crash course in information retrieval
DESCRIPTION
Just enough Information Retrieval theory to find your way around Apache Lucy.
Terminology
Lucy uses some terminology from the field of information retrieval which may be unfamiliar to many users. “Document” and “term” mean pretty much what you’d expect them to, but others such as “posting” and “inverted index” need a formal introduction:
document - An atomic unit of retrieval.
term - An attribute which describes a document.
posting - One term indexing one document.
term list - The complete list of terms which describe a document.
posting list - The complete list of documents which a term indexes.
inverted index - A data structure which maps from terms to documents.
Since Lucy is a practical implementation of IR theory, it loads these abstract, distilled definitions down with useful traits. For instance, a “posting” in its most rarefied form is simply a term-document pairing; in Lucy, the class MatchPosting fills this role. However, by associating additional information with a posting like the number of times the term occurs in the document, we can turn it into a ScorePosting, making it possible to rank documents by relevance rather than just list documents which happen to match in no particular order.
TF/IDF ranking algorithm
Lucy uses a variant of the well-established “Term Frequency / Inverse Document Frequency” weighting scheme. A thorough treatment of TF/IDF is too ambitious for our present purposes, but in a nutshell, it means that…
in a search for
skate park
, documents which score well for the comparatively rare termskate
will rank higher than documents which score well for the more common termpark
.a 10-word text which has one occurrence each of both
skate
andpark
will rank higher than a 1000-word text which also contains one occurrence of each.
A web search for “tf idf” will turn up many excellent explanations of the algorithm.
Module Install Instructions
To install Lucy, copy and paste the appropriate command in to your terminal.
cpanm Lucy
perl -MCPAN -e shell install Lucy
For more information on module installation, please visit the detailed CPAN module installation guide.