27 Feb 2018 08:40:23 UTC
- Distribution: Lucy
- Source (raw)
- Browse (raw)
- How to Contribute
- Clone repository
- Testers (1249 / 3 / 7)
- KwaliteeBus factor: 1
- License: apache_2_0
- Perl: v5.8.3
- Activity24 month
- Download (1.12MB)
- MetaCPAN Explorer
- Subscribe to distribution
- This version
- Latest version++ed by:6 non-PAUSE usersNWELLNHOF Nick Wellnhoferand 1 contributors
- The Apache Lucy Project <dev at lucy dot apache dot org>
Lucy::Analysis::SnowballStopFilter - Suppress a “stoplist” of common words.
my $stopfilter = Lucy::Analysis::SnowballStopFilter->new( language => 'fr', ); my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new( analyzers => [ $tokenizer, $normalizer, $stopfilter, $stemmer ], );
A “stoplist” is collection of “stopwords”: words which are common enough to be of little value when determining search results. For example, so many documents in English contain “the”, “if”, and “maybe” that it may improve both performance and relevance to block them.
Before filtering stopwords:
("i", "am", "the", "walrus")
After filtering stopwords:
SnowballStopFilter provides default stoplists for several languages, courtesy of the Snowball project, or you may supply your own.
|-----------------------| | ISO CODE | LANGUAGE | |-----------------------| | da | Danish | | de | German | | en | English | | es | Spanish | | fi | Finnish | | fr | French | | hu | Hungarian | | it | Italian | | nl | Dutch | | no | Norwegian | | pt | Portuguese | | sv | Swedish | | ru | Russian | |-----------------------|
my $stopfilter = Lucy::Analysis::SnowballStopFilter->new( language => 'de', ); # or... my $stopfilter = Lucy::Analysis::SnowballStopFilter->new( stoplist => \%stoplist, );
Create a new SnowballStopFilter.
stoplist - A hash with stopwords as the keys.
language - The ISO code for a supported language.
my $inversion = $snowball_stop_filter->transform($inversion);
Take a single Inversion as input and returns an Inversion, either the same one (presumably transformed in some way), or a new one.
inversion - An inversion.
Lucy::Analysis::SnowballStopFilter isa Lucy::Analysis::Analyzer isa Clownfish::Obj.