31 Dec 2014 03:15:36 UTC
- Distribution: Lucy
- Source (raw)
- Browse (raw)
- How to Contribute
- Clone repository
- Testers (1056 / 65 / 0)
- KwaliteeBus factor: 1
- License: apache_2_0
- Perl: v5.8.3
- Activity24 month
- Download (1.06MB)
- MetaCPAN Explorer
- Subscribe to distribution
- This version
- Latest version++ed by:6 non-PAUSE usersCREAMYG Marvin Humphreyand 1 contributors
- The Apache Lucy Project <dev at lucy dot apache dot org>
Lucy::Analysis::Normalizer - Unicode normalization, case folding and accent stripping.
my $normalizer = Lucy::Analysis::Normalizer->new; my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new( analyzers => [ $tokenizer, $normalizer, $stemmer ], );
Normalizer is an Analyzer which normalizes tokens to one of the Unicode normalization forms. Optionally, it performs Unicode case folding and converts accented characters to their base character.
If you use highlighting, Normalizer should be run after tokenization because it might add or remove characters.
my $normalizer = Lucy::Analysis::Normalizer->new( normalization_form => 'NFKC', case_fold => 1, strip_accents => 0, );
normalization_form - Unicode normalization form, can be one of 'NFC', 'NFKC', 'NFD', 'NFKD'. Defaults to 'NFKC'.
case_fold - Perform case folding, default is true.
strip_accents - Strip accents, default is false.
Lucy::Analysis::Normalizer isa Lucy::Analysis::Analyzer isa Clownfish::Obj.