Text::TokenStream::Token - class to model scanned tokens
my $token = Text::TokenStream::Token->new(
type => 'identifier',
text => 'hello',
position => $position,
This class represents tokens that Text::TokenStream finds in its input.
This class uses Moo, and inherits the standard new constructor.
A hashref of parts of the input text; defaults to an empty hashref; read-only. Can be used to model the structure of individual tokens with greater precision.
A boolean; default false; read-only. Indicates whether this token occurred with no preceding whitespace.
A non-negative integer; required; read-only. The 0-based offset from the start of the input where the token was found.
Any string; required; read-only. The literal text matched by the relevant rule.
A non-empty string containing only qr/\w/a characters, that does not begin with a digit; required; read-only. Used to distinguish types of token.
Takes one argument, either a string or a coderef. If it's a string, returns the result of comparing the token's "text_for_matching" against it with eq. If it's a coderef, runs the coderef in scalar context, with the token instance as its first argument, and additionally with $_ set to the token instance, and returns a boolean indicating whether the coderef returned a truthy value.
Takes no arguments. Returns a string representation of the token, suitable for debugging.
Takes no arguments. Returns the same as "text" (but can be overridden in subclasses, for example to return a case-folded version of the text for some or all tokens).
Aaron Crane, <email@example.com>
Copyright 2021 Aaron Crane.
This library is free software and may be distributed under the same terms as perl itself. See http://dev.perl.org/licenses/.
To install Text::TokenStream, copy and paste the appropriate command in to your terminal.
perl -MCPAN -e shell
For more information on module installation, please visit the detailed CPAN module installation guide.