• NAME
• SYNOPSIS
• DESCRIPTION
• CONSTRUCTOR
• ATTRIBUTES
  • captures
  • cuddled
  • position
  • text
  • type
• OTHER METHODS
  • matches
  • repr
  • text_for_matching
• AUTHOR
• COPYRIGHT
• LICENCE

NAME

Text::TokenStream::Token - class to model scanned tokens

SYNOPSIS

    my $token = Text::TokenStream::Token->new(
        type => 'identifier',
        text => 'hello',
        position => $position,
    );

DESCRIPTION

This class represents tokens that Text::TokenStream finds in its input.

CONSTRUCTOR

This class uses Moo, and inherits the standard new constructor.

ATTRIBUTES

captures

A hashref of parts of the input text; defaults to an empty hashref; read-only. Can be used to model the structure of individual tokens with greater precision.

cuddled

A boolean; default false; read-only. Indicates whether this token occurred with no preceding whitespace.

position

A non-negative integer; required; read-only. The 0-based offset from the start of the input where the token was found.

text

Any string; required; read-only. The literal text matched by the relevant rule.

type

A non-empty string containing only qr/\w/a characters, that does not begin with a digit; required; read-only. Used to distinguish types of token.

OTHER METHODS

matches

Takes one argument, either a string or a coderef. If it's a string, returns the result of comparing the token's "text_for_matching" against it with eq. If it's a coderef, runs the coderef in scalar context, with the token instance as its first argument, and additionally with $_ set to the token instance, and returns a boolean indicating whether the coderef returned a truthy value.

repr

Takes no arguments. Returns a string representation of the token, suitable for debugging.

text_for_matching

Takes no arguments. Returns the same as "text" (but can be overridden in subclasses, for example to return a case-folded version of the text for some or all tokens).

AUTHOR

Aaron Crane, <arc@cpan.org>

COPYRIGHT

Copyright 2021 Aaron Crane.

LICENCE

This library is free software and may be distributed under the same terms as perl itself. See http://dev.perl.org/licenses/.