Author image Marvin Humphrey
and 1 contributors

Add one token to the batch.

Add many tokens to the batch, by supplying the string to be tokenized, and arrays of token starts and token ends (specified in bytes).

Proceed to the next item in the TokenBatch. Returns true if the TokenBatch ends up located at valid token.