Trait texlang::vm::TokenStream

source ·

pub trait TokenStream {
    type S: TexlangState;

    // Required methods
    fn next_or(&mut self) -> Result<Option<Token>>;
    fn back(&mut self, token: Token);
    fn vm(&self) -> &VM<Self::S>;

    // Provided methods
    fn next<E: EndOfInputError>(&mut self, err: E) -> Result<Token> { ... }
    fn peek(&mut self) -> Result<Option<Token>> { ... }
    fn commands_map(&self) -> &Map<Self::S> { ... }
    fn state(&self) -> &Self::S { ... }
}

Expand description

A stream of tokens generated on demand.

This trait describes a general stream of tokens where the front of the stream may retrieved using TokenStream::next or peeked at using TokenStream::peek. In practice, all TokenStreams in Texlang are either ExecutionInput, ExpansionInput or UnexpandedStream. This trait exists to allow a generic function to accept any of these types.

§Note on lazy loading

The simplest example of a stream is a vector of tokens. However, streams are more general than this and can encompass situations in which the full contents cannot be determined in advance. This can be thought of as “lazy loading” for the tokens. The classic example of this kind of stream comes from the following LaTeX snippet:

\makeatletter \do@

Assuming the default TeX catcode map, if we were to parse this input all at once we would get three tokens: the control sequence makeatletter, the control sequence do, and a single character token with value @ and catcode “other”. This is not the correct result, though: the first control sequence changes the tokenization rules such that @ is now an admissible character in the name of a control sequence. The correct input is thus the control sequence makeatletter followed by the control sequence do@.