Trait texlang::vm::TokenStream

source ·

pub trait TokenStream {
    type S;

    // Required methods
    fn next(&mut self) -> Result<Option<Token>, Box<Error>>;
    fn peek(&mut self) -> Result<Option<&Token>, Box<Error>>;
    fn vm(&self) -> &VM<Self::S>;

    // Provided methods
    fn consume(&mut self) -> Result<(), Box<Error>> { ... }
    fn commands_map(&self) -> &Map<Self::S> { ... }
    fn state(&self) -> &Self::S { ... }
    fn trace(&self, token: Token) -> SourceCodeTrace { ... }
    fn trace_end_of_input(&self) -> SourceCodeTrace { ... }
}

Expand description

A stream of tokens generated on demand.

This trait describes a general stream of tokens where the front of the stream may retrieved using TokenStream::next or peeked at using TokenStream::peek. In practice, all TokenStreams in Texlang are either ExecutionInput, ExpansionInput or UnexpandedStream. This trait exists to allow a generic function to accept any of these types.

Note on lazy loading

The simplest example of a stream is a vector of tokens. However, streams are more general than this and can encompass situations in which the full contents cannot be determined in advance. This can be thought of as “lazy loading” for the tokens. The classic example of this kind of stream comes from the following LaTeX snippet:

\makeatletter \do@

Assuming the default TeX catcode map, if we were to parse this input all at once we would get three tokens: the control sequence makeatletter, the control sequence do, and a single character token with value @ and catcode “other”. This is not the correct result, though: the first control sequence changes the tokenization rules such that @ is now an admissible character in the name of a control sequence. The correct input is thus the control sequence makeatletter followed by the control sequence do@.

Required Associated Types§

source

type S

The type of the custom state in the VM.

Required Methods§

source

fn next(&mut self) -> Result<Option<Token>, Box<Error>>

Gets the next token in the stream.

This method is almost the same as the next method in Rust’s iterator trait, except a stream can return an error.

As with iterators, a result of Ok(None) indicates that the stream is exhausted.

source

fn peek(&mut self) -> Result<Option<&Token>, Box<Error>>

Peeks at the next token in the stream without removing it.

In many situations it is necessary to examine the next token without consuming it. For example when reading an integer from a stream, one needs to peek at the next token to see if it is a digit and thus extends the currently parsed integer. Consuming the token with TokenStream::next is not correct in this situation if the token is not a digit.

For consumers, it is important to note that the peek method requires a mutable reference to the stream. This is because some mutable processing may be needed in order to determine what the next token is. For example:

When reading tokens from a file, peeking at the next token may involve reading more bytes from the file and thus mutating the file pointer. sThis mutations is easy to undo in general.
When performing expansion on a stream, the next token in the stream may need to be expanded rather than returned. The next token will be the first token in the expansion in this case, or the following token in the remaining stream if the expansion returns no tokens. This mutation is generally irreversible.

source

fn vm(&self) -> &VM<Self::S>

Returns a reference to the VM.

Provided Methods§

source

fn consume(&mut self) -> Result<(), Box<Error>>

Consumes the next token in the stream without returning it.

This method is mostly to make code self-documenting. It is typically used in situations where a peek has already occurred, and the token itself is not needed.

source