Texcraft

Texcraft is a project to create an LLVM-style infrastructure for building TeX software. It is designed as a loosely coupled collection of Rust libraries which compose together to create different kinds of TeX programs.

This work-in-progress website hosts the user-facing documentation.

The Texcraft Playground is example of what can be built with Texcraft.

Design of the documentation

The Texcraft documentation is designed with the Divio taxonomy of documentation in mind. In this taxonomy, there are four kinds of documentation: tutorials, how-to guides, references, and explainers. Because Texcraft is a small project so far, we don't have significant documentation of each type. Right now we have:

• The Texlang user guide, which is mostly a grounds-up tutorial on how to use Texlang. The goal is for it to be possible to read the user guide from the starting introduction through to the end. But we also hope that you can jump into arbitrary sections that interest you, without having to slog through the prior chapters.

• Reference documentation that is autogenerated using rustdoc.

• Occasionally some explainers that are high-level and theoretical. Some parts of the Texlang user guide (such as the introduction) have this style.

There are currently no how-to guides - we think the tutorials are enough for the moment.

What's in a name?

Texcraft is a contraction of TeX and craft. The verb craft was chosen, in part, to pay homage to Robert Nystrom's book Crafting Interpreters. This book is an inspiration both because of its wonderful exposition of the process of developing a language interpreter and because the book itself is so beautifully typeset. (Unfortunately the methods of the book do not directly apply to TeX because TeX is not context free, has dynamic lexing rules, and many other problems besides.) We hope Texcraft will eventually enable people to craft their own TeX distributions.

The name Texcraft is written using the letter casing rule for proper nouns shared by most langauges that use a Roman alphabet: the first letter is in uppercase, and the remaining letters are in lowercase. To quote Robert Bringhurst, "an increasing number of persons and institutions, from archy and mehitabel, to PostScript and TrueType, come to the typographer in search of special treatment [...] Logotypes and logograms push typography in the direction of heiroglyphics, which tend to be looked at rather than read."

Introductory concepts

Running the Texlang VM

Before writing any custom TeX primitives it's good to know how to run the Texlang virtual machine (VM). This way you can manually test the primitives you write to ensure they're working as you expect. In the long run you may decide to lean more on unit testing, even when initially developing commands, rather than manual testing things out. But it's still good to know how to run the VM.

If you just want to see the minimal code to do this, jump down to the first code listing.

Running the VM is generally a four step process:

1. Specify the built-in commands/primitives you want the VM to include.

2. Initialize the VM using its new constructor. At this point you will need to decide which concrete state type you're using. The state concept was described in high-level terms in the previous section, and we will gain hands-on experience with it the primitives with state section. For the moment, to keep things simple, we're just going to use a pre-existing state type that exists in the Texlang standard library crate: ::texlang_stdlib::testing::State.

3. Load some TeX source code into the VM using the VM's push_source method.

4. Call the VM's run method, or some other helper function, to run the VM.

One minor complication at this point is that to call the VM's run method directly one needs to provide so-called "VM handlers". These are Rust functions that tell the VM what to do when it encounters certain kinds of TeX tokens. For example, when a real typesetting VM sees the character a, it typesets the character a. Handlers are described in detail in the VM hooks and handlers section. For the moment, we're going to get around the handlers problem entirely by instead running the VM using the ::texlang_stdlib::script::run function. This function automatically provides handlers such that when the VM sees a character, it just prints the character to the terminal.

With all of this context, here is a minimal code listing that runs the Texlang VM:

#![allow(unused)]
fn main() {
extern crate texlang_stdlib;
extern crate texlang;
use texlang::{vm, command};
use texlang_stdlib::StdLibState;
use texlang_stdlib::script;

// 1. Create a new map of built-in commands.
//    In this book's next section we will add some commands here.
let built_in_commands = std::collections::HashMap::new();

// 2. Initialize the VM.
let mut vm = vm::VM::<StdLibState>::new_with_built_in_commands(built_in_commands);

// 3. Add some TeX source code the VM.
vm.push_source("input.tex", r"Hello, World.");

// 4. Run the VM and write the results to stdout.
script::set_io_writer(&mut vm, std::io::stdout());
script::run(&mut vm).unwrap();
}

When you run this code, you will simply see:

Hello, World.

In this case the VM has essentially nothing to do: it just passes characters from the input to the handler and thus the terminal. To see the VM doing a little work at least, change the source code to this:

#![allow(unused)]
fn main() {
extern crate texlang_stdlib;
extern crate texlang;
use texlang::{vm, command};
let mut vm = vm::VM::<()>::new_with_built_in_commands(Default::default());
// 3. Add some TeX source code the VM.
vm.push_source("input.tex", r"Hello, {World}.");
}

The output here is the same as before - in particular, the braces {} disappear:

Hello, World.

The braces disappear because they are special characters in the TeX grammar, and are used to denote the beginning and ending of a group. The VM consumes these characters internally when processing the input.

Another thing the VM can do is surface an error if the input contains an undefined control sequence. Because we haven't provided any built-in commands, every control sequence is undefined. Changing the VM setup to the following:

#![allow(unused)]
fn main() {
extern crate texlang_stdlib;
extern crate texlang;
use texlang::{vm, command};
use texlang_stdlib::StdLibState;
use texlang_stdlib::script;
let mut vm = vm::VM::<StdLibState>::new_with_built_in_commands(Default::default());
// 3. Add some TeX source code the VM.
vm.push_source("input.tex", r"\controlSequence");

// 4. Run the VM and write the results to stdout.
script::set_io_writer(&mut vm, std::io::stdout());
if let Err(err) = script::run(&mut vm){
    println!["{err}"];
}
}

results in the following output:

Error: undefined control sequence \controlSequence
 >>> input.tex:1:1
  | 
1 | \controlSequence
  | ^^^^^^^^^^^^^^^^ control sequence

In general, though, the VM can't do much if no built-in commands are provided. In the next section we will learn how to write some.

Simple expansion and execution primitives

Primitives with state (the component pattern)

Primitive tags

VM hooks and handlers

TeX variables

The documentation so far has covered two of three type of primitives in TeX: expansion primitives and execution primtives. In this section we discuss the third type: variables.

The TeX language includes typed variables. There are a few possible types which are listed below. The Texlang variables API is the mechanism by which TeX variables are implemented in Texlang. Ultimately the API provides a way for memory in Rust to be reference in TeX source code.

[History] Variables in the TeXBook

This section can be freely skipped.

The TeXBook talks a lot about the different variables that are available in the original TeX engine, like \year and \count. The impression one sometimes gets from the TeXBook is that these variables are heterogeneous in terms of how they are handled by the interpreter. For example, on page 271 when describing the grammar of the TeX language, we find the following rule:

<internal integer> -> <integer parameter> | <special integer> | \lastpenalty
    <countdef token> | \count<8-bit number>
    ...

This makes it seems that an "integer parameter" behaves differently to a "special integer", or to a register accessed via \count, even though all of these variables have the same concrete type (a 32 bit integer). To the best of our current knowledge, this is not the case at all! It appears that there is a uniform way to handle all variables of the same concrete type in TeX, and this is what Texlang does. The benefit of this approach is that it makes for a much simpler API, both for adding new variables to a TeX engine or for consuming variables.

Singleton versus array variables

In TeX, for each variable type, there are two categories of variable: singleton variables (like \year) and array variables (like \count N, where N is the index of the variable in the registers array). Both of these cases are handled in the same way in the Texlang API.

In Texlang, the control sequences \year and \count are not considered variables themselves. This is because without reading the N after \count, we don't actually know which memory is being referred to. Instead, \year and \count are variable commands (of type Command). A variable command is resolved to obtain a variable (of type Variable). A variable is an object that points to a specific piece of memory like an i32 in the state.

For singleton variables, resolving a command is a no-op. The command itself has enough information to identify the memory being pointed at.

For array variables, resolving a command involves determining the index of the variable within the array. The index has type Index, which is a wrapper around Rust's usize. The index is determined using the command's index resolver, which has enum type IndexResolver. There are a two different ways the index can be resolved, corresponding to different variants in the enum type.

Implementing a singleton variable

Variables require some state in which to store the associated value. We assume that the component pattern is being used. In this case, the variable command is associated with a state struct which will be included in the VM state as a component. The value of a variable is just a Rust field of the correct type in the component:

#![allow(unused)]
fn main() {
extern crate texlang;
pub struct MyComponent {
    my_variable_value: i32
}
}

To make a Texlang variable out of this i32 we need to provide two things: an immutable getter and a mutable getter. These getters have the signature RefFn and MutRefFn respectively. Both getters accept a reference to the state and an index, and return a reference to the variable. (The index is only for array variables, and is ignored for singleton variables.)

For the component and variable above, our getters look like this:

#![allow(unused)]
fn main() {
extern crate texlang;
pub struct MyComponent {
    my_variable_value: i32
}
use texlang::vm::HasComponent;
use texlang::variable;

fn getter<S: HasComponent<MyComponent>>(state: &S, index: variable::Index) -> &i32 {
    &state.component().my_variable_value
}
fn mut_getter<S: HasComponent<MyComponent>>(state: &mut S, index: variable::Index) -> &mut i32 {
    &mut state.component_mut().my_variable_value
}
}

Once we have the getters, we can create the TeX variable command:

#![allow(unused)]
fn main() {
extern crate texlang;
pub struct MyComponent {
    my_variable_value: i32
}
use texlang::vm::HasComponent;
fn getter<S: HasComponent<MyComponent>>(state: &S, index: variable::Index) -> &i32 {
   &state.component().my_variable_value
}
fn mut_getter<S: HasComponent<MyComponent>>(state: &mut S, index: variable::Index) -> &mut i32 {
   &mut state.component_mut().my_variable_value
}
use texlang::variable;
use texlang::command;

pub fn my_variable<S: HasComponent<MyComponent>>() -> command::BuiltIn<S> {
    return variable::Command::new_singleton(
        getter,
        mut_getter,
    ).into()
}
}

The function Command::new_singleton creates a new variable command associated to a singleton variable. We cast the variable command into a generic command using the into method. This command can now be included in the VM's command map and the value can be accessed in TeX scripts!

As usual with the component pattern, the code we write works for any TeX engine whose state contains our component.

Finally, as a matter of style, consider implementing the getters inline as closures. This makes the code a little more compact and readable. With this style, the full code listing is as follows:

#![allow(unused)]
fn main() {
extern crate texlang;
use texlang::vm::HasComponent;
use texlang::variable;
use texlang::command;

pub struct MyComponent {
    my_variable_value: i32
}

pub fn my_variable<S: HasComponent<MyComponent>>() -> command::BuiltIn<S> {
    return variable::Command::new_singleton(
        |state: &S, index: variable::Index| -> &i32 {
            &state.component().my_variable_value
        },
        |state: &mut S, index: variable::Index| -> &mut i32 {
            &mut state.component_mut().my_variable_value
        },
    ).into()
}
}

Implementing an array variable

The main difference between singleton and array variables is that we need to use the index arguments that were ignored above.

In this section we will implement an array variable with 10 entries. In the component, we replace the i32 with an array of i32s:

#![allow(unused)]
fn main() {
pub struct MyComponent {
    my_array_values: [i32; 10]
}
}

The getter functions use the provided index argument to determine the index to use for the array:

#![allow(unused)]
fn main() {
extern crate texlang;
pub struct MyComponent {
    my_array_values: [i32; 10]
}
use texlang::vm::HasComponent;
use texlang::variable;
fn getter<S: HasComponent<MyComponent>>(state: &S, index: variable::Index) -> &i32 {
    &state.component().my_array_values[index.0 as usize]
}
}

The above listing raises an important question: what if the array access is out of bounds? The Rust code here will panic, and in Texlang this is the correct behavior. Texlang always assumes that variable getters are infallible. This is the same as assuming that an instantiated [Variable] type points to a valid piece of memory and is not (say) dangling.

Next, we construct the command. Unlike the singleton command, this command will need to figure out the index of the variable. As with \count, our command will do this by reading the index from the input token stream. In the variables API, we implement this by providing the following type of function:

#![allow(unused)]
fn main() {
extern crate texlang;
use texlang::*;
use texlang::traits::*;

fn index<S: TexlangState>(token: token::Token, input: &mut vm::ExpandedStream<S>) -> command::Result<variable::Index> {
    let index = parse::Uint::<10>::parse(input)?;
    return Ok(index.0.into())
}
}

Finally we create the command. This is the same as the singleton case, except we pass the index function above as an index resolver with the Dynamic variant:

#![allow(unused)]
fn main() {
extern crate texlang;
use texlang::*;
fn getter<S: HasComponent<MyComponent>>(state: &S, index: variable::Index) -> &i32 {
  panic![""]
}
fn mut_getter<S: HasComponent<MyComponent>>(state: &mut S, index: variable::Index) -> &mut i32 {
  panic![""]
}
fn index_resolver<S>(token: token::Token, input: &mut vm::ExpandedStream<S>) -> command::Result<variable::Index> {
  panic![""]
}
pub struct MyComponent {
    my_array_values: [i32; 10]
}
use texlang::vm::HasComponent;
pub fn my_array<S: HasComponent<MyComponent>>() -> command::BuiltIn<S> {
    return variable::Command::new_array(
        getter,
        mut_getter,
        variable::IndexResolver::Dynamic(index_resolver),
    ).into()
}
}

Implementing a \countdef type command

In Knuth's TeX, the \countdef command is an execution command with the following semantics. After executing the TeX code \countdef \A 1, the control sequence \A will be a variable command pointing to the same memory as \count 1. One way of thinking about it is that \A aliases \count 1.

Using the Texlang variables API it is possible to implement the analogous command for the \myArray command implemented above. The implementation is in 3 steps:

1. The target (e.g. \A) is read using texlang::parse::Command::parse.

2. The index (e.g. 1) is read using usize::parse, just like in the previous section.

3. A new variable command is then created and added to the commands map. This command is created using [Command::new_array] just as above, except in the index resolver we use the [IndexResolver::Static] variant with the index calculated in part 2.

For a full example where this is all worked out, consult the implementation of \countdef in the Texlang standard library.

TeX variable types

Not all variable types have been implemented yet.

Parsing the TeX grammar

Error handling

Unit testing

When writing code that uses Texlang it's generally expected that unit tests will also be written to verify the primitives being implemented work as expected.

Software projects are more likely to have high-quality, extensive unit tests if it is easy to write and maintain such tests. In order to support easy unit-testing of code that uses Texlang, the Texcraft project includes a specific crate for writing unit tests called texlang_testing. The crate currently supports three kinds of unit tests:

• Expansion equality tests: verifying that two TeX snippets expand to the same output.

• Failure tests: verifying that a TeX snippet fails to run.

• Serde tests: verifying that a Texlang VM can be successfully serialized and deserialized in the middle of executing a TeX snippet.

This crate is used extensively in the Texlang standard library. Browsing some of the Rust source code will give a good sense of how tests are written using this library.

For information on writing unit tests using this crate, consult the crate's documentation.

Format files (a.k.a. serialization and deserialization)

Knuth's original implementation of TeX includes a feature called "format files". A TeX format is a set of general-purpose macros and other configurations such as category code mappings that are included as a preamble in TeX documents. The plain TeX format was developed by Knuth concurrently with the initial implementation of TeX. Nowadays the LaTeX format is so ubiquitous as to be synonymous with TeX itself.

A format file is created by running TeX, inputting the format definitions (which are in regular .tex files) and then dumping the state of the interpreter using the \dump primitive. The resulting format file has the file extension .fmt. Subsequent runs of the VM can then read the format file and apply the definitions "at high speed". The format file mechanism is essentially a performance optimization that gets the format into the interpreter faster than parsing the .tex definitions each time. This optimization was probably especially important when TeX was being developed in the early 80s and computers were much slower than today.

A modern perspective on format files is that they are a mechanism for serializing and deserializing the state of a TeX virtual machine. Texlang includes support for such serialization and deserialization of VMs. In fact, Texlang's (de)serialization feature is strictly more powerful than the format files mechanism in Knuth's TeX:

• Texlang's (de)serialization feature is implemented using the Rust library Serde, and is thus independent of any specific serialization format. Texlang VMs can be (de)serialized to and from any format compatible with Serde. All of the unit tests in the Texlang project are run for three formats: message pack, bincode, and JSON.

• Texlang VMs can be serialized irrespective of their internal state. With format files this is not the case: format files cannot be created when there is a current active group, or when typesetting has already started.

This latter property opens up some exciting use cases for Texlang (de)serialization, especially checkpoint compilation. In theory, after shipping out each PDF page a Texlang VM could checkpoint its progress by serializing itself and persisting the bytes in the filesystem. Then, when the same TeX document is compiled, the interpreter could check if the document hasn't changed up to a certain checkpoint. If so, instead of recompiling the entire document, the checkpoint could be deserialized and compilation could continue from the checkpoint. This would offer genuine O(1) generation of the Nth page in a TeX document.

Making VMs (de)serializable

Texlang VMs are generic over the state. In order for a Texlang VM to be (de)serializable, it is only necessary that the state itself be (de)serializable using Serde. I.e., the state must satisfy the serde::Serialize and serde::Deserialize traits. As usual, implementations of these traits can usually be generated automatically using Serde's derive macro.

A note on tags

In a previous section we discussed primitive tags. These provide unique identifiers that are generated using a global counter at runtime. Tags sometimes appear in the state, but they are not safe to serialize and deserialize. Deserialized tags may collide with new tags generated using the global counter.

Instead, when (de)serializing a component that contains a tag, the tag field should be skipped when serializing and the value should be provided manually when deserializing. This can be achieved using Serde's skip field attribute:

#![allow(unused)]
fn main() {
extern crate serde;
extern crate texlang;
use serde::{Serialize, Deserialize};
use texlang::command;

static TAG: command::StaticTag = command::StaticTag::new();

fn get_tag() -> command::Tag {
    TAG.get()
}

#[derive(Serialize, Deserialize)]
struct Component {
    variable_value: i32,
    #[serde(skip, default="get_tag")]
    tag: command::Tag,
}

// This Default implementation is not needed for (de)serializing.
// However it illustrates how to ensure that new and deserialized components have the same tag.
impl Default for Component {
    fn default() -> Self {
        Self {
            variable_value: 0,
            tag: get_tag(),
        }
    }
}
}

Another approach is to extract the tag to its own sub-struct and then manually implement the Default trait for that sub-struct. In this case we instruct Serde to use the default value for the sub-struct when deserializing:

#![allow(unused)]
fn main() {
extern crate serde;
extern crate texlang;
use serde::{Serialize, Deserialize};
use texlang::command;

static TAG: command::StaticTag = command::StaticTag::new();

#[derive(Serialize, Deserialize)]
struct Component {
    variable_value: i32,
    #[serde(skip)]
    tags: Tags,
}

struct Tags {
    tag: command::Tag,
}

impl Default for Tags {
    fn default() -> Self {
        Self {
            tag: TAG.get(),
        }
    }
}
}

Serializing and deserializing VMs in Rust

The previous subsection discussed how to make VMs (de)serializable; in this subsection we actually do it.

Serializing VMs is pretty straightforward because Texlang's VM satisfies Serde's Serialize trait. Thus to output an empty VM to JSON:

#![allow(unused)]
fn main() {
extern crate serde_json;
extern crate texlang;
use texlang::vm;

let built_in_commands = Default::default();
let vm = vm::VM::<()>::new_with_built_in_commands(built_in_commands);
let serialized_vm = serde_json::to_string_pretty(&vm).unwrap();
println!["{serialized_vm}"];
}

Deserialization is a little more tricky because the serialized bytes are insufficient to reconstruct the VM. The VM's built-in commands must be provided again at deserialization time. This is because, fundamentally, Texlang primitives are Rust function pointers and these cannot be serialized and deserialized.

The easiest way to support deserialization is to implement [vm::HasDefaultBuiltInCommands] for the state type. For a given state type, this trait provides the default set of built-in commands for that type. If this trait is implemented, the VM automatically satisfies the serde::Deserialize trait and the type be used in the idiomatic serde way.

#![allow(unused)]
fn main() {
extern crate serde;
extern crate serde_json;
extern crate texlang;
use std::collections::HashMap;
use texlang::vm;
use texlang::command;

#[derive(Default, serde::Serialize, serde::Deserialize)]
struct State;
impl vm::TexlangState for State {}
impl vm::HasDefaultBuiltInCommands for State {
    fn default_built_in_commands() -> HashMap<&'static str, command::BuiltIn<Self>> {
        // Returning an empty set of built-in commands, but in general this will be non-empty.
        HashMap::new()
    }
}

// When `vm::HasDefaultBuiltInCommands` is implemented for the state,
// the VM's plain `new` constructor can be used.
let original_vm = vm::VM::<State>::new();
let serialized_vm = serde_json::to_string_pretty(&original_vm).unwrap();
println!["{serialized_vm}"];

let deserialized_vm: vm::VM::<State> = serde_json::from_str(&serialized_vm).unwrap();
}

If the state doesn't implement [vm::HasDefaultBuiltInCommands], or you are using a non-default set of built-in commands, deserialization can be done in one of two ways. First way: use the VM::deserialize_with_built_in_commands helper function that accepts a Serde deserializer and the built-in commands:

#![allow(unused)]
fn main() {
extern crate serde_json;
extern crate texlang;
use texlang::vm;

let built_in_commands = Default::default();
let vm = vm::VM::<()>::new_with_built_in_commands(built_in_commands);
let serialized_vm = serde_json::to_string_pretty(&vm).unwrap();
let built_in_commands = Default::default();

let mut deserializer = serde_json::Deserializer::from_str(&serialized_vm);
let vm = vm::VM::<()>::deserialize_with_built_in_commands(&mut deserializer, built_in_commands);
}

Second way: first deserialize the bytes to a vm::serde::DeserializedVM type, and the convert this type into a regular VM using the vm::serde::finish_deserialization function:

#![allow(unused)]
fn main() {
extern crate serde_json;
extern crate texlang;
use texlang::vm;

let built_in_commands = Default::default();
let vm = vm::VM::<()>::new_with_built_in_commands(built_in_commands);
let serialized_vm = serde_json::to_string_pretty(&vm).unwrap();
let built_in_commands = Default::default();

let deserialized_vm: Box<vm::serde::DeserializedVM<()>> = serde_json::from_str(&serialized_vm).unwrap();
let vm = vm::serde::finish_deserialization(deserialized_vm, built_in_commands);
}

Serializing VMs in TeX

Using Rust code like in the previous subsection, it's possible to write TeX primitives that serialize the VM and write the result to a file - i.e., write a format file! The Texlang standard library includes an implementation of the \dump primitive that does this. The texcraft binary accepts a --format-file argument that reads the format files, and continues from where it left off.

Fuzz testing

tfm crate

From the root of the repository:

cargo fuzz run --fuzz-dir crates/tfm/fuzz --dev fuzz_tftopl

The --dev flag may be optional, but on my computer it results in linker errors.

If a failing input is found and Texcraft does not panic, the fuzzing harness will automatically add data for a test case to crates/tfm/bin/tests/data. A new test can then be added in convert.rs.

In the case when Texcraft panics (or in general), run the fuzzer with TEXCRAFT_FUZZ_OUTPUT_ALWAYS=1 to always write test cases out.

License

The Texcraft project is dual-licensed under the Apache License, Version 2.0 or the MIT license.

This dual-license scheme is the convention in the Rust ecosystem. The Texcraft project aims to keep this license scheme in the long run. However the TeX ecosystem at large generally prefers the GPL, and some future parts of Texcraft may need to be GPL'd too depending on how the project develops. See use of existing TeX source code below for more information.

There are some additional files in the repository, used only for testing, that have different licenses.

Contributing

Copyrights in the Texcraft project are retained by their contributors. No copyright assignment is required to contribute to the Texcraft project. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Texcraft by you, as defined in the Apache-2.0 license, shall be dual licensed as described above, without any additional terms or conditions.

Use of existing TeX source code

Rust code in the Texcraft project is written manually. However its logical content is sometimes heavily based on source code in the wider TeX ecosystem written by Donald Knuth and others. When developing parts of Texcraft that are clear re-implementations of existing TeX functionality (for example implementing TeX user-defined macros) we typically study the original source code closely, reference the original source code within the Texcraft source (example) and in some cases simply translate small source code fragments into Rust (example). This development process is essentially required in order to fulfill our goal of 100% compatibility with the original implementation of TeX.

This process does impact licensing because from a copyright perspective Texcraft is a derived work of the source code we consult this closely. (Our process is the opposite of a "clean room design" in which one attempts to re-implement a system without consulting the source code and thus be free of copyright restrictions.) So far all of the sources we've used are in the public domain, and this allows us to dual-license our code as above.

We have used the following sources:

Index of Texcraft Rust crates

The documentation on docs.rs is the latest uploaded version, which may be quite old. The texcraft.dev version is from the most recent deployment of the website.