maili

Welcome to the hands-on guide for getting started with maili!

maili connects the OP Stack, debuting unique types and interfaces linked to Ethereum via OP-Alloy primitives.

📖 Development Status

maili is in active development, and is not yet ready for use in production. During development, this book will evolve quickly and may contain inaccuracies.

Please open an issue if you find any errors or have any suggestions for improvements, and also feel free to contribute to the project!

Sections

Getting Started

To get started with maili, add its crates as a dependency and take your first steps.

Building with maili

Walk through types and functionality available in different maili crates.

Examples

Get hands-on experience using maili crates for critical OP Stack functionality.

Contributing

Contributors are welcome! It is built and maintained by op-rs contributors, members of OP Labs and the broader open source community.

maili follows and expands the OP Stack standards set in the specs. The contributing guide breaks down how the specs integrate with maili and how to contribute to maili.

Licensing

maili is licensed under the combined Apache 2.0 and MIT License, along with a SNAPPY license for snappy encoding use.

Installation

[maili][maili] consists of a number of crates that provide a range of functionality essential for interfacing with any OP Stack chain.

The most succinct way to work with maili is to add the maili crate with the full feature flag from the command-line using Cargo.

cargo add maili --features full

Alternatively, you can add the following to your Cargo.toml file.

maili = { version = "0.5", features = ["full"] }

For more fine-grained control over the features you wish to include, you can add the individual crates to your Cargo.toml file, or use the maili crate with the features you need.

After maili is added as a dependency, crates re-exported by maili are now available.

#![allow(unused)]
fn main() {
use maili::{
   protocol::BlockInfo,
   provider::ext::engine::OpEngineApi,
};
}

Features

The maili defines many feature flags including the following.

Default

  • std
  • k256
  • serde

Full enables the most commonly used crates.

  • full

The k256 feature flag enables the k256 feature on the maili-consensus crate.

  • k256

Arbitrary enables arbitrary features on crates, deriving the Arbitrary trait on types.

  • arbitrary

Serde derives serde's Serialize and Deserialize traits on types.

  • serde

Additionally, individual crates can be enabled using their shorthand names. For example, the protocol feature flag provides the maili-protocol re-export so maili-protocol types can be used from maili through maili::protocol::InsertTypeHere.

Crates

no_std

As noted above, the following crates are no_std compatible.

To add no_std support to a crate, ensure the check_no_std script is updated to include this crate once no_std compatible.

Building

This section offers in-depth documentation into the various maili crates. Some of the primary crates and their types are listed below.

Engine RPC Types

Protocol

maili-protocol crate

The maili-protocol crate contains types, constants, and methods specific to Optimism derivation and batch-submission.

maili-protocol supports no_std.

Background

Protocol types are primarily used for L2 chain derivation. This section will break down L2 chain derivation as it relates to types defined in maili-protocol - that is, from the raw L2 chain data posted to L1, to the Batch type. And since the Batch type naively breaks up into the payload attributes, once executed, it becomes the canonical L2 block! Note though, this provides an incredibly simplified introduction. It is advised to reference the specs for the most up-to-date information regarding derivation.

The L2 chain is derived from data posted to the L1 chain - either as calldata or blob data. Data is iteratively pulled from each L1 block and translated into the first type defined by maili-protocol: the Frame type.

Frames are parsed from the raw data. Each Frame is a part of a Channel, the next type one level up in deriving L2 blocks. Channels have IDs that frames reference. Frames are added iteratively to the Channel. Once a Channel is ready, it can be used to read a Batch.

Since a Channel stitches together frames, it contains the raw frame data. In order to turn this Channel data into a Batch, it needs to be decompressed using the respective (de)compression algorithm (see the channel specs for more detail on this). Once decompressed, the raw data can be decoded into the Batch type.

Sections

Core Derviation Types (discussed above)

Other Critical Protocol Types

BlockInfo and L2BlockInfo Types

Optimism defines block info types that encapsulate minimal block header information needed by protocol operations.

BlockInfo

The BlockInfo type is straightforward, containing the block hash, number, parent hash, and timestamp.

L2BlockInfo

The L2BlockInfo extends the BlockInfo type for the canonical L2 chain. It contains the "L1 origin" which is a set of block info for the L1 block that this L2 block "originated".

L2BlockInfo provides a from_block_and_gensis method to construct the L2BlockInfo from a block and ChainGenesis.

Frames

Frames are the lowest level data format in the OP Stack protocol.

Where Frames fit in the OP Stack

Transactions posted to the data availability layer of the rollup contain one or multiple Frames. Frames are chunks of raw data that belong to a given Channel, the next, higher up data format in the OP Stack protocol. Importantly, a given transaction can contain a variety of frames from different channels, allowing maximum flexibility when breaking up channels into batcher transactions.

Contents of a Frame

A Frame is comprised of the following items.

  • A ChannelId which is a 16 byte long identifier for the channel that the given frame belongs to.
  • A number that identifies the index of the frame within the channel. Frames are 0-indexed and are bound to u16 size limit.
  • data contains the raw data within the frame.
  • is_last marks if the frame is the last within the channel.

Frame Encoding

When frames are posted through a batcher transaction, they are encoded as a contiguous list with a single byte prefix denoting the derivation version. The encoding can be represented as the following concatenated bytes.

encoded = DERIVATION_VERSION_0 ++ encoded_frame_0 ++ encoded_frame_1 ++ ..

Where DERIVATION_VERSION_0 is a single byte (0x00) indicating the derivation version including how the frames are encoded. Currently, the only supported derivation version is 0.

encoded_frame_0, encoded_frame_1, and so on, are all Frames encoded as raw bytes. A single encoded Frame can be represented by the following concatenation of it's fields.

encoded_frame = channel_id ++ frame_number ++ frame_data_length ++ frame_data ++ is_last

Where ++ represents concatenation. The frame's fields map to it's encoding.

  • channel_id is the 16 byte long Frame::id.
  • frame_number is the 2 byte long (or u16) Frame::number.
  • frame_data_length and frame_data provide the necessary details to decode the Frame::data, where frame_data_length is 4 bytes long (or u32).
  • is_last is a single byte Frame::is_last.

maili's Frame Type

maili-protocol provides the Frame type with a few useful methods. Frames can be encoded and decoded using the Frame::encode and Frame::decode methods. Given the raw batcher transaction data or blob data containing the concatenated derivation version and contiguous list of encoded frames, the Frame::parse_frame and Frame::parse_frames methods provide ways to decode single and multiple frames, respectively.

Channels

Taken from the OP Stack specs, Channels are a set of sequencer batches (for any L2 blocks) compressed together.

Where Channels fit in the OP Stack

L2 transactions are grouped into what are called sequencer batches. In order to obtain a better compression ratio when posting these L2 transactions to the data availability layer, sequencer batches are compressed together into what is called a Channel. This ultimately reduces data availability costs. As previously noted in the Frame section, Channels may not "fit" in a single batcher transaction, posting the data to the data availability layer. In order to accommodate large Channels, a tertiary Frame data type breaks the Channel up into multiple Frames where a batcher transaction then consists of one or multiple Frames.

Contents of a Channel

A Channel is comprised of the following items.

  • A ChannelId which is a 16 byte long identifier for the channel. Notice, Frames also contain a ChannelId, which is the identical to this identifier, since frames "belong" to a given channel.
  • A BlockInfo that marks the L1 block at which the channel is "opened" at.
  • The estimated size of the channel (as a usize) used to drop the channel if there is a data overflow.
  • A boolean if the channel is "closed". This indicates if the last frame has been buffered, and added to the channel.
  • A u16 indicating the highest frame number within the channel.
  • The frame number of the last frame (where is_last set to true).
  • A mapping from Frame number to the Frame itself.
  • A BlockInfo for highest L1 inclusion block that a frame was included in.

Channel Encoding

Channel encoding is even more straightforward than that of a Frame. Simply, a Channel is the concatenated list of encoded Frames.

Since each Frame contains the ChannelId that corresponds to the given Channel, constructing a Channel is as simple as calling the Channel::add_frame method for each of its Frames.

Once the Channel has ingested all of it's Frames, it will be marked as "ready", with the Channel::is_ready method returning true.

The Channel Type

As discussed above, the Channel type is expected to be populated with Frames using its Channel::add_frame method. Below we demonstrate constructing a minimal Channel using a few frames.

#![allow(unused)]
fn main() {
use maili_protocol::{Channel, Frame};

// Construct a channel at the given L1 block.
let id = [0xee; 16];
let block = BlockInfo::default();
let mut channel = Channel::new(id, block);

// The channel will consist of 3 frames.
let frame_0 = Frame { id: [0xee; 16], number: 0, ..Default::default() };
let frame_1 = Frame { id: [0xee; 16], number: 1, ..Default::default() };
let frame_2 = Frame { id: [0xee; 16], number: 2, is_last: true, ..Default::default() };

// Add the frames to the channel.
channel.add_frame(frame_0);
channel.add_frame(frame_1);
channel.add_frame(frame_2);

// Since the last frame was ingested,
// the channel should be ready.
assert!(channel.is_ready());
}

There are a few rules when adding a Frame to a Channel.

Notice, Frames can be added out-of-order so long as the Channel is still open, and the frame hasn't already been added.

Batches

A Batch contains a list of transactions to be included in a specific L2 block. Since the Delta hardfork, there are two Batch types or variants: SingleBatch and SpanBatch.

Where Batches fit in the OP Stack

The Batch is the highest-level data type in the OP Stack derivation process that comes prior to building payload attributes. A Batch is constructed by taking the raw data from a Channel, decompressing it, and decoding the Batch from this decompressed data.

Alternatively, when looking at the Batch type from a batching perspective, and not from the derivation perspective, the Batch type contains a list of L2 transactions and is compressed into the Channel type. In turn, the Channel is split into frames which are posted to the data availability layer through batcher transactions.

Contents of a Batch

A Batch is either a SingleBatch or a SpanBatch, each with their own contents. Below, these types are broken down in their respective sections.

SingleBatch Type

The SingleBatch type contains the following.

  • A BlockHash parent hash that represents the parent L2 block.
  • A u64 epoch number that identifies the epoch for this batch.
  • A BlockHash epoch hash.
  • The timestamp for the batch as a u64.
  • A list of EIP-2718 encoded transactions (represented as Bytes).

In order to validate the SingleBatch once decoded, the SingleBatch::check_batch method should be used, providing the rollup config, l1 blocks, l2 safe head, and inclusion block.

SpanBatch Type

The SpanBatch type (available since the Delta hardfork) comprises the data needed to build a "span" of multiple L2 blocks. It contains the following data.

  • The parent check (the first 20 bytes of the block's parent hash).
  • The l1 origin check (the first 20 bytes of the last block's l1 origin hash).
  • The genesis timestamp.
  • The chain id.
  • A list of SpanBatchElements. These are similar to the SingleBatch type but don't contain the parent hash and epoch hash for this L2 block.
  • Origin bits.
  • Block transaction counts.
  • Span batch transactions which contain information for transactions in a span batch.

Similar to the SingleBatch type discussed above, the SpanBatch type must be validated once decoded. For this, the SpanBatch::check_batch method is available.

After the Holocene hardfork was introduced, span batch validation is greatly simplified to be forwards-invalidating instead of backwards-invalidating, so a new SpanBatch::check_batch_prefix method provides a way to validate each batch as it is loaded, in an iterative fashion.

Batch Encoding

The first byte of the decompressed channel data is the BatchType, which identifies whether the batch is a SingleBatch or a SpanBatch. From there, the respective type is decoded, and derived in the case of the SpanBatch.

The Batch encoding format for the SingleBatch is broken down in the specs.

The Batch Type

The Batch type itself only provides two useful methods.

  • timestamp returns the timestamp of the Batch
  • deocde, constructs a new Batch from the provided raw, decompressed batch data and rollup config.

Within each Batch variant, the individual types contain more functionality.

Examples

Examples for working with maili-* crates.

Transform Frames into a Batch

note

This example performs the reverse transformation as the batch-to-frames example.

caution

Steps and handling of types with respect to chain tip, ordering of frames, re-orgs, and more are not covered by this example. This example solely demonstrates the most trivial way to transform individual Frames into a Batch type.

This example walks through transforming Frames into the Batch types.

Walkthrough

The high level transformation is the following.

raw bytes[] -> frames[] -> channel -> decompressed channel data -> Batch

Given the raw, batch-submitted frame data as bytes (read in with the hex! macro), the first step is to decode the frame data into Frames using Frame::decode. Once all the Frames are decoded, the Channel can be constructed using the ChannelId of the first frame.

note

Frames may also be added to a Channel once decoded with the Channel::add_frame method.

When the Channel is Channel::is_ready(), the frame data can taken from the Channel using Channel::frame_data(). This data is represented as Bytes and needs to be decompressed using the respective compression algorithm depending on which hardforks are activated (using the RollupConfig). For the sake of this example, brotli is used (which was activated in the Fjord hardfork). Decompressed brotli bytes can then be passed right into Batch::decode to wind up with the example's desired Batch.

Running this example:

  • Clone the examples repository: git clone git@github.com:op-rs/maili.git
  • Run: cargo run --example frames_to_batch
//! This example decodes raw [Frame]s and reads them into a [Channel] and into a [SingleBatch].

use alloy_consensus::{SignableTransaction, TxEip1559};
use alloy_eips::eip2718::{Decodable2718, Encodable2718};
use alloy_primitives::{hex, Address, BlockHash, Bytes, PrimitiveSignature, U256};
use maili_protocol::{decompress_brotli, Batch, BlockInfo, Channel, Frame, SingleBatch};
use op_alloy_consensus::OpTxEnvelope;
use op_alloy_genesis::RollupConfig;

fn main() {
    // Raw frame data taken from the `encode_channel` example.
    let first_frame = hex!("60d54f49b71978b1b09288af847b11d200000000004d1b1301f82f0f6c3734f4821cd090ef3979d71a98e7e483b1dccdd525024c0ef16f425c7b4976a7acc0c94a0514b72c096d4dcc52f0b22dae193c70c86d0790a304a08152c8250031d091063ea000");
    let second_frame = hex!("60d54f49b71978b1b09288af847b11d2000100000046b00d00005082edde7ccf05bded2004462b5e80e1c42cd08e307f5baac723b22864cc6cd01ddde84efc7c018d7ada56c2fa8e3c5bedd494c3a7a884439d5771afcecaf196cb3801");

    // Decode the raw frames.
    let decoded_first = Frame::decode(&first_frame).expect("decodes frame").1;
    let decoded_second = Frame::decode(&second_frame).expect("decodes frame").1;

    // Create a channel.
    let id = decoded_first.id;
    let open_block = BlockInfo::default();
    let mut channel = Channel::new(id, open_block);

    // Add the frames to the channel.
    let l1_inclusion_block = BlockInfo::default();
    channel.add_frame(decoded_first, l1_inclusion_block).expect("adds frame");
    channel.add_frame(decoded_second, l1_inclusion_block).expect("adds frame");

    // Get the frame data from the channel.
    let frame_data = channel.frame_data().expect("some frame data");
    println!("Frame data: {}", hex::encode(&frame_data));

    // Decompress the frame data with brotli.
    let config = RollupConfig::default();
    let max = config.max_rlp_bytes_per_channel(open_block.timestamp) as usize;
    let decompressed = decompress_brotli(&frame_data, max).expect("decompresses brotli");
    println!("Decompressed frame data: {}", hex::encode(&decompressed));

    // Decode the single batch from the decompressed data.
    let batch = Batch::decode(&mut decompressed.as_slice(), &config).expect("batch decodes");
    assert_eq!(
        batch,
        Batch::Single(SingleBatch {
            parent_hash: BlockHash::ZERO,
            epoch_num: 1,
            epoch_hash: BlockHash::ZERO,
            timestamp: 1,
            transactions: example_transactions(),
        })
    );

    println!("Successfully decoded frames into a Batch");
}

fn example_transactions() -> Vec<Bytes> {
    let mut transactions = Vec::new();

    // First Transaction in the batch.
    let tx = TxEip1559 {
        chain_id: 10u64,
        nonce: 2,
        max_fee_per_gas: 3,
        max_priority_fee_per_gas: 4,
        gas_limit: 5,
        to: Address::left_padding_from(&[6]).into(),
        value: U256::from(7_u64),
        input: vec![8].into(),
        access_list: Default::default(),
    };
    let sig = PrimitiveSignature::test_signature();
    let tx_signed = tx.into_signed(sig);
    let envelope: OpTxEnvelope = tx_signed.into();
    let encoded = envelope.encoded_2718();
    transactions.push(encoded.clone().into());
    let mut slice = encoded.as_slice();
    let decoded = OpTxEnvelope::decode_2718(&mut slice).unwrap();
    assert!(matches!(decoded, OpTxEnvelope::Eip1559(_)));

    // Second transaction in the batch.
    let tx = TxEip1559 {
        chain_id: 10u64,
        nonce: 2,
        max_fee_per_gas: 3,
        max_priority_fee_per_gas: 4,
        gas_limit: 5,
        to: Address::left_padding_from(&[7]).into(),
        value: U256::from(7_u64),
        input: vec![8].into(),
        access_list: Default::default(),
    };
    let sig = PrimitiveSignature::test_signature();
    let tx_signed = tx.into_signed(sig);
    let envelope: OpTxEnvelope = tx_signed.into();
    let encoded = envelope.encoded_2718();
    transactions.push(encoded.clone().into());
    let mut slice = encoded.as_slice();
    let decoded = OpTxEnvelope::decode_2718(&mut slice).unwrap();
    assert!(matches!(decoded, OpTxEnvelope::Eip1559(_)));

    transactions
}

Transform a Batch into Frames

note

This example performs the reverse transformation as the frames-to-batch example.

caution

Steps and handling of types with respect to chain tip, ordering of frames, re-orgs, and more are not covered by this example. This example solely demonstrates the most trivial way to transform an individual Batch into Frames.

This example walks through transforming a Batch into Frames.

Effectively, this example demonstrates the encoding process from an L2 batch into the serialized bytes that are posted to the data availability layer.

Walkthrough

The high level transformation is the following.

Batch -> decompressed batch data -> ChannelOut -> frames[] -> bytes[]

Given the Batch, the first step to encode the batch using the Batch::encode() method. The output bytes need to then be compressed prior to adding them to the ChannelOut.

note

The ChannelOut type also provides a method for adding the Batch itself, handling encoding and compression, but this method is not available yet.

Once compressed using the compress_brotli method, the compressed bytes can be added to a newly constructed ChannelOut. As long as the ChannelOut has ready_bytes(), Frames can be constructed using the ChannelOut::output_frame() method, specifying the maximum frame size.

Once Frames are returned from the ChannelOut, they can be Frame::encode into raw, serialized data ready to be batch-submitted to the data-availability layer.

Running this example:

  • Clone the examples repository: git clone git@github.com:op-rs/maili.git
  • Run: cargo run --example batch_to_frames
//! An example encoding and decoding a [SingleBatch].
//!
//! This example demonstrates EIP-2718 encoding a [SingleBatch]
//! through a [ChannelOut] and into individual [Frame]s.
//!
//! Notice, the raw batch is first _encoded_.
//! Once encoded, it is compressed into raw data that the channel is constructed with.
//!
//! The [ChannelOut] then outputs frames individually using the maximum frame size,
//! in this case hardcoded to 100, to construct the frames.
//!
//! Finally, once [Frame]s are built from the [ChannelOut], they are encoded and ready
//! to be batch-submitted to the data availability layer.

#[cfg(feature = "std")]
fn main() {
    use alloy_primitives::BlockHash;
    use maili_protocol::{
        Batch, ChannelId, ChannelOut, CompressionAlgo, SingleBatch, VariantCompressor,
    };
    use op_alloy_genesis::RollupConfig;

    // Use the example transaction
    let transactions = example_transactions();

    // Construct a basic `SingleBatch`
    let parent_hash = BlockHash::ZERO;
    let epoch_num = 1;
    let epoch_hash = BlockHash::ZERO;
    let timestamp = 1;
    let single_batch = SingleBatch { parent_hash, epoch_num, epoch_hash, timestamp, transactions };
    let batch = Batch::Single(single_batch);

    // Create a new channel.
    let id = ChannelId::default();
    let config = RollupConfig::default();
    let compressor: VariantCompressor = CompressionAlgo::Brotli10.into();
    let mut channel_out = ChannelOut::new(id, &config, compressor);

    // Add the compressed batch to the `ChannelOut`.
    channel_out.add_batch(batch).unwrap();

    // Output frames
    while channel_out.ready_bytes() > 0 {
        let frame = channel_out.output_frame(100).expect("outputs frame");
        println!("Frame: {}", alloy_primitives::hex::encode(frame.encode()));
        if channel_out.ready_bytes() <= 100 {
            channel_out.close();
        }
    }

    assert!(channel_out.closed);
    println!("Successfully encoded Batch to frames");
}

#[cfg(feature = "std")]
fn example_transactions() -> Vec<alloy_primitives::Bytes> {
    use alloy_consensus::{SignableTransaction, TxEip1559};
    use alloy_eips::eip2718::{Decodable2718, Encodable2718};
    use alloy_primitives::{Address, PrimitiveSignature, U256};
    use op_alloy_consensus::OpTxEnvelope;

    let mut transactions = Vec::new();

    // First Transaction in the batch.
    let tx = TxEip1559 {
        chain_id: 10u64,
        nonce: 2,
        max_fee_per_gas: 3,
        max_priority_fee_per_gas: 4,
        gas_limit: 5,
        to: Address::left_padding_from(&[6]).into(),
        value: U256::from(7_u64),
        input: vec![8].into(),
        access_list: Default::default(),
    };
    let sig = PrimitiveSignature::test_signature();
    let tx_signed = tx.into_signed(sig);
    let envelope: OpTxEnvelope = tx_signed.into();
    let encoded = envelope.encoded_2718();
    transactions.push(encoded.clone().into());
    let mut slice = encoded.as_slice();
    let decoded = OpTxEnvelope::decode_2718(&mut slice).unwrap();
    assert!(matches!(decoded, OpTxEnvelope::Eip1559(_)));

    // Second transaction in the batch.
    let tx = TxEip1559 {
        chain_id: 10u64,
        nonce: 2,
        max_fee_per_gas: 3,
        max_priority_fee_per_gas: 4,
        gas_limit: 5,
        to: Address::left_padding_from(&[7]).into(),
        value: U256::from(7_u64),
        input: vec![8].into(),
        access_list: Default::default(),
    };
    let sig = PrimitiveSignature::test_signature();
    let tx_signed = tx.into_signed(sig);
    let envelope: OpTxEnvelope = tx_signed.into();
    let encoded = envelope.encoded_2718();
    transactions.push(encoded.clone().into());
    let mut slice = encoded.as_slice();
    let decoded = OpTxEnvelope::decode_2718(&mut slice).unwrap();
    assert!(matches!(decoded, OpTxEnvelope::Eip1559(_)));

    transactions
}

#[cfg(not(feature = "std"))]
fn main() {
    /* not implemented for no_std */
}

Contributing

Thank you for wanting to contribute! Before contributing to this repository, please read through this document and discuss the change you wish to make via issue.

Dependencies

Before working with this repository locally, you'll need to install a few dependencies:

Optional

Pull Request Process

  1. Create an issue for any significant changes. Trivial changes may skip this step.
  2. Once the change is implemented, ensure that all checks are passing before creating a PR. The full CI pipeline can be run locally via the Justfiles in the repository.
  3. Be sure to update any documentation that has gone stale as a result of the change, in the README files, the book, and in rustdoc comments.
  4. Once your PR is approved by a maintainer, you may merge your pull request yourself if you have permissions to do so. Otherwise, the maintainer who approves your pull request will add it to the merge queue.

Working with OP Stack Specs

The OP Stack is a set of standardized open-source specifications that powers Optimism, developed by the Optimism Collective.

maili is a rust implementation of distinguished OP Stack types, transports, middleware and more. Not all types and implementation details in maili are present in the OP Stack specs, and on the flipside, not all specifications are implemented by maili. That said, maili is entirely based off of the specs, and new functionality or core modifications to maili must be reflected in the specs.

As such, the first step for introducing changes to the OP Stack is to open a pr in the specs repository. These changes should target a protocol upgrade so that all implementations of the OP Stack are able to synchronize and implement the changes.

Once changes are merged in the OP Stack specs repo, they may be added to maili in a backwards-compatible way such that pre-upgrade functionality persists. The primary way to enable backwards-compatibility is by using timestamp-based activation for protocol upgrades.

Licensing

maili is dually licensed under the Apache 2.0 and the MIT license.

Glossary

This document contains definitions for terms used throughout the maili book.