⚡ Zig Guide LiveUnofficialbut fully verified
✓ Zig 0.17.0-dev.1503+1f1bee62eOn an older Zig?

Three Ways to Frame

The previous chapter said a stream has no message boundaries. Framing is how you put them back. There are three ways, every protocol you have used picks one of them, and the choice is visible in the hex below.

const std = @import("std");

// ---------------------------------------------------------------- length

/// Two bytes of length, then exactly that many bytes. The receiver knows
/// how much to wait for before it has read a single byte of payload.
fn writeLengthPrefixed(w: *std.Io.Writer, payload: []const u8) !void {
    var header: [2]u8 = undefined;
    std.mem.writeInt(u16, &header, @intCast(payload.len), .big);
    try w.writeAll(&header);
    try w.writeAll(payload);
}

fn readLengthPrefixed(r: *std.Io.Reader) ![]const u8 {
    const header = try r.take(2);
    const len = std.mem.readInt(u16, header[0..2], .big);
    return r.take(len);
}

// ------------------------------------------------------------- delimiter

/// A byte that cannot appear in the payload ends the message. Cheap to
/// write, cheap to read by hand, and it costs you that byte forever.
fn writeDelimited(w: *std.Io.Writer, payload: []const u8) !void {
    try w.writeAll(payload);
    try w.writeByte('\n');
}

// ------------------------------------------------------- self-describing

/// The payload says how to read itself. Here a single tag byte selects
/// between a fixed-width integer and a length-prefixed string, which is
/// how RESP, protobuf and every tagged format work.
const Value = union(enum) {
    int: i32,
    text: []const u8,
};

fn writeTagged(w: *std.Io.Writer, value: Value) !void {
    switch (value) {
        .int => |n| {
            try w.writeByte('i');
            var b: [4]u8 = undefined;
            std.mem.writeInt(i32, &b, n, .big);
            try w.writeAll(&b);
        },
        .text => |s| {
            try w.writeByte('s');
            try writeLengthPrefixed(w, s);
        },
    }
}

fn printValue(w: *std.Io.Writer, value: Value) !void {
    switch (value) {
        .int => |n| try w.print("int {d}\n", .{n}),
        .text => |s| try w.print("text \"{s}\"\n", .{s}),
    }
}

fn readTagged(r: *std.Io.Reader) !Value {
    const tag = try r.takeByte();
    return switch (tag) {
        'i' => .{ .int = std.mem.readInt(i32, (try r.take(4))[0..4], .big) },
        's' => .{ .text = try readLengthPrefixed(r) },
        else => error.BadTag,
    };
}

pub fn main(init: std.process.Init) !void {
    var buf: [512]u8 = undefined;
    var file_writer = std.Io.File.stdout().writerStreaming(init.io, &buf);
    const out = &file_writer.interface;

    var wire: [64]u8 = undefined;

    // Length prefix. Note the payload is untouched: it may contain any
    // byte at all, including the newline the next scheme cannot carry.
    {
        var w: std.Io.Writer = .fixed(&wire);
        try writeLengthPrefixed(&w, "PING");
        try out.print("length:    {x}\n", .{w.buffered()});

        var r: std.Io.Reader = .fixed(w.buffered());
        try out.print("           read back \"{s}\"\n\n", .{try readLengthPrefixed(&r)});
    }

    // Delimiter. Readable on the wire, which is why HTTP and SMTP use it.
    {
        var w: std.Io.Writer = .fixed(&wire);
        try writeDelimited(&w, "PING");
        try out.print("delimiter: {x}\n", .{w.buffered()});

        var r: std.Io.Reader = .fixed(w.buffered());
        try out.print("           read back \"{s}\"\n\n", .{try r.takeDelimiterExclusive('\n')});
    }

    // Self-describing. Costs a tag byte and buys you a payload whose type
    // the receiver did not have to agree on in advance.
    {
        var w: std.Io.Writer = .fixed(&wire);
        try writeTagged(&w, .{ .int = 4096 });
        try writeTagged(&w, .{ .text = "PING" });
        try out.print("tagged:    {x}\n", .{w.buffered()});

        var r: std.Io.Reader = .fixed(w.buffered());
        try out.writeAll("           read back ");
        try printValue(out, try readTagged(&r));
        try out.writeAll("           read back ");
        try printValue(out, try readTagged(&r));
        try out.writeByte('\n');
    }

    // What the delimiter scheme cannot do. The payload contains the
    // delimiter, so the reader stops early and silently: no error, just a
    // truncated message and the rest treated as the next one.
    {
        var w: std.Io.Writer = .fixed(&wire);
        try writeDelimited(&w, "two\nlines");

        var r: std.Io.Reader = .fixed(w.buffered());
        const first = try r.takeDelimiterExclusive('\n');
        try out.print("delimiter with a '\\n' in the payload: \"{s}\"", .{first});
        try out.print(" ({d} of {d} bytes)\n", .{ first.len, "two\nlines".len });
    }

    try out.flush();
}

Length prefix

00 04 50 49 4e 47
^^^^^ length      ^^^^^^^^^^^ payload

The receiver knows exactly how many bytes to wait for before it reads a single byte of payload. That is what makes the parser in the last chapter possible, and it is why this is the default choice for anything binary.

The payload is untouched, so it may contain any byte at all, including the newline the next scheme cannot carry. The cost is that you must know the length before you start writing, which means either buffering the whole message or knowing its size in advance.

Delimiter

50 49 4e 47 0a
^^^^^^^^^^^ payload  ^^ '\n'

Cheap to write, cheap to read, and readable on the wire, which is why HTTP, SMTP, IMAP and Redis all use it for their text parts. You can debug it with nc and your eyes.

The cost is permanent: that byte can never appear in a payload. The last line of the output shows what happens when it does. The reader stops early, returns "two" instead of "two\nlines", and reports no error at all, because from its point of view the message ended exactly where a message is supposed to end. The rest gets treated as the next message.

Real protocols solve this with escaping or with a length prefix on the body only, which is exactly what HTTP does: delimited headers, then Content-Length and a raw body.

Self-describing

69 00 00 10 00   73 00 04 50 49 4e 47
^^ 'i'           ^^ 's'

A tag byte says how to read what follows. Here i means a fixed-width integer and s means a length-prefixed string, so the receiver does not have to have agreed in advance on what this field holds.

That is the shape of RESP, of protobuf wire types, of MessagePack and of CBOR. The cost is a byte per value and a switch that has to reject unknown tags rather than trusting them.

Choosing

Both of the others are special cases of a length prefix: a delimiter is a length you have to go looking for, and a tag is a length whose interpretation comes with it. If you are designing something new and have no reason to prefer readability, use a length prefix and stop thinking about it.

If you want it debuggable by hand, use a delimiter for the envelope and a length for the body. That combination has survived thirty years of HTTP.