⚡ Zig Guide LiveUnofficialbut fully verified
✓ Zig 0.17.0-dev.2375+d8aab4878What's newOn an older Zig?

Slugs Are Arithmetic

Updated · Code verified against Zig 0.17.0-dev.2375+d8aab4878

A URL shortener’s slug can be the database row id written in base62, which makes every slug unique with no collision check. The URL it points to needs checking before it is stored, because a shortener is an open redirect by design.

The problem

The service needs a short name for every stored URL. The tempting design is a random string, checked against the database for collisions and retried. A simpler design uses what the database already has. It hands every row a unique integer, the bigserial id. A slug that is just that integer written compactly is unique because the ids are.

Compact means a bigger base. Base 10 writes a billion as ten characters; base 62, using the digits and both alphabets, writes it as six.

The plan

  1. encode: divide the id by 62 repeatedly, emitting one character per remainder. The one-billionth link is still six characters.
  2. decode: the reverse, for the redirect path, so the lookup can use the primary key. Reject anything outside the alphabet before it reaches a query.
  3. checkTarget: what the service will agree to redirect to. This is security logic.
const std = @import("std");

// 62 characters that pass through a URL unchanged. They need no escaping,
// include no punctuation that looks like another character, and are not
// changed by copy and paste.
const alphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";

/// The row id, rewritten in base 62. The database already guarantees ids
/// are unique, so the slugs are too. The code needs no random draw,
/// collision check or retry loop. An id fits in 11 characters even at the
/// u64 limit.
fn encode(id: u64, buf: *[11]u8) []const u8 {
    var rest = id;
    var i: usize = buf.len;
    while (true) {
        i -= 1;
        buf[i] = alphabet[@intCast(rest % 62)];
        rest /= 62;
        if (rest == 0) break;
    }
    return buf[i..];
}

/// The other direction, for the redirect handler: slug back to id, so
/// the database lookup can use the primary key. Anything outside the
/// alphabet is rejected here, before it gets near a query.
fn decode(slug: []const u8) !u64 {
    if (slug.len == 0 or slug.len > 11) return error.BadSlug;
    var id: u64 = 0;
    for (slug) |ch| {
        const digit = std.mem.findScalar(u8, alphabet, ch) orelse
            return error.BadSlug;
        id = try std.math.add(u64, try std.math.mul(u64, id, 62), digit);
    }
    return id;
}

/// What the service will shorten. The scheme check is required: a
/// redirect service that stores `javascript:` URLs sends every visitor
/// to code written by whoever submitted one.
fn checkTarget(url: []const u8) !void {
    const has_scheme = std.mem.startsWith(u8, url, "https://") or
        std.mem.startsWith(u8, url, "http://");
    if (!has_scheme) return error.SchemeNotAllowed;
    for (url) |ch| {
        // Control characters would let a stored URL smuggle bytes into
        // the HTTP response that redirects to it. A space makes the URL
        // invalid.
        if (ch <= ' ' or ch == 0x7f) return error.BadCharacter;
    }
}

pub fn main(init: std.process.Init) !void {
    var out_buf: [2048]u8 = undefined;
    var file_writer = std.Io.File.stdout().writerStreaming(init.io, &out_buf);
    const out = &file_writer.interface;

    var buf: [11]u8 = undefined;

    // The mapping, at the sizes a shortener actually sees.
    try out.writeAll("-- encode --\n");
    for ([_]u64{ 0, 7, 61, 62, 100_000, 1_000_000_000, std.math.maxInt(u64) }) |id| {
        try out.print("id {d} -> \"{s}\"\n", .{ id, encode(id, &buf) });
    }

    // The redirect route depends on reversibility, so the test checks it
    // instead of assuming it.
    for ([_]u64{ 0, 1, 61, 62, 3843, 1 << 20, 1 << 40, std.math.maxInt(u64) }) |sample| {
        if (try decode(encode(sample, &buf)) != sample) return error.RoundtripBroken;
    }
    try out.writeAll("\nroundtrip: decode(encode(id)) == id held for every size tried\n");

    try out.writeAll("\n-- decode rejects --\n");
    for ([_][]const u8{ "zig_1", "", "zzzzzzzzzzzz" }) |slug| {
        _ = decode(slug) catch |err| {
            try out.print("\"{s}\" -> {t}\n", .{ slug, err });
        };
    }

    try out.writeAll("\n-- target validation --\n");
    const targets = [_][]const u8{
        "https://ziglang.org/download/",
        "http://old-but-fine.example",
        "javascript:alert(1)",
        "https://zig lang.org",
        "ziglang.org",
    };
    for (targets) |t| {
        if (checkTarget(t)) {
            try out.print("ok      {s}\n", .{t});
        } else |err| {
            try out.print("{t}: {s}\n", .{ err, t });
        }
    }

    try out.flush();
}

Why the scheme check is not pedantry

A shortener sends visitors wherever its stored URLs point. The check on what it agrees to store is the only thing that stops attackers from abusing that. A stored javascript: URL turns the redirect into script execution in the visitor’s browser. Control characters could let a stored URL smuggle extra bytes into the HTTP response that carries the redirect. So the check allows exactly two schemes and refuses the space, everything below it, and DEL. The routes chapter runs it on create and update both, before anything touches storage.

What the id leaks

Deriving slugs from the sequence has one cost: slugs are guessable. 4c92 tells anyone that 4c91 probably exists, and roughly how many links the service has issued. This project accepts that because it is a demonstration, built to show the mechanism. A service whose links must be unguessable, or that should not leak its own size, encodes a permutation of the id instead of the id itself. That keeps the slugs unique and scrambles the order. The change would stay inside these two functions.

Variations

  • Sixty-two, not sixty-four: base64 needs + and / or their URL-safe stand-ins. Base62 spends two characters to produce slugs that survive any copy-paste untouched.
  • Case matters: q0U and Q0u are different links. A shortener whose links people read aloud over a phone drops the uppercase half, and its slugs become a third longer.