Unicode

const std = @import("std");
const expect = std.testing.expect;
const expectEqualStrings = std.testing.expectEqualStrings;

test "len counts bytes, not characters" {
    const s = "héllo"; // é is two bytes in UTF-8
    try expect(s.len == 6);
    try expect(try std.unicode.utf8CountCodepoints(s) == 5);
}

test "validate untrusted input first" {
    try expect(std.unicode.utf8ValidateSlice("héllo"));
    // 0xff never appears in well-formed UTF-8.
    try expect(!std.unicode.utf8ValidateSlice(&.{ 0xff, 0xfe }));
}

test "iterate codepoints" {
    // Utf8View.init validates once; iteration after that cannot fail.
    const view = try std.unicode.Utf8View.init("héllo");
    var it = view.iterator();

    try expect(it.nextCodepoint().? == 'h');
    try expect(it.nextCodepoint().? == 'é');
    try expect(it.nextCodepoint().? == 'l');

    // nextCodepointSlice returns the raw bytes instead of a u21,
    // useful when slicing the original string.
    var rest = view.iterator();
    _ = rest.nextCodepointSlice();
    const e_bytes = rest.nextCodepointSlice().?;
    try expect(e_bytes.len == 2);
}

test "encode a codepoint to bytes" {
    var buf: [4]u8 = undefined;
    const len = try std.unicode.utf8Encode('→', &buf);
    try expect(len == 3);
    try expectEqualStrings("→", buf[0..len]);
}

test "slicing bytes can split a character" {
    const s = "héllo";
    // s[0..2] cuts é in half: one valid byte, one dangling continuation.
    try expect(!std.unicode.utf8ValidateSlice(s[0..2]));
    // Iterate codepoint slices when you need safe cut points.
    try expect(std.unicode.utf8ValidateSlice(s[0..3]));
}

`len` is bytes

"héllo".len is 6, not 5, because é encodes to two bytes. Nothing in the language layer knows about UTF-8; source files are required to be valid UTF-8, but a []const u8 at runtime is just bytes. When you need character counts, std.unicode.utf8CountCodepoints walks the encoding.

Validate once, then iterate

Utf8View.init checks the whole slice up front and returns an error for malformed input. After that, its iterator cannot fail, so the loop body stays clean. nextCodepoint yields u21 values; nextCodepointSlice yields the raw bytes of each character, which is the right tool when you are slicing the original string at character boundaries.

For input you do not control (network data, file contents), run utf8ValidateSlice before treating bytes as text.

Slicing can cut a character in half

s[0..2] on "héllo" produces one valid byte followed by half of é. Byte indices from indexOf are always safe cut points when the needle itself is valid UTF-8, but arbitrary arithmetic on indices is not. If you truncate strings for display, iterate codepoint slices and stop at the last full character.

Unicode

len is bytes

Validate once, then iterate

Slicing can cut a character in half

`len` is bytes