Recipe: Pass Data Across the Boundary
A string cannot go between JavaScript and a Zig wasm module as a value. Thus the string goes through the linear memory of the module:
- The exported
allocfunction of the module returns an offset. - The host writes the bytes at that offset.
- The host calls into the module with the offset and a length.
The problem
The last recipe gave numbers to the module, and a number fits in a wasm value. A string does not fit. A slice, a struct and a buffer do not fit either. Linear memory is the only place where both sides can see them. The two sides must agree on the meaning of the bytes there.
The plan
- Export an
allocfunction, so that the host can reserve space in linear memory. - Export the memory. The reactor build does this automatically.
- The host writes bytes at the returned offset, then calls in with
(offset, len). The host reads a result back from memory at an offset that the module reports.
const std = @import("std");
// A small arena that gives scratch space to the host. A real module exports
// a general allocator. This arena keeps each step visible in the code.
var scratch: [4096]u8 = undefined;
var used: usize = 0;
export fn alloc(len: usize) [*]u8 {
const start = used;
used += len;
return scratch[start..].ptr;
}
export fn reset() void {
used = 0;
}
// Sum the bytes that the host put at ptr[0..len].
export fn sumBytes(ptr: [*]const u8, len: usize) u32 {
var total: u32 = 0;
for (ptr[0..len]) |b| total += b;
return total;
}
// Change to upper case in place. Thus the host reads the answer from the
// same offset.
export fn upperInPlace(ptr: [*]u8, len: usize) void {
for (ptr[0..len]) |*b| b.* = std.ascii.toUpper(b.*);
}
test "the round trip a JS host would drive" {
reset();
// Host: reserve space, and copy bytes into linear memory.
const msg = "zig";
const dst = alloc(msg.len);
@memcpy(dst[0..msg.len], msg);
// Host: call in with (offset, length).
try std.testing.expectEqual(@as(u32, 'z' + 'i' + 'g'), sumBytes(dst, msg.len));
// Host: change the bytes in place, then read them back.
upperInPlace(dst, msg.len);
try std.testing.expectEqualStrings("ZIG", dst[0..msg.len]);
}The test in the snippet is the script of the host, written in Zig. It does these steps:
- Reserve space.
- Copy bytes in.
- Call
sumBytes. - Change the bytes in place with
upperInPlace. - Read the bytes back.
A JavaScript host does the same steps through instance.exports and a
typed-array view. The browser recipe shows how to
build the module as a reactor and call it. This page covers the memory
protocol itself.
Why an offset and not a pointer
Inside a wasm module, a Zig [*]u8 is an offset into linear memory, counted
from zero. It is not an address in the process of the host. When alloc
returns [*]u8, JavaScript sees a plain integer. The integer tells where in
the shared array the bytes go.
The host never dereferences a wasm pointer directly. The host uses that number as an index into the memory buffer.
The JavaScript side
WebAssembly.Memory gives an ArrayBuffer, and you look at it through typed
arrays. The protocol only writes into the buffer and reads back from it:
const { instance } = await WebAssembly.instantiate(bytes, {});
const { alloc, reset, sumBytes, upperInPlace, memory } = instance.exports;
function pass(str) {
const utf8 = new TextEncoder().encode(str);
const ptr = alloc(utf8.length);
// A fresh view each time: growing memory detaches the old ArrayBuffer.
new Uint8Array(memory.buffer, ptr, utf8.length).set(utf8);
return { ptr, len: utf8.length };
}
reset();
const { ptr, len } = pass("zig");
console.log(sumBytes(ptr, len)); // 330 (z + i + g)
upperInPlace(ptr, len);
const out = new Uint8Array(memory.buffer, ptr, len);
console.log(new TextDecoder().decode(out)); // "ZIG"
The one trap
memory.buffer can change. Any allocation can grow the linear memory of the
module. When the memory grows, the old ArrayBuffer becomes detached, and
each typed array over it reads as empty.
Thus make the Uint8Array view immediately before you use it. Never keep the
view across a call that can allocate. For this reason, the pass helper above
makes a new view each time.
Who owns the memory
In this snippet, the module owns the memory. alloc gives out slices of a
fixed 4 KiB buffer, and reset takes back all of the buffer at one time. This
is a bump allocator.
A bump allocator is sufficient for the work of one request. In that work, the host fills a buffer, calls the module, and continues. For data that must stay longer, do one of these:
- Export a real allocator, with
free. - Let the host keep a record of the offsets that it got.
In both cases, the rule stays the same. The module manages its linear memory. The host only uses offsets that the module gave out.
Variations
- Return a string: the module cannot return a slice. Return an offset, and give the length through a second export. Or write both into a small struct in memory, and return the offset of that struct.
- Strings are UTF-8 on both ends: Zig source is UTF-8, and the
TextEncoderof JavaScript emits UTF-8. Thus bytes that you copy across need no new encoding. - Grow on demand:
allochere cannot fail in a useful way. A realalloccalls@wasmMemoryGrowwhen the buffer is full. The freestanding recipe shows this.