← 返回日报
略读 预计 1 分钟

std.Io.Writer.Allocating ate all my memory

摘要

文章通过一个打印数字的示例,指出 Zig 标准库 std.Io.Writer.Allocating 的 drain 实现存在内存过度分配问题。drain 支持 vectored I / O 和 splat 参数,但在 data.len==1 且 splat==1 的常见情况下,实现会为 pattern 重复预留内存,导致实际分配量是所需的两倍。作者用简化代码展示问题,并提到 Claude 帮助定位,最后建议简单场景改用 ArrayList 以避免此问题。

荐读理由

如果你在 Zig 项目中使用 Allocating Writer,需注意其 drain 实现会因 splat 参数重复预留内存,即使 splat=1 也会多分配一倍;改用 ArrayList 可规避。该案例也提醒检查其他语言标准库的向量写入实现是否存在类似过度分配陷阱。

原文

std.Io.Writer.Allocating ate all my memory

If I told you that the following prints 1665:

var b: std.ArrayList(u8) = try .initCapacity(init.gpa, 1024);
try b.appendSlice(init.gpa, "a" ** 1025);
std.debug.print("{d}\n", .{b.capacity});

Would you be able to guess what this prints?

var w: Io.Writer.Allocating = try .initCapacity(init.gpa, 1024);
try w.writer.writeAll("a" ** 1025);
std.debug.print("{d}\n", .{w.writer.buffer.len});

Like me, you might be surprised to see 3204. What's even weirder is that if you split the write, you get a more reasonable 1668:

var w: Io.Writer.Allocating = try .initCapacity(init.gpa, 1024);
try w.writer.writeAll("a" ** 1024);
try w.writer.writeAll("a");
std.debug.print("{d}\n", .{w.writer.buffer.len});

What's going on here? It appears to be a bug in the drain implementation of std.Io.Writer.Allocating. drain is the one method a Writer has to implement, and, besides self, it takes two parameters:

fn drain(w: *Writer, data: []const []const u8, splat: usize) Error!usiz

It takes a list of values to write (to support vectored I/O) and a "splat" count which is the number of times the last value in data should be written. splat is particularly useful, I believe, for compression. Here's a relevant line from zstd/Decompress.zig:

try w.splatByteAll(d.literal_streams.one[0], len);

Where writer.splatByteAll finds its way to calling drain. So we have some idea of drain's parameters, but why does it grow so much. Here's a simplified version of Allocating's drain function:

fn drain(self: *Allocating, data: []const []const u8, splat: usize) !usize {
  const pattern = data[data.len - 1];
  const splat_len = pattern.len * splat;
  const start_len = self.writer.end;

  for (data) |bytes| {
    try self.ensureUnusedCapacity(bytes.len + splat_len + 1);
    @memcpy(self.writer.buffer[self.writer.end..][0..bytes.len], bytes);
    self.writer.end += bytes.len;
  }

}

Can you spot the issue? I couldn't, but Claude could. For the common case where data.len == 1 and splat == 1, we're actually reserving 2x the memory: once for the data and once for splat_len which is the data again (what the code calls the pattern). If we called drain(&.{"hello", " "}, 100), the code would need to reserve space 5 bytes for "hello" and 100 bytes for the pattern (" ".len * 100). But the implementation reserves the splat space for every value, including the pattern itself.

ArrayList doesn't suffer from this: it has no splat. If your use-case is simple, if you're just appending bytes, you might want to stick with it.

Lobsters · 2 赞 · 0 评 讨论 → 阅读原文 →

这条对你有帮助吗?