arrowout

Overview ¶

Package arrowout exports an ursus batch as Arrow, which is how anything outside this module reads ursus's output without a copy.

Why this is mostly free ¶

ursus already stores Arrow's memory layout. data.Column holds *memory.Buffer allocated through arrowx, validity is an LSB-numbered bitmap, strings are int32 offsets plus a character buffer, lists are offsets plus a child. A data.Column IS an Arrow array with a different Go type wrapped around the same bytes, so the conversion is a re-wrap and the bytes never move.

Except at two seams, and both would be silent ¶

• Decimal and Int128. i128.Int128 is {Hi int64, Lo uint64}; arrow's decimal128.Num is {lo uint64, hi int64}. The widths match, the types look compatible, and a zero-copy wrap would swap the high and low 64 bits of every value — structurally valid and wrong by about 2^64. Every integer Sum outputs Int128, so this is the common path, not a corner. The words are swapped here.

• Time at second or millisecond resolution. ursus stores every Time as int64 ticks; Arrow puts those two units on Time32 and only microseconds and nanoseconds on Time64. So they narrow. Step 57's wrap is what makes the narrowing total: a time of day is below 86_400_000 milliseconds, which fits int32 with room to spare, so this cannot be the silently-wrapped int32 that step 49 removed from rescaleTemporal and step 57 removed from the Parquet writer.

The exported buffers are refcount-inert, deliberately ¶

arrow.Buffer.Release() at a zero refcount does `b.buf, b.length = nil, 0` — it nils the slice header INSIDE the shared Buffer. So a Buffer handed to a caller who releases it twice stops having data, and anything else pointing at that same Buffer stops having data with it. Silently: no panic, just an empty column.

Every buffer here is therefore re-wrapped with memory.NewBufferBytes, which produces a Buffer with no allocator and no parent — and Release on one of those is unconditionally a no-op. The data is still shared, because NewBufferBytes does not copy; only the refcounting is severed. The exported record's lifetime is then completely independent of the frame's, in both directions, which is the model arrowx states for ursus's own memory: "ursus hides Arrow's Retain/Release refcounting completely."

What that guard is worth TODAY, stated exactly ¶

For the payload it is belt-and-braces, and saying otherwise would be the prose claim this project counts as its largest defect class. data.Column keeps its *memory.Buffer fields unexported and offers only RawFixed/RawOffsets/RawChars, which return []byte — so the payload path cannot hand out a live Buffer whether or not this package wraps. The guard matters the day someone adds the obvious accessor returning c.fixed directly, which is a change this file would otherwise not notice.

The one path that yields a live Buffer at all is validity: bitmap.View.Buffer() allocates when the bitmap carries a non-zero bit offset, which is what a non-byte-aligned Column.Slice produces. Even there no ordinary sequence empties it — Record.Release is itself refcounted, so releasing a record twice does not release its buffers twice, and this package keeps a reference that stops the count reaching zero.

So the wrapping prevents nothing that is reachable today. It is kept because it makes the contract a SENTENCE rather than a case analysis — "Release on an exported buffer is a no-op", true unconditionally and safe to promise in the public doc — and TestReleaseIsInert asserts that property directly rather than through a scenario that cannot happen.