Module expert_pack

Module expert_pack 

Source
Expand description

2026-09-25: The expert-store format. pack_record and unpack_record build and parse one record in memory, with no I/O. ExpertIndex is the store’s manifest.json, from which a reader rebuilds the record spec and layout. The layer files are written and read in expert_pack_fs.rs.

Owner: metrale-storage experts. Invariants:

  • ExpertIndex::load and ExpertFileReader::open reject a manifest whose version is not ExpertRecordHeader::VERSION.

Structs§

ExpertFileReader
2026-09-25: Reads records back with positional reads (tier::pio). expert_tier.rs and expert_peer.rs open stores through it.
ExpertFileWriter
2026-09-25: Creates one file per MoE layer, sized to bytes_per_layer, writes records at their offsets, and writes manifest.json in finish. The files are opened without O_DIRECT.
ExpertIndex
2026-09-25: The store manifest, manifest.json beside the per-layer files: the geometry a reader needs to rebuild the spec and layout and open the files.
ProjData
2026-09-25: One projection’s packed and scale bytes, as pack_record writes them.
ProjView
2026-09-25: One projection’s sub-buffers inside a parsed record.

Functions§

pack_record
2026-09-25: Build one stride-byte record: the header at offset 0, each projection’s bytes at the spec’s offsets, zeros elsewhere. Fails when stride is below spec.raw_bytes() or a projection’s lengths differ from the spec.
unpack_record
2026-09-25: Parse a record of at least spec.raw_bytes() bytes into its header and projection views. Fails on a short buffer or a header whose magic or version differs.