We Measured LLM Serialization Across Five Tokenizers
Six data formats, five tokenizer vocabularies, 315 measurements with provider token counts as ground truth. Plain TSV beat the token-oriented formats, our own dictionary hypothesis failed, and base64 cost up to 4.6x. The rig and raw data are open.
Read the study