VocabX cuts SLM inference cost by 77% via specialized vocab

Hugging Face researchers have published findings from their VocabX paper, demonstrating that specializing a model's vocabulary can reduce inference costs by 77% in production environments. Using an automated Portuguese essay grader built on Qwen3-4B as a case study, the team replaced the general-purpose tokenizer with a domain-specific one that prioritizes cost efficiency over token frequency. The method merges repetitive output sequences, such as JSON structures, into single 'macro tokens' and reduces the vocabulary size from approximately 152,000 to 65,000 entries. This optimization targets the memory-bound nature of the decoding phase, where generating each token is significantly more expensive than reading input.