New framework accelerates RL rollouts by 50%

Together AI has unveiled a novel framework called distribution-aware speculative decoding (DAS) that addresses the rollout bottleneck in reinforcement learning (RL) post-training. This framework can achieve up to a 50% reduction in rollout time while maintaining the quality of model outputs. DAS employs an adaptive suffix tree drafter and a length-aware scheduling strategy to optimize GPU utilization and reduce latency caused by long-tail rollouts. This advancement is particularly significant as the AI community continues to develop larger models, making the rollout phase increasingly critical in training efficiency.