Skip to content (access key 's')
Logo of Technion
Logo of CS Department
Events

The Taub Faculty of Computer Science Events and Talks

Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency
event speaker icon
Itai Elam (M.Sc. Thesis Seminar)
event date icon
Wednesday, 12.08.2026, 12:30
event location icon
Taub 601 & Zoom
event speaker icon
Advisor: Prof. Avi Mendelson

Pipeline parallelism is a key technique for training large language models in distributed settings, but existing schedules face a fundamental trade off between hardware utilization and optimization consistency. Synchronous methods preserve correct optimization but suffer from pipeline bubbles, while asynchronous methods eliminate these bubbles at the cost of stale model weights and additional mechanisms such as weight stashing or prediction. In this work, we present PACI (Pipeline Asynchronous training with Controlled Inconsistency), an asynchronous pipeline training algorithm that achieves bubble free execution while maintaining bounded weight inconsistency without weight stashing, weight prediction, additional parameter copies, or global synchronization. Evaluated on GPT-2 pretraining, PACI matches the convergence, model quality, and memory footprint of a synchronous schedule while reducing training time-to-accuracy by up to 1.69x.