## Overview This case study comes from a conference presentation by a member of MosaicML (now acquired by Databricks), sharing lessons learned from training their open-source MPT (MosaicML Pretrained Transformer) models. The presenter leads the ML Runtime team responsible for developing the core training infrastructure. The talk provides a practitioner's perspective on the challenges and best practices of training large language models at scale, with specific focus on operational aspects that are critical for production LLM development. MosaicML's motivation for training MPT was threefold: to provide the community with high-quality open-source models with commercially permissive licenses, to demonstrate different LLM capabilities (chat, long context, code generation), and to battle-test their training platform. The case study reveals that they successfully trained MPT-7B for approximately $250,000 on pre-training alone (plus a few thousand dollars for fine-tuning), significantly lower than the $500K-$1M+ that most practitioners expected based on an informal social media poll they conducted. ## Business Context and Value Proposition The presentation opens by framing why organizations would want to train their own LLMs rather than using existing APIs or open-source models. Four key drivers are identified: **Privacy**: Organizations with sensitive business data may not want to expose even prompts to third-party APIs. The presenter notes well-documented breach cases that reinforce this concern. **Quality**: Sometimes the required capability simply doesn't exist in available models. The example given is specialized technical documentation—like a user manual for the Large Hadron Collider—where general-purpose models like ChatGPT would be inadequate. **Cost**: Training a smaller, specialized model can be more economical than paying inference costs for large third-party models. A 3B parameter model hosted at a fraction of the cost may serve the use case adequately. **Latency**: Production applications often have strict latency requirements and need to handle high volumes of users, which may be incompatible with third-party API limitations. The presentation references four customer case studies to illustrate these points: Metadialogue (Arabic chatbot), Replit (code generation, starting from a 3B MPT model), Patronus (PII detection for data leak prevention), and Hyperwrite (writing assistant with auto-translation capabilities). ## Model Architecture and Training Approaches The MPT models are decoder-only transformer models following the GPT architecture. The presentation explains the autoregressive generation process where the model predicts the next token, which is then appended to the input for the next prediction cycle, continuing until a stop condition is reached. The training pipeline offers several paths: - **Pre-training from scratch**: Self-supervised learning on large token counts to build general capabilities and knowledge - **Continued pre-training**: Taking an existing model and further training on domain-specific tokens - **Instruction fine-tuning**: Supervised training to teach specific behaviors and response patterns - **RLHF (Reinforcement Learning from Human Feedback)**: Using preference data to improve response quality For MPT, MosaicML chose to start from scratch with their own optimized architecture, using randomized weights. They pre-trained and fine-tuned the model but did not apply RLHF. The result was two primary variants: MPT-7B and MPT-30B, with additional specialized versions including 8K and 64K context length models, and a chat-tuned version using the Alpaca dataset (though this version is not commercially licensable due to the ChatGPT-generated training data). ## Planning and Evaluation The presentation emphasizes that rigorous planning is essential before committing millions of dollars to training runs. Key planning activities include: **Defining the problem clearly**: The presenter stresses questioning whether an LLM is even necessary. If a smaller model or alternative approach can solve the problem, that's preferable given the complexity of LLM training. **Establishing evaluation criteria**: Using the example of building a coding assistant, relevant evaluations would include HumanEval (a code generation benchmark), in-context learning evaluations, and custom evaluations for specific capabilities. The presenter notes that evaluation is a topic worthy of an entire conference on its own. **Vibe checks**: Beyond formal metrics, MosaicML uses conversational testing with fun prompts like "If you were an AI model, what would you name yourself?" and the "banana bread test" where they actually cook recipes generated by the model and evaluate the results. **Performance KPIs**: Understanding cost and latency requirements in the context of the specific problem being solved. ## Compute Budgeting with Scaling Laws A significant portion of the presentation covers compute budgeting using Chinchilla scaling laws. These laws provide equations to estimate the amount of data needed to reach a target quality level at minimum training cost. For example, a 30B parameter model would need approximately 600 billion tokens using the Chinchilla-optimal approach. However, the presenter offers an important caveat: Chinchilla scaling laws only optimize for training compute, not inference. A 175B parameter model might achieve similar loss to a 30B model with fewer tokens, but hosting the larger model is significantly more expensive. MosaicML learned this lesson the hard way initially, training expensive-to-host models before considering inference costs. The practical recommendation is often to train smaller models on considerably more data than Chinchilla-optimal, achieving better accuracy at lower total cost of ownership when considering both training and inference. For concrete estimates, training a 30B parameter model on 600 billion tokens requires approximately 3,000 H100-days and costs around $500,000 at $7 per H100-hour. The presenter notes this is within 10-15% accuracy and doesn't include the smaller experimental runs that precede the main training run. ## Data Preparation as a Critical Bottleneck The presentation emphasizes that data preparation often takes more time than actual training. Key data processing activities include: **Data cleaning**: Removing duplicates, filtering toxicity, stripping PII, fixing encoding issues. Training on poorly encoded data wastes compute and can corrupt the tokenizer. **Deduplication**: Internet crawl data contains massive redundancy. Removing duplicates improves model quality and reduces token count, directly reducing training cost. **Pre-tokenization and concatenation**: Rather than tokenizing on-the-fly during training (which wastes compute), MosaicML pre-tokenizes all datasets. Additionally, they concatenate short documents to fill the context window rather than padding, ensuring every compute cycle contributes to learning. **Shuffling**: The presenter cannot emphasize enough the importance of shuffle quality at scale. Each training batch should be a representative sample of the full dataset distribution. Poor shuffling leads to what they call "wavy boys"—noisy, spiky loss curves that indicate suboptimal training. An important operational lesson shared is that MosaicML initially underinvested in data infrastructure. They started with two beefy machines running custom distributed Python code, taking days or weeks for basic tasks. After the Databricks acquisition, leveraging Apache Spark reduced these operations to hours. The recommendation is to invest in scalable data exploration and ETL architecture from the start. ## Distributed Training Infrastructure Scaling from one GPU to thousands requires careful infrastructure design. The math is straightforward: 3,000 GPU-days becomes 6 days on 512 GPUs or 3 days on 1,024 GPUs. However, achieving this requires strong linear scaling. MosaicML uses FSDP (Fully Sharded Data Parallel), a relatively simple parallelism strategy built into PyTorch and pioneered by Microsoft with DeepSpeed. This scales well with fast interconnects and avoids the complexity of more exotic parallelism approaches. The platform they built, called the Mosaic ML platform, provides: - Integration with experiment trackers (Weights & Biases, MLflow) - LM Foundry with pre-optimized, numerically stable model implementations - Streaming dataset library for fast data loading from object storage without expensive network-attached storage - Automatic model checkpointing and resumption for hardware failures - Orchestration and scheduling that makes scaling from one GPU to thousands a single configuration change ## Fault Tolerance at Scale Hardware failures are inevitable at scale. The rule of thumb shared is approximately one failure per 1,000 GPU-days. These failures are typically disruptive—jobs die completely. Common causes are CUDA errors and hardware issues (bad GPUs, network interconnect problems). The presenter contrasts their MPT-7B training log (clean, with simple entries for hardware failure, checkpoint, resume, continue) with the publicly available OPT training log (which documents significant operational challenges). Their system handles failures with automatic checkpointing and resumption, typically losing only 20-30 minutes of progress. The recommendation is to either choose a platform that handles these challenges or assemble best-of-breed components yourself. The key capabilities needed are: compatible software stacks (correct PyTorch/CUDA versions), scheduling and orchestration, model checkpointing, and comprehensive logging. ## Experimentation Philosophy A recurring theme is the importance of experimentation and skepticism. The presenter explicitly says "don't trust anyone"—including conference speakers, published literature, and your own intuition. Things that work in one context may not work in another due to different data distributions or model architectures. The practical advice is: - Don't bank multi-million dollar training runs on untested assumptions - Use scaling laws to test hypotheses at smaller scales (they extrapolate reasonably well) - Build stop points into training runs to verify the model is performing as expected - Iterate quickly—this isn't traditional software engineering with bounded scope - Start small (1B parameters), solve a problem, then scale up as needed - Try RAG, prompt engineering, and other techniques before committing to training ## Process and Discipline The final lesson draws on the presenter's background in chip design, where tape-outs cost tens of millions of dollars and require rigorous risk management. Similarly, expensive LLM training runs require: **Pre-flight checklists**: Define everything that must be verified before launching a $500K-$5M training run. The presenter mentions they may publish a template blog post. **Designated on-call**: Technology fails, and someone needs to be available to pull in the right people when it does. **Training logs**: Maintain detailed logs and learn from them. If you're going to repeat this process, build the institutional knowledge to do it efficiently. ## Results and Impact The MPT-7B model held its own against Falcon and Llama v1 at the time of release. MPT-30B performed comparably or better on certain dimensions than 40B parameter models like Falcon. The presenter acknowledges that better models have since been released (Mistral, Llama v2), but this speaks to the rapid pace of the field rather than any shortcoming in their approach. The broader impact is demonstrating that high-quality LLM training is accessible to organizations beyond the largest tech companies, provided they invest appropriately in infrastructure, data preparation, and operational processes.