Model training on Spot, checkpoint-resumed.

The team had assumed Spot was unusable for training because a single interruption mid-run would lose 30 hours of progress. That had been true with their original training code, which kept all state in instance memory.

We refactored the training loop to use PyTorch Lightning’s built-in checkpointing, with checkpoints written to S3 every 30 minutes. SageMaker’s managed Spot mode handled the interruption-and-resume automatically; checkpoints replayed from the most recent S3 snapshot. Step Functions orchestrated the multi-stage training pipeline.

Interruptions averaged about one per training run, with a 6-12 minute recovery cost. The 12% wall-clock overhead was small compared to the 74% cost reduction. The team now experiments more freely — three concurrent training runs cost less than one used to.