Building a reliable cloud native foundation for distributed AI training

AI workloads are changing what platform teams need from infrastructure. Provisioning GPUs and standing up a cluster no longer makes a platform “AI-ready.” Once training spans more than one node, the bottlenecks show up in places application platforms rarely treat as first-class concerns: inter-node communication, shared storage, placement, topology, and validation.

Our internal ML platform supports training and inference workloads behind product experiences such as search and ranking. As these workloads grew, some training jobs outgrew the practical limits of a single machine. Models moved into the tens of billions of parameters, and a single node stopped being able to hold the model, its optimizer state, and a workable batch size at the same time. Atlassian therefore needed a platform that could make distributed training reliable and repeatable, without exposing ML teams to the underlying infrastructure complexity.

For distributed AI, performance is not just optimization. It is part of correctness.

The workload problem behind the infrastructure work

Adding GPUs was the easy part. The platform had to make three things predictable:

Communication performance. Workers need fast, low-overhead inter-node communication for synchronization traffic, including gradients, model state, and collective operations, so a distributed job keeps progressing together. The gap here is not marginal: on current-generation GPU hardware, a socket-based path can leave a multi-node job running at roughly half the speed the same hardware delivers over RDMA.

Shared data access. Workers need high-throughput shared storage to read training data, write checkpoints, and access intermediate artifacts concurrently without turning storage into a bottleneck or causing long pauses and uneven progress.

Operational predictability. Hardware placement, network topology, storage, and validation need to work together so a job does not silently run on a degraded path.

Two technologies cover the first two:

RDMA (Remote Direct Memory Access), which enables lower-overhead, high-throughput communication between GPU nodes.

Lustre, a parallel distributed filesystem designed for high-throughput shared access to training data and checkpoints.

ML teams should not have to manage either one. That is the platform’s job.

Why this matters beyond one company: None of this is specific to us. As AI adoption grows, platform teams keep hitting the same wall: distributed systems, accelerators, storage and scheduling have to behave as one platform, not four separate layers.

What happened before the high-performance network and storage design

Before RDMA and Lustre, distributed jobs ran. How fast they ran was anyone’s guess. Communication and storage delays surfaced as low GPU utilization, uneven step times, and runs that took far longer than they should have.

The worst of it was silent RDMA fallback. A misconfiguration sent collective traffic over sockets, and the job carried on at a fraction of the speed it should have reached.

That made the degradation easy to miss, and expensive to ignore.

This was not limited to one environment. On our other cloud, the device plugin that advertises the RDMA fabric to Kubernetes has been in CrashLoopBackOff on every fabric-capable production node from the day it was deployed. It was not a regression; it had never worked. We found it 271 days later, by accident, while verifying an unrelated GPU operator upgrade. The cause was mundane: a mirrored container image resolved to a single-architecture manifest that did not match the nodes, so the plugin failed at exec. A stopgap had been applied to staging months earlier, and production never received the follow-up.

Two failure modes matter here. A job that explicitly requests the fabric stays pending indefinitely, which is loud and easy to diagnose. A job that does not request it runs over sockets with no error at all, which is both the default and the expensive case.

Three lessons generalize from that outage:

A capability with no consumers is invisible, however expensive it was to buy. No alert existed for “this DaemonSet has never been ready”, and that absence was the real defect.

Pod age and restart counts mislead when nodes autoscale. Every new node presents a fresh crash loop, which reads as a recent break when the underlying condition is months old.

A healthy environment beside a broken one is a trap. Staging passing told us nothing about production, because staging had quietly diverged onto a different fix.

Both failures share a cause that has nothing to do with networking. Multi-node demand is still low next to single-node work. Few jobs touch the fabric, so nothing generates the signal that would show it is broken. That argues for synthetic validation rather than better dashboards: a dashboard shows what your jobs did, not what a fabric nobody used would have done.

Why a new network and storage design was needed

We treated this as a platform design problem rather than a tuning exercise.

We integrated RDMA-capable networking and shared high-throughput storage into the platform, absorbing topology and cloud-specific complexity so ML teams could run distributed training reliably without managing the underlying infrastructure.

Platform concern

Before

After

Inter-node GPU communication

Standard socket/TCP path could become a hidden bottleneck

RDMA-capable path for faster collective communication

Shared training storage

Less efficient path for concurrent checkpoint and dataset access

Lustre-based shared high-throughput storage

Operational confidence

Jobs could appear healthy while underperforming

Validation made transport and performance behavior visible

User experience

Risk of infrastructure details leaking to ML teams

Platform-managed capability with consistent workflow

How we achieved it

That meant changes at every layer:

RDMA-capable cluster and node-pool setup. Clusters are created with the multi-networking support the fabric requires, and GPU node pools are provisioned against a specific reservation and location rather than a generic pool.

GPU-specific network interfaces and network mappings. Each GPU node carries dedicated RDMA interfaces alongside its ordinary one, and each node pool is mapped to the RDMA network and subnet matching where its hardware physically sits.

Node-level dependencies, drivers, and readiness controls. The RDMA userspace libraries, the collective communication runtime, and the network plugin are installed on the node image. A node that fails its readiness check never becomes schedulable, so jobs cannot land on a partially configured node.

Lustre integration for shared storage. The filesystem is exposed through a CSI driver as an ordinary ReadWriteMany PersistentVolumeClaim, mounted at the same path in every worker pod, with per-namespace and per-job subdirectories providing isolation.

Placement and scheduling constraints for the right hardware and topology. Jobs are pinned to node pools with the correct hardware, drivers and fabric wiring, and gang scheduling ensures a distributed job either receives all of its workers or waits, rather than half-starting and holding GPUs idle.

Workload-level configuration and validation. The networking and storage plumbing is injected into the pod spec at admission time so ML teams never hand-write it, and the transport is confirmed before a job is treated as healthy.

The platform absorbs network, storage and topology complexity so that submitting a distributed training job stays ordinary.

One of the key lessons was that RDMA depended on physical topology, not just Kubernetes or workload configuration. GPU reservations could move across datacenters within the same zone, and the RDMA subnet mapping had to remain aligned with where the hardware actually lived. That meant the solution needed to support multiple RDMA network mappings and safe node-pool transitions instead of assuming one static configuration forever. How much of this you inherit depends on where you run. On one of our two clouds the managed fabric handles reservation and zone changes itself, and the platform team never sees them.

We also had to treat reservation changes as platform transitions: introduce a new node pool, move workloads safely, and then retire the old path. In practice that proved more reliable than trying to encode the whole problem as a one-time setup task.

What problems we faced along the way

None of this was a feature flag. We had to solve both the availability of suitable GPU nodes across two of the top hyperscalers and the challenge of introducing a network design that pushed beyond each provider’s usual patterns.

GPU node availability across clouds: securing enough compatible GPU capacity was a constraint in both clouds, affecting planning, placement, and the ability to move workloads between providers.

A new ask for cloud providers: this combination of GPU placement, high-performance networking, and shared storage was not a standard, one-size-fits-all request. There was no single approach that worked across providers, so the design had to adapt to each cloud’s capabilities and constraints.

Topology, reservation, and configuration timing: the correct network path depended on underlying physical placement, and some information needed to select the right RDMA mapping was not always available early enough. This is not universal, and the difference matters if you run in more than one place. On one of our two clouds the managed fabric absorbs reservation and zone changes, and none of it is visible to the platform team. On the other, the mapping is ours to maintain. Do not assume the behavior transfers.

Reservation topology is a provider constraint, not a platform choice: we have been allocated GPUs inside a single block, and we have been allocated them spread across several. Whether topology-aware scheduling can help once an allocation spans blocks is still an open question for us.

Shared storage carries its own operational cost: the parallel filesystem solved throughput, but onboarding a new region or cluster still means standing up a new filesystem instance by hand. We traded a throughput problem for an operational one.

Cross-layer, cross-team delivery: networking, node setup, storage, scheduling, and workload integration all had to line up, requiring close iteration across infrastructure, networking, and ML platform boundaries.

Important lesson: for distributed training, “the job ran” is not a sufficient success signal. You need validation that confirms it ran on the intended transport and storage path.

In practice our validation is coarser than we would like. There is no per-job telemetry that pins a slowdown on the transport. What we have is which storage path a job used, whether its transport was sockets or RDMA, and total training time. That combination is enough to catch the failures described here.

What changed after the new design

Treating RDMA, Lustre, placement and validation as one concern bought us more than a benchmark bump.

We ended up with a more reliable training foundation: fewer hidden performance failures, better GPU utilization, and multi-node workloads that were practical to run repeatedly.

The measured results were significant:

Measurement

TCP fallback

RDMA

Method

Peak bus bandwidth

–

355 GB/s

2-node NCCL all-reduce, H200 141GB nodes

Median step time

12.36 s

6.07 s

Qwen2.5-14B FSDP supervised fine-tune, 16 H200 GPUs across 2 nodes

Training throughput

1.0×

2.04×

Controlled A/B, identical ~360 s model-load phase

100-step benchmark

~1,950 s

~836 s

Separate fine-tune benchmark, 2 nodes

Speed is the least interesting part. What those numbers buy is GPU-hour efficiency, capacity planning we can trust, and confidence that the next model up will scale.

The broader cloud-native lesson

The lesson is not really about RDMA or Lustre. It is about what happens when you stop treating them as separate problems.

Cloud-native platforms already know how to orchestrate complex distributed systems. AI training pushes those platforms into a new set of constraints where network fabric, storage behavior, accelerator scheduling, and validation all become part of the product experience.

That is the job. Keep the user-facing story simple, and let the platform absorb the rest.

If you are building this: treat networking, storage, scheduling, readiness and validation as one system. That is what makes distributed training predictable as it scales.

Central takeaway: serious distributed AI training becomes viable when performance-critical infrastructure is designed as an integrated platform problem, not as a collection of independent features.

References

NCCL documentation: collectives, transports and environment variables

nccl-tests: the benchmark suite behind the bandwidth figures

GPUDirect RDMA documentation

The Lustre parallel filesystem

Kubeflow Trainer: distributed training on Kubernetes

Kueue: job queueing and gang scheduling for Kubernetes

Kubernetes CSI documentation