Introduction
Much of the world's most valuable data is created at the edge: on phones, vehicles, industrial machines and hospital systems. Often that data cannot be moved to a central place to train a model. It may be too sensitive, as with medical or personal data. It may be restricted by regulation such as GDPR. Or it may simply be too large or too costly to transfer. Federated learning solves this by bringing the training to the data instead. Each device trains on its own data and shares only a model update, and the updates are combined into a shared global model. The raw data never leaves the device.
Federated learning is easy to demonstrate with a handful of clients but much harder to run at real-world scale. A production cross-device deployment may involve tens or hundreds of thousands of devices. These devices are spread across countries and continents, connect over networks of varying quality, and are only available some of the time. The infrastructure has to handle all of this: onboarding and tracking a very large fleet of clients, keeping long-distance traffic manageable, and continuing to make progress when individual clients or servers drop out. A platform that works for a hundred clients in a single data centre gives no guarantee that it will hold up under these conditions.
This post shows how Scaleout Edge handles these conditions. We evaluated the platform with a target of 100,000 simulated clients across ten global regions. Although extreme initial load kept the active pool slightly below 100,000, the system successfully maintained continuous training. We first outline the architecture, deployment, and experimental setup, before sharing the final results and hardware requirements to guide future large-scale deployments.
Scaleout Edge Architecture
The Scaleout Edge architecture follows a three-tier design
Tier 1 consists of the clients, anything from drones, trucks to smartphones, factory PLCs, or hospital servers, each training on local data (which never leaves the clients) and producing a model update for the current round. The client side SDKs are the main entry point for developers. They specify machine learning code for computing model updates, integrate with their own application (as needed), and specify model validation and telemetry routines. They also have the possibility to specify custom server-side model aggregation schemes.
Tier 2 consists of combiner/s and a reducer. Every client is assigned to a single combiner. During each round each combiner waits until it has received all of its clients’ updates, or until the round timeout elapses. It then proceeds to aggregate its updates into a partial model. When every combiner has produced its partial model (or the round timeout is reached), the reducer merges the partial models into a new global model. That global model is then sent to the clients for validation, and the process is iterated until the desired number of rounds have been executed.
Tier 3 houses the control plane. The controller orchestrates each training round, tracks the lineage of global models, and maintains a global state. Alongside it, a discovery service detects inbound client connections and routes each new client to an available combiner.
Scaleout Edge is built for fault tolerance and scalability
The three-tier architecture makes horizontal scaling straightforward. When 100 000 clients were started, 20 combiners were deployed, each handling 5000 clients. Because the combiners never speak to one another and are solely responsible for its designated clients, Scaleout Edge provides a high fault-tolerance. An unresponsive combiner only affects its respective clients and the remaining combiners can continue to aggregate their partial models, which allows the reducer to produce a new global model and keep the training session moving forward.
Fault tolerance and scalability also extend to the control plane. Because Scaleout Edge is deployed on Kubernetes, components can be scaled out on demand. During the startup of the 100,000 clients, heavy load caused timeouts in the API server, and the run began to crash. Scaling the API server to 4 replicas and the NGINX ingress to 3 replicas successfully mitigated this; the load was distributed, allowing the run to recover and the clients to connect successfully.
Geographically Distributed Deployment
To resemble a real cross-device deployment, the clients and combiners were spread over ten regions on five continents: Finland, Belgium, Poland, three locations in the US, Brazil, India, Japan and Australia (Figure 2).
The control plane runs in a single hub in Finland, a Kubernetes cluster that holds the controller, API server, database, model storage and server functions. Every other region holds only the parts that need to be close to the clients: two combiners and three client machines, which together host 10,000 clients. The hub runs its own two combiners, which gives 20 combiners in total, each serving 5,000 clients.
Clients connect only to the combiners in their own region, so the bulk of the traffic, model downloads and uploads, never leaves the region. Only the combiners communicate with the hub across continents, to fetch the global model, report their clients and send their partial models for reduction. This is what lets the system scale: adding a region adds local capacity without adding long-distance traffic from the clients.
Experimental Setup
Task and model
Each client trains a small fully connected neural network, with two hidden layers of 64 and 32 units, to classify handwritten digits from MNIST. Each client draws its own random local dataset of 600 training and 100 test images. In each round a selected client first evaluates the incoming global model on its local test set and reports loss and accuracy. It then trains for one epoch and returns its updated model. The updates are combined with FedAvg, weighted by each client's number of training examples. Each round has a timeout of 400 seconds, after which the combiners aggregate whatever updates they have received, so slow or unresponsive clients cannot stall the round.
Running 100,000 clients
Each client is a separate process with its own connection, identity and data, so from the platform's point of view it behaves like an independent device. Each client machine runs about 3,300 clients. To prevent the control plane from being overwhelmed, the deployment was designed to bring clients online gradually, one region at a time. Additionally, communication across the entire deployment, including all clients, was strictly constrained to limit network overhead.
Experimental Results
Convergence
MNIST with randomly sampled local data is a toy task, so the accuracy itself is not the point. The model is there to give the clients real training work and to show that the updates from 100,000 clients are aggregated correctly. The global model converged as expected (Figures 3 and 4), and the run was stopped after 30 rounds once accuracy had levelled off.
Simulating Realistic participation through server functions
In real cross-device deployments only a small fraction of devices are available at any moment, because they are charging, offline or in use. This was simulated with a server function, a small piece of custom Python code that runs on the server and overrides which clients are selected for each round. The client code is unchanged. Each client is available with a probability of 10%, and its availability is re-drawn every 10–60 seconds. A client therefore moves in and out of availability over the course of the run, as a real device would. Each round selects only the clients that are available at that moment.
Participation per round
About 8% of the 100,000 clients, roughly 8,300, contributed an update in each round. The rate stayed flat from the first round to the last, so neither the number of rounds nor the accumulated load degraded participation. Every region contributed in every round. The European and US regions each contributed close to 1,000 updates per round, while the regions furthest from the hub contributed fewer, with Sydney the lowest at about 480 (Figure 5). This is because fewer of their clients managed to connect during start-up. In future deployments, this bottleneck could be avoided by onboarding distant regions more slowly.
Coverage of the client population
Because a different subset is selected each round, the global model learns from a growing share of the population over time. After 30 rounds, 89% of all clients had contributed at least once (Figure 6). Coverage ranges from about 95% in the best-connected regions down to 62% in Sydney. The typical client took part in only 2–3 of the 30 rounds, so the training was spread across the whole population and not dominated by a few clients.
Reference Deployment
The whole experiment ran on standard cloud virtual machines, with no specialised hardware and no GPUs.
| Component | Machine size | Count |
|---|---|---|
| Control plane (Kubernetes nodes) | 8 vCPU, 32 GB RAM | 3–5 |
| Regional combiners | 2 vCPU, 8 GB RAM | 20 |
| Client machines | 32 vCPU, 128 GB RAM | 30 |
Table 1: Hardware used for the 100,000-client run.
The control plane was installed on Kubernetes with the Scaleout Edge Helm chart. To handle 100,000 clients connecting at once, the API server was scaled to 4 replicas and the ingress to 3. Each regional combiner is a single small 2-vCPU machine serving 5,000 clients.
Scaling horizontally with multiple lightweight VMs is more cost-effective than provisioning large, high-compute instances, while simultaneously reducing the impact of any single node failure. Deploying multiple combiners per region provides this necessary redundancy: if a combiner were to fail, its clients are designed to reconnect to another available combiner. This architecture ensures that local hardware failures do not delay global model updates.
Finally, the client machines in this experiment exist only to simulate the workload; in a real deployment, these would be the actual edge devices.
As shown in the analysis of how each combiner spends a round, combiners spend the vast majority of their time either waiting for clients to complete local training and communication or sitting idle while waiting for other combiners and the central control plane to finish aggregation. In regions closest to the hub, such as Hamina, Belgium, and Warsaw, combiners finish client updates quickly and spend most of their time idle. Conversely, in regions furthest from the hub like Sydney and Tokyo, combiners spend most of the round time waiting for client communications and local execution within the round timeout window.
Conclusions
This evaluation demonstrates how Scaleout Edge manages large-scale, distributed federated learning. It illustrates that the platform can scale to 100,000 clients by leveraging a decentralized three-tier architecture. By offloading workload to regional combiners and keeping communication thresholds to a minimum, the system limits network overhead and mitigates control-plane bottlenecks. Additionally, the inherent redundancy in the architecture helps prevent intermittent client availability from stalling global model updates, while the regional combiners decouple local network disruptions from the global training process.
Ultimately, this deployment validates the platform's architectural design for handling large-scale machine learning workloads at the edge.
Acknowledgements
This work was made possible by funding from Vinnova under the project “Validating a System Development Kit for edge federated learning” (2023-01890).