As connected vehicles, drones, and industrial devices continue to generate vast amounts of data, industries face growing challenges in processing this information efficiently, while respecting privacy and minimizing communication costs. This is especially critical in areas like automotive safety, predictive maintenance, and autonomous systems.
To address this, we will lead a newly approved two-year research project funded by Vinnova, Sweden’s innovation agency for sustainable growth. The project brings together Scaleout, AI Sweden, and Zenseact with the shared goal of enhancing fleet intelligence using Federated Learning (FL) and Mixture-of-Experts (MoE) models.
Federated Learning (FL) enables devices to collaboratively train AI models without transferring sensitive raw data to a central location. However, FL faces its own challenges. Communication overhead can be high, especially with large models or frequent updates, and data is often highly varied across devices. Edge devices also have limited computational and energy resources, making it difficult to deploy and maintain complex models efficiently.
Mixture-of-Experts (MoE) architectures are designed to address the limitations of conventional neural networks in handling diverse and complex data. In an MoE model, a neural network is divided into several specialized sub-models, known as experts. Each expert is trained to handle a specific context or type of data, such as a particular region or weather condition. For each input, only the most relevant experts are used, which reduces the computational and communication demands on each device, making large AI models more efficient by activating only the necessary parts for each specific task.
We are developing a federated learning framework that integrates Mixture-of-Experts (MoE) models for applications such as automotive fleet perception. The system segments data by context, allowing each edge device to train and use expert models tailored to its environment. Only compact, context-specific model updates are communicated, using techniques like Low-Rank Adaptation to further minimize bandwidth usage. The central server aggregates these specialized experts into a global model, which can be dynamically composed based on the current needs of each device.
Unlike traditional MoE systems that rely on dynamic routing to select experts at runtime, our approach uses predefined contextual information to assign data to experts. This simplifies deployment and reduces resource requirements, making MoE particularly suitable for edge environments, where computational and communication constraints are critical. By combining these approaches, the project aims to deliver AI systems that are both resource-efficient and highly accurate, optimized for real-time decision-making at the edge.
The integration of Mixture-of-Experts with Federated Learning is expected to improve scalability, efficiency, and adaptability in environments with diverse and privacy-sensitive data. By reducing both computation and communication requirements, this architecture is suited for automotive fleets, autonomous vehicles, and other edge applications. The project also addresses important technical questions, such as ensuring diversity among experts when data distributions overlap, integrating new experts from additional clients without destabilizing the global model, and further reducing communication without increasing local computation or energy consumption.
Ultimately, this work aims to provide a foundation for more robust, scalable, and context-aware AI at the edge, enabling the deployment of advanced AI systems in real-world environments. Updates on progress and results will be shared as the project develops.
%201.svg)
