HPE Swarms of Machine Learning – The New Stack

The idea of ​​spreading machine learning is not new. Google was one of the first to implement it widely by training Android phones to perform automatic keyboard completion. And with the growth of the IoT, there is the incentive to push ML model processing to the edge, an idea often associated with the emerging concept of fog computing. The idea is not new, but implementation and commercialization are still in their infancy. Now HPE has come up with ‘Swarm Learning’, a new twist on federated learning that uses blockchain technology.

Federated learning is an approach that remotely trains models on data sources where they live, then communicates the trained models only back to a hub where the finished model is determined through a consensus process. The obvious use case is IoT, where data streams are generated by remote devices at a scale where it wouldn’t make sense to send all that data back to the cloud for model training or inference. Commercially, providers like Integrate.ai and Devron are starting to deliver solutions for managing federated learning, but as just noted, this solution space is still in its infancy.

Enter Blockchain

A potential drawback of these approaches is that they depend on a hub, which can become a bottleneck or a single source of interference. That’s where HPE’s Swarm Learning approach comes in. It not only collects and scales training and inference workloads, but also does so using blockchain approaches.

At first glance, what seems more buzzy than “Machine Learning” and “Blockchain?” But there is a real method to the madness. As envisioned, “Swarm Machine Learning” targets privacy or regulatory scenarios that prevent or discourage data movement. That is the reason for the blockchain. For example, you may have a group of hospitals collaborating on a study to apply machine learning to disease prevention, detection, or outcomes, but patient data is the immovable barrier.

HPE deploys a blockchain based on Ethereum technology that will keep data in place, train models and run locally, in an environment where exchanged model results become tamper-proof. HPE sets up a Swarm network where individual nodes register, and those nodes then perform the modeling.

It contains several components. It starts with Swarm Learning libraries delivered as containers that can run on any target infrastructure built on Kubernetes. The models themselves remain intact; HPE claims that the models can be deployed on the swarm with just four extra lines of code. Then there is the Swarm Network, the blockchain, a control plane and a license server.

Potential Use Scenarios

There are many possible use cases for distributed learning. In healthcare, hospitals around the world can apply ML to identify cancers on MRI images, creating a large base of training data that does not need to be moved to a central location. Consortia of financial institutions can work together to build fraud detection or personalization models for global data sets. Global marketing campaigns through franchisee retail networks are another example where wealth modeling could be shared in a way that respects local ownership of data. And, of course, there are instances where global models span data that regulations do not allow to cross borders.

For now, Swarm ML is still in the early adopter phase for HPE, with plans to eventually produce it as part of HPE Ezmeral MLOps. While it doesn’t necessarily require a professional service, we expect that early adopters will likely need expert help for the jumpstart.

Featured image of PollyDot from Pixabay.

Leave a Comment