Google’s AutoML-Zero Evolves Machine Learning Algorithms Using Basic Math
Algorithms seem to be everywhere nowadays, working their magic behind recommendation engines or helping doctors detect rare genetic disorders. But while machine learning tools are gradually becoming more accessible to non-experts, designing machine learning algorithms themselves typically requires some expert know-how — and even then, it’s no guarantee that AI models will perform correctly, as sometimes their inner workings can be a mystery even to the initiated.
So it’s little wonder that the field of automated machine learning or AutoML has been garnering interest as of late, allowing non-experts to quickly apply machine learning to real-world problems without having to master its more arcane aspects first. And here’s an even more mind-blowing idea: what if machine learning algorithms could be programmed to automatically evolve better versions of themselves?
The notion of machine-evolved algorithms could potentially mean more creative solutions — ones that surpass the ability of human minds to conceive — in addition to dodging the limits that human bias might place on them. Google researchers seem to have achieved this innovation recently with AutoML-Zero, a new kind of automated machine learning that takes its cues from evolutionary biology. Instead of using human-defined rules and complex equations, the team’s pre-print paper describes how their system uses simple, high school-level mathematics and evolutionary algorithms, allowing it to create algorithms that can be selected and adapted to the task at hand, much like how only the “fittest” members of a species will survive in the process of natural selection.
As Esteban Real, a software engineer at Google and the paper’s lead author explained to us: “The field of AutoML aims to discover machine learning algorithms automatically. The goal is two-fold: we want to see if we can find something that we missed in our manual discovery and we want to apply it to domains where there are not enough human experts to go around. AutoML-Zero is a specific approach to AutoML that focuses on discovering algorithms from scratch, with as little human bias as possible.”
Generations of Algorithms
Building upon previous AutoML work done by both Google and other researchers, Real and his colleagues Chen Liang, David So and Quoc Le developed an intriguing approach that uses component functions, which act as basic building blocks for more complex algorithms, and also uses an updated version of a search method known as regularized evolution.
Similar to the Darwinism, the best-performing algorithm out of a population of “parent” algorithms will be copied to produce new generations of “child” algorithms, which are further “mutated” and winnowed down to the algorithm that’s most optimized for a particular task. While the team’s work takes prior research one step beyond, theirs is the first to evolve a full program from the ground up (though it’s important to note that the evolved algorithms stop evolving once they are taken out of the framework). In addition, to streamline the artificial evolutionary process, the team divided up large datasets into smaller chunks that were then used to evaluate child algorithms, via distributed processing.
To explain it further, Real likens their approach to building artificial neural networks as using even more fundamental components that function like bricks and mortar, rather than using the conventional algorithmic equivalent of pre-built rooms.
Architecture search “has traditionally built algorithms by combining large sophisticated human-designed components like batch-norm layers, dropout, and convolutions (these are like the ‘bedrooms, kitchens, and bathrooms’ that make up the resulting ‘house’ of the neural network),” said Real. “The goal of AutoML-Zero is to do it out of simple components that don’t need much human design (‘bricks and mortar’). Thus, AutoML-Zero has a much harder task to solve but also more potential to discover new things. Essentially, AutoML-Zero builds algorithms from basic components; this is much like building a house out of brick and mortar instead of prefabricated rooms.”
Surprisingly enough, when left to its own devices, the system will eventually evolve to produce classic algorithms like backpropagation and linear regression, even without the interference of human biases. For now, however, AutoML-Zero is only capable of evolving algorithms of only 10 to 20 lines long, while modern machine learning algorithms call for much longer programs. But with further work to improve the framework’s search efficiency, Real speculates that where there is an absence of human influence, that’s where AutoML will shine.
“This work demonstrates the potential to accelerate the discovery of new machine learning algorithms. This can be potentially helpful in the many domains where machine learning is making a difference, especially for the new applications where human expertise is also lacking.”
Read more in the team’s paper, and check out AutoML-Zero’s code on GitHub.
Images: Photo by Johannes Plenio via Unsplash; Google