AutoML-Zero: How is Google’s new automated ML Algorithm.

AutoML-Zero: How is Google’s new automated ML Algorithm.

Self Evolution of Machine Learning Algorithms

Machine learning algorithms have achieved remarkable milestones while automating our lives. It is possible because of decades of research in the areas of machine learning and artificial intelligence. Designing a new machine learning model and training the model on real data that includes the optimization of the hyper-parameters is still a challenging job for the machine learning researchers. It is very time consuming and complex process. This challenge has populated a new area of research which is called Automated Machine Learning in short AutoML . It is actually an automated mechanism to evolve the new Machine Learning algorithm with less human participation. The aim of AutoML is to automate human research time while designing a new machine learning algorithm. A lot of research has happened in the area of AutoML. But it is facing two major challenges, the first one is a human-designed component bias which makes the AutoML less innovative and biased. The other one is the handling of the constrained search spaces. These two aspects severely affect the ability of any AutoML scheme. Recently, the Google AI team has tried to address this and named it AutoML-Zero . So let’s see what is AutoML-Zero.

AutoML-Zero proposes a mechanism for the automatic search for whole ML algorithms with little restrictions and with simple mathematical operations. It aims to search a fine-grained space simultaneously for the modern, optimization procedure and initialization . The beauty of AutoML-Zero is it requires very less human design and provides a scope to discover even non-neural network algorithms. Let us try to understand the framework of AutoML-Zero. In the framework of the AutoML-Zero, ML algorithms are proposed as a computer program with three components: Setup, Predict and Learn. It actually does the job of initialization, prediction and learning. These functions use simple mathematical operations on a small memory. Interestingly, the proposed scheme is capable of searching through 10,000 models/second/CPU core. Let us see how this method works.

AutoML-Zero leads towards the automatic discovery of algorithms that performs well on a given set of ML tasks. First of all search experiments explore a very large space of algorithms and determines the optimal and generalizable algorithm. Each experiment of these kinds identifies a candidate algorithm, two searches are performed one is random search for baseline and another one is evolutionary search as the main search to accomplish this task. Let us understand the search space and search methods in detail.

As stated earlier the algorithms are treated as computer programs that act on a small virtual memory with separate address spaces for scalar, vector and matrix variables. The evaluation of candidate algorithms follows the following framework:

The measure of the quality of the algorithms is evaluated by the search method:

A population is initialized with empty programs. It then evolves in repeating cycles to produce better and better learning algorithms. At each cycle, two (or more) random models compete and the most accurate model gets to be a parent. The parent clones itself to produce a child, which gets mutated. That is, the child’s code is modified in a random way, which could mean, for example, arbitrarily inserting, removing or modifying a line in the code. The mutated algorithm is then evaluated on image classification tasks.

To reach a throughput of 2k-10k algorithms/sec/cpu core, along with the use of small proxy tasks two additional modifications are made: A functional equivalence checker is employed that detects equivalent supervised algorithms, in spite of different implementations. It is achieved by recording the prediction of an algorithm after executing 10 training and 10 validation steps on a fixed set of examples. Secondly, some hurdles are added to reach further 5x throughput.

The following image depicts the progress of one evolution experiment.

Ultimately, the proposed method by the authors shows a good enhancement in the results to its competitive AutoML algorithms. For more details and implementations, the learners should visit the following pages:

References:

Real, E., Liang, C., So, D. R., & Le, Q. V. (2020). AutoML-Zero: Evolving Machine Learning Algorithms From Scratch. arXiv preprint arXiv:2003.03384 .