AI Bibliography

Deep Neural Networks (DNNs) have emerged as powerful models for numerous deep learning (DL) applications, such as image recognition, machine translation, object detection, and semantic segmentation. The abundance of successful, real-world DNN-based products and applications in our daily lives (e.g., 2D human pose estimation, Speech-to-Text APIs, real-time virtual-reality image rendering on head-mounted displays, to name a few) has ignited an ever-increasing interest in pushing the performance of DNNs to achieve state-of-the-art results. DNNs have been traditionally designed by human experts in an expensive and meticulous process, which has been dubbed by many researchers as more of an art than science. However, as modern DNN models become increasingly deeper and larger, the task of hand-tailored DNN design has become a daunting challenge.

AutoML presents a promising path for alleviating the engineering costs and the complexity that are intrinsic to the design of neural networks, by automating the tuning of DNN hyperparameters (e.g., the number of layers, the type of operations per layer, etc.) and by formulating the design of DNNs as a hyperparameter optimization problem. In fact, we are witnessing a proliferation of novel AutoML approaches, with formulations spanning many different methodologies, such as black-box hyperparameter optimization (HPO) and Neural Architecture Search (NAS). Notably, commercial interest in AutoML has grown dramatically in recent years, as demonstrated by the vast computational resources committed to industry-driven AutoML research and by the plethora of commercial cloud-based services and frameworks. Overall, AutoML methodologies constitute a research topic of paramount importance, since the commoditization of “push-button” DL solutions without the need for DL experts is expected to have significant reverberations across multiple industries.