Hyperparameter Optimization

AI Research

Self-Tuning Networks: Amortizing the Hypergradient Computation for Hyperparameter Optimization

Microsoft Research shares this amazing talk on  the optimization of many deep learning hyperparameters can be formulated as a bilevel optimization problem. While most black-box and gradient-based approaches require many independent training runs, we aim to adapt hyperparameters online as the network trains. The main challenge is to approximate the response Jacobian, which captures how […]

Read More