Name
Abstract | ||
|---|---|---|
We investigate deep Bayesian neural networks with Gaussian weight priors and a class of ReLU-like nonlinearities. Bayesian neural networks with Gaussian priors are well known to induce an L2, "weight decay", regularization. Our results characterize a more intricate regularization effect at the level of the unit activations. Our main result establishes that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer. We show that first layer units are Gaussian, second layer units are sub-exponential, and units in deeper layers are characterized by sub-Weibull distributions. Our results provide new theoretical insight on deep Bayesian neural networks, which we corroborate with simulation experiments. |
Year | Venue | Field |
|---|---|---|
2019 | international conference on machine learning | Pattern recognition,Computer science,Weight decay,Gaussian,Regularization (mathematics),Bayesian neural networks,Artificial intelligence,Prior probability |
DocType | Citations | PageRank |
Conference | 0 | 0.34 |
References | Authors | |
0 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Mariia Vladimirova | 1 | 0 | 1.01 |
| J. J. Verbeek | 2 | 3944 | 181.44 |
| Pablo Mesejo | 3 | 2 | 2.44 |
| Julyan Arbel | 4 | 3 | 3.12 |