Paper Info
Title
Learning and Recovery in the ReLU Model
Abstract
Rectified linear units, or ReLUs, have become a preferred activation function for artificial neural networks. In this paper we consider two basic learning problems assuming that the underlying data follow a generative model based on a simple network with ReLU activations. The first problem we study corresponds to learning a generative model in the presence of nonlinearity (modeled by the ReLU functions). Given a set of signal vectors y <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> ∈ R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sup> , i = 1, 2, . . . , n, we aim to learn the network parameters, i.e., the d × k matrix A, under the model y <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> = ReLU (Ac <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> + b), where b ∈ R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sup> is a random bias vector. We show that it is possible to recover the column space of A within an error of O(d) (in Frobenius norm) under certain conditions on the distribution of b. The second problem we consider is that of robust recovery of the signal in the presence of outliers. In this setting, we are interested in recovering the latent vector c from its noisy nonlinear images of the form v = ReLU(Ac) + e + w, where e E Rd denotes the outliers with sparsity s and w E Rd denote the dense but small noise. We show that the LASSO algorithm recovers c E Rk within an ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> -error of O(√((k + s) log d)/d) when A is a random Gaussian matrix.
Year
DOI
Venue
2019
10.1109/ALLERTON.2019.8919900
2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
Keywords
DocType
ISSN
ReLU networks,robust signal recovery,matrix estimation,generative models,dictionary learning
Conference
2474-0195
ISBN
Citations 
PageRank 
978-1-7281-3152-8
0
0.34
References 
Authors
9
2
Name
Order
Citations
PageRank
Arya Mazumdar130741.81
Ankit Singh Rawat246533.94