Name
Abstract | ||
|---|---|---|
In this paper, we investigate the principle that good explanations are hard to vary in the context of deep learning. We show that averaging gradients across examples -- akin to a logical OR of patterns -- can favor memorization and `patchwork\u0027 solutions that sew together different strategies, instead of identifying invariances. To inspect this, we first formalize a notion of consistency for minima of the loss surface, which measures to what extent a minimum appears only when examples are pooled. We then propose and experimentally validate a simple alternative algorithm based on a logical AND, that focuses on invariances and prevents memorization in a set of real-world tasks. Finally, using a synthetic dataset with a clear distinction between invariant and spurious mechanisms, we dissect learning signals and compare this approach to well-established regularizers. |
Year | Venue | DocType |
|---|---|---|
2021 | ICLR | Conference |
Citations | PageRank | References |
0 | 0.34 | 0 |
Authors | ||
5 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Parascandolo, Giambattista | 1 | 41 | 3.51 |
| Neitz, Alexander | 2 | 3 | 1.39 |
| Orvieto, Antonio | 3 | 0 | 3.04 |
| Luigi Gresele | 4 | 1 | 2.44 |
| Bernhard Schölkopf | 5 | 23120 | 3091.82 |