Name
Abstract | ||
|---|---|---|
Many papers are chasing state-of-the-art (SOTA) numbers, and more will do so in the future. SOTA-chasing comes with many costs. SOTA-chasing squeezes out more promising opportunities such as coopetition and interdisciplinary collaboration. In addition, there is a risk that too much SOTA-chasing could lead to claims of superhuman performance, unrealistic expectations, and the next AI winter. Two root causes for SOTA-chasing will be discussed: (1) lack of leadership and (2) iffy reviewing processes. SOTA-chasing may be similar to the replication crisis in the scientific literature. The replication crisis is yet another example, like evaluation, of over-confidence in accepted practices and the scientific method, even when such practices lead to absurd consequences. |
Year | DOI | Venue |
|---|---|---|
2022 | 10.1017/S1351324922000043 | NATURAL LANGUAGE ENGINEERING |
Keywords | DocType | Volume |
State-of-the-art, Evaluation, Benchmarks, Leaderboards, Root causes, Leadership, Reviewing, Replication crisis | Journal | 28 |
Issue | ISSN | Citations |
2 | 1351-3249 | 0 |
PageRank | References | Authors |
0.34 | 0 | 2 |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Kenneth Ward Church | 1 | 0 | 1.01 |
| Valia Kordoni | 2 | 0 | 0.68 |