BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

Paper Info

Title
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

Abstract
We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34 k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.

Year	DOI	Venue
2022	10.1109/JSTSP.2022.3182537	IEEE Journal of Selected Topics in Signal Processing
Keywords	DocType	Volume
Giant model,large-scale self-supervisedlearning,self-supervised learning,semisupervised learning,speech recognition	Journal	16
Issue	ISSN	Citations
6	1932-4553	2
PageRank	References	Authors
0.37	25	26

Authors (26 rows)

Cited by (2 rows)

References (25 rows)

Name	Order	Citations	PageRank
Yu Zhang	1	442	41.79
Daniel S. Park	2	22	3.46
Wei Han	3	2	0.37
James Qin	4	13	3.68
Anmol Gulati	5	24	3.31
Joel Shor	6	55	5.47
Lorena Álvarez	7	504	36.47
Yuanzhong Xu	8	224	9.30
Yanping Huang	9	210	9.80
Shibo Wang	10	2	0.37
Zongwei Zhou	11	2	0.70
Bo Li	12	206	42.46
Min Ma	13	2	0.70
William Chan	14	357	24.67
Jiahui Yu	15	260	25.83
Yongqiang Wang	16	2	0.37
liangliang cao	17	1816	90.71
Khe Chai Sim	18	300	31.13
Bhuvana Ramabhadran	19	1779	153.83
Tara N. Sainath	20	3497	232.43
Françoise Beaufays	21	2	0.37
Zhifeng Chen	22	2747	106.75
Quoc V. Le	23	8501	366.59
Chung-Cheng Chiu	24	248	28.00
Ruoming Pang	25	1092	92.99
Yonghui Wu	26	1065	72.78