Document Type
Conference Proceeding
Publication Date
2023
Abstract
Large scale transformer models, trained with massive datasets have become the standard in natural language processing. The huge size of most transformers make research with these models impossible for those with limited computational resources. Additionally, the enormous pretraining data requirements of transformers exclude pretraining them with many smaller datasets that might provide enlightening results. In this study, we show that transformers can be significantly reduced in size, with as few as 5.7 million parameters, and still retain most of their downstream capability. Further we show that transformer models can retain comparable results when trained on human-scale datasets, as few as 5 million words of pretraining data. Overall, the results of our study suggest transformers function well as compact, data efficient language models and that complex model compression methods, such as model distillation are not necessarily superior to pretraining reduced size transformer models from scratch.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Publication Information
Fields, Clayton and Kennington, Casey. (2023). "Exploring Transformers as Compact, Data-Efficient Language Models". In J. Jiang, D. Reitter, and S. Deng (Eds.), Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL) (pp. 521-531). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.conll-1.35