Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding

Mengnan Du; Subhabrata Mukherjee; Yu Cheng; Milad Shokouhi; Xia Hu; Ahmed Hassan Awadallah

Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding

Mengnan Du, Subhabrata Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Hassan Awadallah

Data Science

New Jersey Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Recent work has focused on compressing pretrained language models (PLMs) like BERT where the major focus has been to improve the in-distribution performance for downstream tasks. However, very few of these studies have analyzed the impact of compression on the generalizability and robustness of compressed models for out-of-distribution (OOD) data. Towards this end, we study two popular model compression techniques including knowledge distillation and pruning and show that the compressed models are significantly less robust than their PLM counterparts on OOD test sets although they obtain similar performance on in-distribution development sets for a task. Further analysis indicates that the compressed models overfit on the shortcut samples and generalize poorly on the hard ones. We further leverage this observation to develop a regularization strategy for robust model compression based on sample uncertainty. Experimental results on several natural language understanding tasks demonstrate that our bias mitigation framework improves the OOD generalization of the compressed models, while not sacrificing the in-distribution task performance.

Original language	American English
Title of host publication	EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	1758-1770
Number of pages	13
ISBN (Electronic)	9781959429449
State	Published - 2023
Event	17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023 - Dubrovnik, Croatia Duration: May 2 2023 → May 6 2023

Publication series

Name	EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

Conference

Conference	17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023
Country/Territory	Croatia
City	Dubrovnik
Period	5/2/23 → 5/6/23

ASJC Scopus subject areas

Computational Theory and Mathematics
Software
Linguistics and Language

Cite this

Du, M., Mukherjee, S., Cheng, Y., Shokouhi, M., Hu, X., & Awadallah, A. H. (2023). Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding. In EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference (pp. 1758-1770). (EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference). Association for Computational Linguistics (ACL).

Du, Mengnan ; Mukherjee, Subhabrata ; Cheng, Yu et al. / Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding. EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2023. pp. 1758-1770 (EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference).

@inproceedings{8c2cdcd254fc462c9de3d39b200c63e5,

title = "Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding",

abstract = "Recent work has focused on compressing pretrained language models (PLMs) like BERT where the major focus has been to improve the in-distribution performance for downstream tasks. However, very few of these studies have analyzed the impact of compression on the generalizability and robustness of compressed models for out-of-distribution (OOD) data. Towards this end, we study two popular model compression techniques including knowledge distillation and pruning and show that the compressed models are significantly less robust than their PLM counterparts on OOD test sets although they obtain similar performance on in-distribution development sets for a task. Further analysis indicates that the compressed models overfit on the shortcut samples and generalize poorly on the hard ones. We further leverage this observation to develop a regularization strategy for robust model compression based on sample uncertainty. Experimental results on several natural language understanding tasks demonstrate that our bias mitigation framework improves the OOD generalization of the compressed models, while not sacrificing the in-distribution task performance.",

author = "Mengnan Du and Subhabrata Mukherjee and Yu Cheng and Milad Shokouhi and Xia Hu and Awadallah, {Ahmed Hassan}",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023 ; Conference date: 02-05-2023 Through 06-05-2023",

year = "2023",

language = "American English",

series = "EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1758--1770",

booktitle = "EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference",

address = "United States",

}

Du, M, Mukherjee, S, Cheng, Y, Shokouhi, M, Hu, X & Awadallah, AH 2023, Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding. in EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 1758-1770, 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, 5/2/23.

Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding. / Du, Mengnan; Mukherjee, Subhabrata; Cheng, Yu et al.
EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2023. p. 1758-1770 (EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding

AU - Du, Mengnan

AU - Mukherjee, Subhabrata

AU - Cheng, Yu

AU - Shokouhi, Milad

AU - Hu, Xia

AU - Awadallah, Ahmed Hassan

PY - 2023

Y1 - 2023

N2 - Recent work has focused on compressing pretrained language models (PLMs) like BERT where the major focus has been to improve the in-distribution performance for downstream tasks. However, very few of these studies have analyzed the impact of compression on the generalizability and robustness of compressed models for out-of-distribution (OOD) data. Towards this end, we study two popular model compression techniques including knowledge distillation and pruning and show that the compressed models are significantly less robust than their PLM counterparts on OOD test sets although they obtain similar performance on in-distribution development sets for a task. Further analysis indicates that the compressed models overfit on the shortcut samples and generalize poorly on the hard ones. We further leverage this observation to develop a regularization strategy for robust model compression based on sample uncertainty. Experimental results on several natural language understanding tasks demonstrate that our bias mitigation framework improves the OOD generalization of the compressed models, while not sacrificing the in-distribution task performance.

AB - Recent work has focused on compressing pretrained language models (PLMs) like BERT where the major focus has been to improve the in-distribution performance for downstream tasks. However, very few of these studies have analyzed the impact of compression on the generalizability and robustness of compressed models for out-of-distribution (OOD) data. Towards this end, we study two popular model compression techniques including knowledge distillation and pruning and show that the compressed models are significantly less robust than their PLM counterparts on OOD test sets although they obtain similar performance on in-distribution development sets for a task. Further analysis indicates that the compressed models overfit on the shortcut samples and generalize poorly on the hard ones. We further leverage this observation to develop a regularization strategy for robust model compression based on sample uncertainty. Experimental results on several natural language understanding tasks demonstrate that our bias mitigation framework improves the OOD generalization of the compressed models, while not sacrificing the in-distribution task performance.

UR - http://www.scopus.com/inward/record.url?scp=85159850694&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85159850694&partnerID=8YFLogxK

M3 - Conference contribution

T3 - EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

SP - 1758

EP - 1770

BT - EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference

PB - Association for Computational Linguistics (ACL)

T2 - 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023

Y2 - 2 May 2023 through 6 May 2023

ER -

Du M, Mukherjee S, Cheng Y, Shokouhi M, Hu X, Awadallah AH. Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding. In EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2023. p. 1758-1770. (EACL 2023 - 17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference).

Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Cite this