Can an L2-neural LM Generalize Filler-gap Dependency?

doi:10.18627/jslg.38.3.202211.323

All Issue

2022 Vol.38, Issue 3 Preview Page

Research Article

Can an L2-neural LM Generalize Filler-gap Dependency?: Choi, Sunjoo¹, Yun, Yungdo¹, Park, Myung-Kwan^1*; ¹Dongguk University

30 November 2022. pp. 323-337

PDF

Abstract

Recent studies have shown that recurrent neural language models (LMs) can understand sentences involving filler-gap dependency (Chowdhury & Zamparelli, 2018; Wilcox et al., 2018, 2019). However, their behavior does not encode the underlying constraints that govern filler-gap acceptability. In this vein, significant issues remain about the extent to which LMs acquire specific linguistic constructions and whether these models recognize an abstract property of syntax in their representations. In this paper, following the lead of Bhattacharya and van Schijndel (2020), we further test whether the L2 neural LM can learn abstract syntactic constraints that have been claimed to govern the behavior of filler-gap constructions. To see this, we implement the L2 neural LM trained on the L2 corpus of English textbooks published in Korea for the last two decades, and then we test the representational overlap between disparate filler-gap constructions based on the syntactic priming paradigm. Unlike the previous studies of L1-neural LMs, we could not find sufficient evidence showing that the L2 neural LM learns a general representation of the existence of filler-gap dependency and the shared underlying constraints.

Keywords

filler-gap dependency

neural language model

syntactic priming

adaptation

References

Baroni, M. 2021. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. arXiv preprint arXiv:2106.08694. 10.1201/9781003205388-1
Bhattacharya, D. and van Schijndel, M. 2020. Filler-gaps that neural networks fail to generalize. Proceedings of the 24th Conference on Computational Natural Language Learning, 486-495. 10.18653/v1/2020.conll-1.39
Bock, J. K. 1986. Syntactic persistence in language production. Cognitive Psychology 18.3, 355-387. 10.1016/0010-0285(86)90004-6
Chang, F., Dell, G. S., and Bock, K. 2006. Becoming syntactic. Psychological Review 113.2, 234. 10.1037/0033-295X.113.2.234 16637761
Chaves, R. P. 2020. What don’t RNN language models learn about filler-gap dependencies? Proceedings of the Society for Computation in Linguistics 3.1, 20-30.
Choi, S. J. and Park, M. K. 2022a. An L2 neural language model of adaptation to dative alternation in English. The Journal of Modern British & American Language & Literature 40.1, 143-159.
Choi, S. J. and Park, M. K. 2022b. Syntactic priming by L2 LSTM language models. The Journal of Studies in Language 22, 547-562.
Choi, S. J. and Park, M. K. 2022c. An L2 neural language model of adaptation. Korean Journal of English Language and Linguistics 37.4. 475-489.
Choi, S. J., Park, M. K., and Kim, E. 2021. How are Korean neural language models ‘surprised’ layerwisely? Journal of Language Sciences 28.4, 301-317. 10.14384/kals.2021.28.4.301
Chomsky, N. 1973. Conditions on transformations. In S. Anderson and P. Kiparsky, editors, A Festschrift for Morris Halle, pages 232-286. New York: Holt, Rinehart & Winston.
Chomsky, N. 1986. Barriers. Massachusetts: MIT Press.
Chowdhury, S. A. and Zamparelli, R. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. Proceedings of the 27th International Conference on Computational Linguistics, 133-144.
Ettinger, A. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics 8, 34-48. 10.1162/tacl_a_00298
Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., and Levy, R. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. arXiv preprint arXiv:1903.03260. 10.18653/v1/N19-1004
Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., and Baroni, M. 2018. Colorless green recurrent networks dream hierarchically. arXiv preprint arXiv:1803.11138. 10.18653/v1/N18-1108
Kaschak, M. P., Kutta, T. J., and Schatschneider, C. 2011. Long-term cumulative structural priming persists for (at least) one week. Memory & Cognition 39.3, 381-388. 10.3758/s13421-010-0042-3 21264596
Kim, E. 2020. The ability of L2 LSTM language models to Learn the filler-gap dependency. Journal of the Korea Society of Computer and Information 25.11, 27-40.
Kodner, J. and Gupta, N. 2020. Overestimation of syntactic representation in neural language models. arXiv preprint arXiv:2004.05067. 10.18653/v1/2020.acl-main.160
Lepori, M. A., Linzen, T., and McCoy, R. T. 2020. Representations of syntax [MASK] useful: Effects of constituency and dependency structure in recursive LSTMs. arXiv preprint arXiv:2005.00019. 10.18653/v1/2020.acl-main.303
Linzen, T., Dupoux, E., and Goldberg, Y. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics 4, 521-535. 10.1162/tacl_a_00115
Pesetsky, D. 1987. Wh-in-situ: Movement and unselective binding. In E. Reuland and A. G. B. ter Meulen, editors, The Representation of (In)Definiteness, pages 98-129.
Prasad, G., Van Schijndel, M., and Linzen, T. 2019. Using priming to uncover the organization of syntactic representations in neural language models. arXiv preprint arXiv:1909.10579. 10.18653/v1/K19-1007 PMC6330790
Ross, J. R. 1967. Constraints on Variables in Syntax. ERIC, ED016965.
Sinclair, A., Jumelet, J., Zuidema, W., and Fernández, R. 2022. Syntactic persistence in language models: Priming as a window into abstract language representations. arXiv preprint arXiv:2109.14989. 10.1162/tacl_a_00504
Tenney, I., Das, D., and Pavlick, E. 2019. BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv:1905.05950. 10.18653/v1/P19-1452
Van Schijndel, M. and Linzen, T. 2018. A neural model of adaptation in reading. arXiv preprint arXiv:1808.09930. 10.18653/v1/D18-1499
Van Schijndel, M., Mueller, A., and Linzen, T. 2019. Quantity doesn’t buy quality syntax with neural language models. arXiv preprint arXiv:1909.00111. 10.18653/v1/D19-1592
Wilcox, E., Levy, R., and Futrell, R. 2019. Hierarchical representation in neural language models: Suppression and recovery of expectations. arXiv preprint arXiv:1906.04068. 10.18653/v1/W19-4819 30366611
Wilcox, E., Levy, R., Morita, T., and Futrell, R. 2018. What do RNN language models learn about filler-gap dependencies? arXiv preprint arXiv:1809.00042. 10.18653/v1/W18-5423

Information

Publisher :The Modern Linguistic Society of Korea
Publisher(Ko) :한국현대언어학회
Journal Title :The Journal of Studies in Language
Journal Title(Ko) :언어연구
Volume : 38
No :3
Pages :323-337
DOI :https://doi.org/10.18627/jslg.38.3.202211.323

[1] Baroni, M. 2021. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. arXiv preprint arXiv:2106.08694. 10.1201/9781003205388-1

[2] Bhattacharya, D. and van Schijndel, M. 2020. Filler-gaps that neural networks fail to generalize. Proceedings of the 24th Conference on Computational Natural Language Learning, 486-495. 10.18653/v1/2020.conll-1.39

[3] Bock, J. K. 1986. Syntactic persistence in language production. Cognitive Psychology 18.3, 355-387. 10.1016/0010-0285(86)90004-6

[4] Chang, F., Dell, G. S., and Bock, K. 2006. Becoming syntactic. Psychological Review 113.2, 234. 10.1037/0033-295X.113.2.234 16637761

[5] Chaves, R. P. 2020. What don’t RNN language models learn about filler-gap dependencies? Proceedings of the Society for Computation in Linguistics 3.1, 20-30.

[6] Choi, S. J. and Park, M. K. 2022a. An L2 neural language model of adaptation to dative alternation in English. The Journal of Modern British & American Language & Literature 40.1, 143-159.

[7] Choi, S. J. and Park, M. K. 2022b. Syntactic priming by L2 LSTM language models. The Journal of Studies in Language 22, 547-562.

[8] Choi, S. J. and Park, M. K. 2022c. An L2 neural language model of adaptation. Korean Journal of English Language and Linguistics 37.4. 475-489.

[9] Choi, S. J., Park, M. K., and Kim, E. 2021. How are Korean neural language models ‘surprised’ layerwisely? Journal of Language Sciences 28.4, 301-317. 10.14384/kals.2021.28.4.301

[10] Chomsky, N. 1973. Conditions on transformations. In S. Anderson and P. Kiparsky, editors, A Festschrift for Morris Halle, pages 232-286. New York: Holt, Rinehart & Winston.

[11] Chomsky, N. 1986. Barriers. Massachusetts: MIT Press.

[12] Chowdhury, S. A. and Zamparelli, R. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. Proceedings of the 27th International Conference on Computational Linguistics, 133-144.

[13] Ettinger, A. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics 8, 34-48. 10.1162/tacl_a_00298

[14] Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., and Levy, R. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. arXiv preprint arXiv:1903.03260. 10.18653/v1/N19-1004

[15] Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., and Baroni, M. 2018. Colorless green recurrent networks dream hierarchically. arXiv preprint arXiv:1803.11138. 10.18653/v1/N18-1108

[16] Kaschak, M. P., Kutta, T. J., and Schatschneider, C. 2011. Long-term cumulative structural priming persists for (at least) one week. Memory & Cognition 39.3, 381-388. 10.3758/s13421-010-0042-3 21264596

[17] Kim, E. 2020. The ability of L2 LSTM language models to Learn the filler-gap dependency. Journal of the Korea Society of Computer and Information 25.11, 27-40.

[18] Kodner, J. and Gupta, N. 2020. Overestimation of syntactic representation in neural language models. arXiv preprint arXiv:2004.05067. 10.18653/v1/2020.acl-main.160

[19] Lepori, M. A., Linzen, T., and McCoy, R. T. 2020. Representations of syntax [MASK] useful: Effects of constituency and dependency structure in recursive LSTMs. arXiv preprint arXiv:2005.00019. 10.18653/v1/2020.acl-main.303

[20] Linzen, T., Dupoux, E., and Goldberg, Y. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics 4, 521-535. 10.1162/tacl_a_00115

[21] Pesetsky, D. 1987. Wh-in-situ: Movement and unselective binding. In E. Reuland and A. G. B. ter Meulen, editors, The Representation of (In)Definiteness, pages 98-129.

[22] Prasad, G., Van Schijndel, M., and Linzen, T. 2019. Using priming to uncover the organization of syntactic representations in neural language models. arXiv preprint arXiv:1909.10579. 10.18653/v1/K19-1007 PMC6330790

[23] Ross, J. R. 1967. Constraints on Variables in Syntax. ERIC, ED016965.

[24] Sinclair, A., Jumelet, J., Zuidema, W., and Fernández, R. 2022. Syntactic persistence in language models: Priming as a window into abstract language representations. arXiv preprint arXiv:2109.14989. 10.1162/tacl_a_00504

[25] Tenney, I., Das, D., and Pavlick, E. 2019. BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv:1905.05950. 10.18653/v1/P19-1452

[26] Van Schijndel, M. and Linzen, T. 2018. A neural model of adaptation in reading. arXiv preprint arXiv:1808.09930. 10.18653/v1/D18-1499

[27] Van Schijndel, M., Mueller, A., and Linzen, T. 2019. Quantity doesn’t buy quality syntax with neural language models. arXiv preprint arXiv:1909.00111. 10.18653/v1/D19-1592

[28] Wilcox, E., Levy, R., and Futrell, R. 2019. Hierarchical representation in neural language models: Suppression and recovery of expectations. arXiv preprint arXiv:1906.04068. 10.18653/v1/W19-4819 30366611

[29] Wilcox, E., Levy, R., Morita, T., and Futrell, R. 2018. What do RNN language models learn about filler-gap dependencies? arXiv preprint arXiv:1809.00042. 10.18653/v1/W18-5423

The Journal of Studies in Language ISSN:1225-4770(Print) 2671-6151(Online) 언어연구

All Issue