Evaluating Intra-Grade Consistency in First and Second-Year High School Mock CSAT Reading Passages via Coh-Metrix

doi:10.18627/jslg.40.1.202405.39

All Issue

2024 Vol.40, Issue 1 Preview Page Next Page

Research Article

Evaluating Intra-Grade Consistency in First and Second-Year High School Mock CSAT Reading Passages via Coh-Metrix: Lee, Hyejin¹, Ryu, Jisu², Jeon, Moongee^2*; ¹Wonkwang University
²Konkuk University

31 May 2024. pp. 39-55

PDF

Abstract

This corpus-informed research, which employs the Coh-Metrix analysis, examines the intra-grade continuity of the first and second-year mock CSAT exams, administered quarterly by four regional educational offices: Seoul in March, Busan in June, Incheon in September, and Gyeonggi Province in November. The corpus consisting of 200 reading passages compiled from the 2023 mock CSAT exams, was analyzed via Coh-Metrix to explore a broad spectrum of linguistic features across lexical, syntactic, textual, and cohesion-related indices. Overall, the results indicated that, except for the second-grade average word length, there was no statistically significant evidence to support strong intra-grade continuity. These findings provide valuable insights into the design and educational practices of the exams, highlighting areas for potential enhancement to meet test objectives more effectively.

Keywords

mock CAST English tests

intra-grade continuity

reading passages

corpus analysis

Coh-Metrix

References

Chang, J. 2019. A comparison of lexical complexity in 2016-2019 CSAT English reading passages by the types of assessment (norm-referenced assessment vs. criterion-referenced assessment). The SNU Journal of Education Research 28.3, 85-110. 10.54346/sjer.2019.28.3.85
Chang, J. 2022. A multi-dimensional comparison of CSAT English reading passages before and after the use of criterion-referenced assessment. Language Research 58.1, 61-89. 10.30961/lr.2021.58.1.61
Choi, Y. 2024. Machine learning-based item difficulty prediction model using item text features. Secondary English Education 17.1, 46-66. 10.20487/kasee.17.1.202402.47
Coltheart, M. 1981. The MRC psycholinguistic database. Quarterly Journal of Experimental Psychology 33, 497-505. 10.1080/14640748108400805
Doo, S. and Choi, I. 2020. A comparative analysis on difficulty of LC and RC of the KCSAT and the mock KCSAT English tests. English Language Assessment 15.2, 119-147. 10.37244/ela.2020.15.2.119
Graesser, A. C., Jeon, M., Cai, Z. and McNamara, D. S. 2008. Automatic analyses of language, discourse, and situation models. In J. Auracher and W. van Peer, (eds.), New Beginnings in Literary Studies. Cambridge: Cambridge Scholars Publishing. 72-88.
Graesser, A. C., Jeon, M., Yan, Y. and Cai, Z. 2007. Discourse cohesion in text and tutorial dialogue. Information Design Journal 15.3, 199-213. 10.1075/idj.15.3.02gra
Graesser, A. C., McNamara, D. S., Louwerse, M. M. and Cai, Z. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavioral Research Methods, Instruments, and Computers 36.2, 193-202. 10.3758/BF03195564 15354684
Kim, J. and Kim, H. 2021. A comparison of the lexical, syntactic, and discourse features between the listening and the reading texts in the College Scholastic Ability Test. Modern English Education 22.1, 45-56. 10.18095/meeso.2021.22.1.45
Kim, K. and Cha, K. 2021. An analysis on the vocabulary of recent CSAT’s English listening assessment: Lexical scales, word difficulty, and high frequency content words. English Language Teaching 33.2, 23-36.
Kim, S. 2021. A corpus-based analysis of linguistic difficulty of the CSAT English section under absolute grading. English Language Assessment 16.2, 205-227. 10.37244/ela.2021.16.2.205
Koh, N. and Shin, J. 2017. A comparison of the level of difficulty in the English reading part of the CSAT: Before and after the EBS-CSAT linkage policy. Secondary English Education 10.4, 3-24.
McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. 2014. Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press. 10.1017/CBO9780511894664
Millis, K. and Just, M. 1994. The influence of connectives on sentence comprehension. Journal of Memory and Language 33, 128-147. 10.1006/jmla.1994.1007
Ministry of Education. 2020. The national curriculum of English language. Notification No. 2020-255 of the Ministry of Education [Supplement 14]. Ministry of Education.
Murray, J. 1997. Connective and narrative text: The role of continuity. Memory and Cognition 25.2, 227-236. 10.3758/BF03201114 9099073
Oh, J. and Shin, Y. 2020. A corpus-based analysis of the linguistic complexity levels of reading passages in the Korean college entrance examination. English Language Teaching 32.2, 109-126.
Shin, Y. 2019. Analyzing CSAT reading passages by using Coh-Metrix and VocaProfile: Focusing on four years from 2016 to 2019. Journal of Language Sciences 26.4, 109-127. 10.14384/kals.2019.26.4.109
Lee, D. 2015. The analysis of linkage of the Korea Educational Broadcasting System (EBS) with the College Scholastic Ability Test (CSAT) & trial test for CSAT and the study on improvements for the linkage (in the English part of the test). Unpublished master’s thesis, Kongju National University.
Lee, D. and Kang, M. 2015. The analysis of linkage of the Korea educational broadcasting system (EBS) with the college scholastic ability test (CSAT) & trial test for CSAT and the study on improvements for the linkage (in the English Part of the Test). Secondary English Education 8.4, 69-94.
Lee, H. 2018. Vocabulary analysis of the English section of the 2017 CSAT and CSAT simulations. International Language and Literature 41, 195-216.
Lee, J. 2022. The comparative analysis on the text difficulty among English reading question types in CSAT English Language Assessment 17.1, 45-63. 10.37244/ela.2022.17.1.45
Lee, J. 2023. A Comparison of Linguistic Aspects in English Reading Passages of 2012-2023 CSAT Mock Tests Between Grading Systems and Between June and September Tests. Journal of the Korea English Education Society 22.1, 1-23.
Park, J. and Lee, D. 2021. An Analysis of the Difficulty of Reading Passages in the CSAT and High School English Textbooks of the 2015 Revised Curriculum. Studies on English Language & Literature 47.3 147-170.
Ryu, J. and Jeon, M. 2023. An analysis of the inter-grade continuity of the reading passages of high school English mock CSAT tests using Coh-Metrix. English Language & Literature Teaching 29.2, 41-59. 10.35828/etak.2023.29.2.41

Information

Publisher :The Modern Linguistic Society of Korea
Publisher(Ko) :한국현대언어학회
Journal Title :The Journal of Studies in Language
Journal Title(Ko) :언어연구
Volume : 40
No :1
Pages :39-55
DOI :https://doi.org/10.18627/jslg.40.1.202405.39

[1] Chang, J. 2019. A comparison of lexical complexity in 2016-2019 CSAT English reading passages by the types of assessment (norm-referenced assessment vs. criterion-referenced assessment). The SNU Journal of Education Research 28.3, 85-110. 10.54346/sjer.2019.28.3.85

[2] Chang, J. 2022. A multi-dimensional comparison of CSAT English reading passages before and after the use of criterion-referenced assessment. Language Research 58.1, 61-89. 10.30961/lr.2021.58.1.61

[3] Choi, Y. 2024. Machine learning-based item difficulty prediction model using item text features. Secondary English Education 17.1, 46-66. 10.20487/kasee.17.1.202402.47

[4] Coltheart, M. 1981. The MRC psycholinguistic database. Quarterly Journal of Experimental Psychology 33, 497-505. 10.1080/14640748108400805

[5] Doo, S. and Choi, I. 2020. A comparative analysis on difficulty of LC and RC of the KCSAT and the mock KCSAT English tests. English Language Assessment 15.2, 119-147. 10.37244/ela.2020.15.2.119

[6] Graesser, A. C., Jeon, M., Cai, Z. and McNamara, D. S. 2008. Automatic analyses of language, discourse, and situation models. In J. Auracher and W. van Peer, (eds.), New Beginnings in Literary Studies. Cambridge: Cambridge Scholars Publishing. 72-88.

[7] Graesser, A. C., Jeon, M., Yan, Y. and Cai, Z. 2007. Discourse cohesion in text and tutorial dialogue. Information Design Journal 15.3, 199-213. 10.1075/idj.15.3.02gra

[8] Graesser, A. C., McNamara, D. S., Louwerse, M. M. and Cai, Z. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavioral Research Methods, Instruments, and Computers 36.2, 193-202. 10.3758/BF03195564 15354684

[9] Kim, J. and Kim, H. 2021. A comparison of the lexical, syntactic, and discourse features between the listening and the reading texts in the College Scholastic Ability Test. Modern English Education 22.1, 45-56. 10.18095/meeso.2021.22.1.45

[10] Kim, K. and Cha, K. 2021. An analysis on the vocabulary of recent CSAT’s English listening assessment: Lexical scales, word difficulty, and high frequency content words. English Language Teaching 33.2, 23-36.

[11] Kim, S. 2021. A corpus-based analysis of linguistic difficulty of the CSAT English section under absolute grading. English Language Assessment 16.2, 205-227. 10.37244/ela.2021.16.2.205

[12] Koh, N. and Shin, J. 2017. A comparison of the level of difficulty in the English reading part of the CSAT: Before and after the EBS-CSAT linkage policy. Secondary English Education 10.4, 3-24.

[13] McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. 2014. Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press. 10.1017/CBO9780511894664

[14] Millis, K. and Just, M. 1994. The influence of connectives on sentence comprehension. Journal of Memory and Language 33, 128-147. 10.1006/jmla.1994.1007

[15] Ministry of Education. 2020. The national curriculum of English language. Notification No. 2020-255 of the Ministry of Education [Supplement 14]. Ministry of Education.

[16] Murray, J. 1997. Connective and narrative text: The role of continuity. Memory and Cognition 25.2, 227-236. 10.3758/BF03201114 9099073

[17] Oh, J. and Shin, Y. 2020. A corpus-based analysis of the linguistic complexity levels of reading passages in the Korean college entrance examination. English Language Teaching 32.2, 109-126.

[18] Shin, Y. 2019. Analyzing CSAT reading passages by using Coh-Metrix and VocaProfile: Focusing on four years from 2016 to 2019. Journal of Language Sciences 26.4, 109-127. 10.14384/kals.2019.26.4.109

[19] Lee, D. 2015. The analysis of linkage of the Korea Educational Broadcasting System (EBS) with the College Scholastic Ability Test (CSAT) & trial test for CSAT and the study on improvements for the linkage (in the English part of the test). Unpublished master’s thesis, Kongju National University.

[20] Lee, D. and Kang, M. 2015. The analysis of linkage of the Korea educational broadcasting system (EBS) with the college scholastic ability test (CSAT) & trial test for CSAT and the study on improvements for the linkage (in the English Part of the Test). Secondary English Education 8.4, 69-94.

[21] Lee, H. 2018. Vocabulary analysis of the English section of the 2017 CSAT and CSAT simulations. International Language and Literature 41, 195-216.

[22] Lee, J. 2022. The comparative analysis on the text difficulty among English reading question types in CSAT English Language Assessment 17.1, 45-63. 10.37244/ela.2022.17.1.45

[23] Lee, J. 2023. A Comparison of Linguistic Aspects in English Reading Passages of 2012-2023 CSAT Mock Tests Between Grading Systems and Between June and September Tests. Journal of the Korea English Education Society 22.1, 1-23.

[24] Park, J. and Lee, D. 2021. An Analysis of the Difficulty of Reading Passages in the CSAT and High School English Textbooks of the 2015 Revised Curriculum. Studies on English Language & Literature 47.3 147-170.

[25] Ryu, J. and Jeon, M. 2023. An analysis of the inter-grade continuity of the reading passages of high school English mock CSAT tests using Coh-Metrix. English Language & Literature Teaching 29.2, 41-59. 10.35828/etak.2023.29.2.41

The Journal of Studies in Language ISSN:1225-4770(Print) 2671-6151(Online) 언어연구

All Issue