Evaluating TOEIC Instruction through the CIPP Model: Insights from an Indonesian Aviation Academy
DOI:
https://doi.org/10.33394/jo-elt.v12i2.18146Keywords:
TOEIC instruction, CIPP model, Aviation English, ESP evaluation, ICAO language proficiency, Self-efficacyAbstract
English proficiency is essential for safe pilot-air traffic controller communication in aviation operations. While many Indonesian flight academies adopt TOEIC as an institutional benchmark for English certification, questions remain about its effectiveness in measuring the communicative competence required for operational aviation contexts. This qualitative study evaluated TOEIC instruction at an Indonesian Flight Academy using the CIPP (Context, Input, Process, Product) evaluation model. Data revealed a significant gap between strong institutional policy endorsement of TOEIC and actual classroom practice. Although institutional frameworks mandated TOEIC certification, curriculum implementation remained insufficiently aligned with ICAO language proficiency standards, instructors lacked specialized aviation English training, and teaching focused predominantly on test preparation rather than authentic communication scenarios. Notably, while cadets achieved respectable TOEIC scores (550–780), their operational communicative readiness during simulated aviation tasks remained limited. The study recommends curriculum realignment linking TOEIC objectives with ICAO descriptors, mandatory ESP professional development for instructors, and integration of simulation-based communicative tasks. Learner self-efficacy emerged as a crucial psychological factor mediating the transfer of language learning to real communication. This evaluation contributes to ESP pedagogy and aviation English policy by demonstrating how institutional, pedagogical, and psychological dimensions interact to shape language instruction effectiveness in professional, safety-critical contexts.
References
Abeyratne, R. (2008). Language proficiency in aeronautical communications: Some recent developments. Air and Space Law, 33(4–5), 356–367. https://doi.org/10.54648/aila2008027
Alderson, J. C. (2009). Air safety, language assessment policy, and policy implementation: The case of aviation English. Annual Review of Applied Linguistics, 29, 168–187. https://doi.org/10.1017/S0267190509090138
Alderson, J. C. (2010). A survey of aviation English tests. Language Testing, 27(1), 51–72. https://doi.org/10.1177/0265532209347196
Aranda, W., & Henneberry, D. (2025). The role of simulated air traffic control environment (SATCE) in the implementation of ICAO level 4 requirements. Advances in Human Factors and Ergonomics, 186. https://doi.org/10.54941/ahfe1006502
Arslan, S. (2021). Examination of a computer-delivered English proficiency test for flight attendants: Test-takers’ experiences. Journal of Educational Technology and Online Learning, 4(4), 673–687. https://doi.org/10.31681/jetol.1003051
Bystrova, B., Nemliy, L., Paziura, N., & Vasiukovych, O. (2019). Problem-based ESP methods for teaching future air traffic controllers to conduct radio exchange in non-routine situations. Advanced Education, 6(12), 74–79. https://doi.org/10.20535/2410-8286.155041
Clark, L., & Williams, G. (2020). English language proficiency in radiotelephony: A survey about its effect on the safety and efficiency of aviation. The Especialist, 41(4). https://doi.org/10.23925/2318-7115.2020v41i4a9
Coertze, S., Conradie, S., Burger, C., & Huddlestone, K. (2014). Aviation English in South African airspace. Stellenbosch Papers in Linguistics Plus, 42, 41–60. https://doi.org/10.5842/42-0-167
Demirdöken, G. (2024). Artificial intelligence in aviation English testing. The Literacy Trek, 10(3), 362–384. https://doi.org/10.47216/literacytrek.1556603
Dewi, T., & Ohi, S. (2023). Investigating English for aviation: An in-depth analysis of aviation English textbooks. Journal of Teaching English for Specific and Academic Purposes, 087. https://doi.org/10.22190/JTESAP230301008D
Farris, C., Trofimovich, P., Segalowitz, N., & Gatbonton, E. (2008). Air traffic communication in a second language: Implications of cognitive factors for training and assessment. TESOL Quarterly, 42(3), 397–410. https://doi.org/10.1002/j.1545-7249.2008.tb00138.x
Fatmawati, F., Rochmawati, L., & S, M. (2023). TOEIC-readiness and TOEIC mock tests: Measuring Civil Aviation Polytechnic of Surabaya cadets’ preparedness for TOEIC test. Jurnal Penelitian, 7(4), 312–322. https://doi.org/10.46491/jp.v7i4.1187
Fauzi, I. (2020). EFL students’ perception on TOEIC practice class at twelfth graders of SMK Kesehatan Husada Pratama. Globish: An English-Indonesian Journal for English Education and Culture, 9(2), 69–78. https://doi.org/10.31000/globish.v9i2.2332
Garcia, A., & Fox, J. (2020). Contexts and constructs: Implications for the testing of listening in pilots’ communication with air traffic controllers. The Especialist, 41(4). https://doi.org/10.23925/2318-7115.2020v41i4a4
García, Á. (2023). Investigating the construct of aeronautical English listening testing: A qualitative analysis of the ICAO rating scale. Journal of Teaching English for Specific and Academic Purposes, 069. https://doi.org/10.22190/JTESAP230220007G
Guedes, D., & Coneglian, F. (2019). Tipos de questões para testes de proficiência das habilidades de leitura e escrita de língua inglesa. Interfaces Científicas – Educação, 7(3), 55–66. https://doi.org/10.17564/2316-3828.2019v7n3p55-66
Hautasaari, A., & Yamashita, N. (2015). Emotion detection in non-native English speakers’ text-only messages by native and non-native speakers. In Human-computer interaction (pp. 183–200). Springer. https://doi.org/10.1007/978-3-319-22701-6_13
Huang, H. (2013). Motivational changes in an English foreign language online reading context. Social Behavior and Personality: An International Journal, 41(5), 715–720. https://doi.org/10.2224/sbp.2013.41.5.715
Kanno, M. (2020). Apathy, agency, and sociocultural mediation in a university English course. JALT PCP, 144–156. https://doi.org/10.37546/jaltpcp2019-18
Kim, H. (2018). What constitutes professional communication in aviation: Is language proficiency enough for testing purposes? Language Testing, 35(3), 403–426. https://doi.org/10.1177/0265532218758127
Kim, H., & Elder, C. (2014). Interrogating the construct of aviation English: Feedback from test takers in Korea. Language Testing, 32(2), 129–149. https://doi.org/10.1177/0265532214544394
Kraśnicka, I. (2016). English with flying colors: Aviation English and the International Civil Aviation Organization. Studies in Logic, Grammar and Rhetoric, 45(1), 111–124. https://doi.org/10.1515/slgr-2016-0019
Masrul, M., & Rasyidah, U. (2023). Washback effect of TOEIC listening and reading as a college exit test in Riau, Indonesia. Journey (Journal of English Language and Pedagogy), 6(3), 698–712. https://doi.org/10.33503/journey.v6i3.3767
Masiulionienė, G., & Tupčiauskaitė, G. (2023). Bridging the gap between ICAO level 4 and 5 in aviation English: Expectations and results. Aviation, 27(3), 197–211. https://doi.org/10.3846/aviation.2023.19763
Mathews, E., Carson, J., Singleton, S., Valdes, E., & Williams, E. (2022). Investigating language factors in aviation accidents. Aviation Psychology and Applied Human Factors, 12(2), 99–108. https://doi.org/10.1027/2192-0923/a000229
Miles, M. B., Huberman, A. M., & Saldaña, J. (2020). Qualitative data analysis: A methods sourcebook (4th ed.). SAGE Publications.
Oleksandrivna, K., & Kharlamova, L. (2019). Implementation of interactive teaching methods in the process of developing professional-communicative proficiency of future pilots. International Journal of Science Annals, 2(1–2), 12–18. https://doi.org/10.26697/ijsa.2019.1-2.02
Park, M. (2018). Innovative assessment of aviation English in a virtual world: Windows into cognitive and metacognitive strategies. ReCALL, 30(2), 196–213. https://doi.org/10.1017/S0958344017000362
Read, J., & Knoch, U. (2009). Clearing the air. Australian Review of Applied Linguistics, 32(3), 21.1–21.11. https://doi.org/10.2104/aral0921
Saputro, I., Nurhidayati, A., Suharno, S., Winiawati, C., Yunika, R., & Kusumadevi, M. (2023). Evaluation of the factory implementation of the CIPP method in Indonesian vocational education. Al-Ishlah: Jurnal Pendidikan, 15(4), 4972–4982. https://doi.org/10.35445/alishlah.v15i4.3903
Setyoko, A., Tunas, B., & Sunaryo, W. (2016). Evaluation of school operational assistance using the CIPP model in Indonesian private Islamic elementary schools. International Journal of Managerial Studies and Research, 4(3). https://doi.org/10.20431/2349-0349.0403007
Susanto, F., & Hamzah, S. (2022). The context, input, process, product (CIPP) evaluation model as a comprehensive framework for evaluating online English learning towards the industrial revolution era 5.0. International Journal of Educational Research & Social Sciences, 3(6), 2235–2247. https://doi.org/10.51601/ijersc.v3i6.544
Torres, J. C. S. (2023). Considerations for aviation-related English teachers’ capacitation: A reflection about training aviation-specialized English instructors. Journal of Teaching English for Specific and Academic Purposes, 11(1), 57–67. https://doi.org/10.22190/JTESAP230230006S
Ulfah, B., & Mukminatien, N. (2023). Bridging language proficiency and scientific knowledge: Integrating government science content into TOEIC-based summative listening tests. Journal of English Language Studies, 8(2), 339–353. https://doi.org/10.30870/jels.v8i2.20014
Vitryak, A., Slipak, B., & Serhii, K. (2016). Plain aviation English: Qualitative and quantitative interpretation. Journal of Social Science Studies, 4(1), 42–54. https://doi.org/10.5296/jsss.v4i1.9478
Wilson, K. (1989). Enhancing the interpretation of a norm-referenced second-language test through criterion-referencing: A research assessment of experience in the TOEIC testing context. ETS Research Report Series, 1989(2). https://doi.org/10.1002/j.2330-8516.1989.tb00153.x
Yin, Y., Yu, G., & He, D. (2023). Implementing the ICAO language proficiency requirements: The localization of aviation English testing. Applied Linguistics, 45(4), 699–717. https://doi.org/10.1093/applin/amad057
Zhao, K., Guo, X., & Gao, X. (2017). Learning English to fly: A study of Chinese cargo airline pilots’ learning engagement. English Today, 33(4), 5–11. https://doi.org/10.1017/S0266078417000219
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Hendra Sudarso, Safitri Era Globalisasi, Dimas Hari Cahyo, Yasyfa’ Kawakibi, Haris Dibdyaningsih
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
License and Publishing Agreement
In submitting the manuscript to the journal, the authors certify that:
- They are authorized by their co-authors to enter into these arrangements.
- The work described has not been formally published before, except in the form of an abstract or as part of a published lecture, review, thesis, or overlay journal.
- That it is not under consideration for publication elsewhere,
- That its publication has been approved by all the author(s) and by the responsible authorities “tacitly or explicitly“ of the institutes where the work has been carried out.
- They secure the right to reproduce any material that has already been published or copyrighted elsewhere.
- They agree to the following license and publishing agreement.
Copyright
Authors who publish with Jo-ELT (Journal of English Language Teaching) Fakultas Pendidikan Bahasa dan Seni Program Studi Pendidikan Bahasa Inggris IKIP agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
Licensing for Data Publication
Jo-ELT (Journal of English Language Teaching) Fakultas Pendidikan Bahasa dan Seni Program Studi Pendidikan Bahasa Inggris IKIP use a variety of waivers and licenses, that are specifically designed for and appropriate for the treatment of data:
- Open Data Commons Attribution License, http://www.opendatacommons.org/licenses/by/1.0/
Other data publishing licenses may be allowed as exceptions (subject to approval by the editor on a case-by-case basis) and should be justified with a written statement from the author, which will be published with the article.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
1.jpg)
