Mode of Delivery: Paper-pen versus Mobile-Based Testing. Are they Comparable?
Article Sidebar
Main Article Content
Abstract
The ubiquity of mobile technologies in education has brought concerns regarding comparability between paper-based (PBT) and mobile-based testing (MBT) formats. This study examined: (1) the effects of test mode on student performance and (2) the score equivalence obtained via paper-based and mobile-based testing. Participants (126) from a university in the south of Thailand completed two quantitative within-subject counterbalanced tests administered via PBT and MBT through the participants’ personal smartphones with a two-week interval between administrations. Descriptive statistics, paired-samples t-tests, Pearson product-moment correlation, and Spearman rank-order correlation were used to analyze the data. Findings showed a statistically significant difference between modes favoring mobile testing indicating a strong mode effect. Moreover, correlation analyses revealed negligible relationships between the two modes, suggesting weak score equivalence and limited rank-order consistency. This research contributes to the growing literature on assessment formats by providing evidence on mode effect and interchangeability. The study provides evidence that MBT may influence student performance and score interpretation; and underscore the necessity for careful validation.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
REFERENCES
Abdillah, L.A. (2016). Ujian Online Mahasiswa Ilmu Komputer Berbasis Smartphone. ArXiv, abs/1701.06428.
Bergstrom, B. (1992). Ability measure equivalence of computer adaptive and pencil and paper tests: A research synthesis. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA. Retrieved from https://eric.ed.gov/?id=ED377228
Boo, J., Vispoel, W. (2012). Computer versus Paper-and-Pencil Assessment of Educational Development: A Comparison of Psychometric Features and Examinee Preferences. Psychological reports. 111. 443-60. 10.2466/10.03.11.PR0.111.5.443-460.
Bugbee, A. C. (1996). The equivalence of paper-and-pencil and computer-based testing. Journal of Research on Computing in Education, 28, 282-299. Retrieved from Education Research Complete, Accession Number: 9605221096
Chapelle, C. (2022). Argument-Based Validation in Testing and Assessment. Sage Research Methods. DOI: https://doi.org/10.4135/9781071878811
Chin, C. H. L., & Donn, J. S. (1991). Effects of computer-based tests on the achievement, anxiety, and attitudes of grade 10 science students. Educational and Psychological Measurement, 51, 735. doi: 10.1177/0013164491513025
Choi, S. W., & Tinkler, T. (2002). Evaluating comparability of paper-and-pencil and computer based assessment in a K-12 setting. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA. 148 Coiro, J. (2007). Exploring changes to reading comprehension on the Internet: Paradoxes and possibilities for diverse adolescent readers. (Doctoral dissertation). Retrieved from http://ctell1.uconn.edu/coiro/coirospencer.pdf
Dorans, N. J., Pommerich, M., & Holland, P. W. (Eds.). (2007). Linking and aligning scores and scales. Springer Science + Business Media. https://doi.org/10.1007/978-0-387-49771-6
Clark-Gareca, Beth. (2010). Language Assessment in Practice. Working Papers in Applied Linguistics and TESOL. 10. 10.7916/D8CV4HB8.
Evans, L. D., Tannehill, R., & Martin, S. (1995). Children’s reading skills: A comparison of traditional and computerized assessment. Behavior Research Methods, Instruments, & Computers, 27(2), 162-165. doi: 10.3758/BF03204723
International Telecommunication Union (2022). E-learning in Thailand: Mapping the digital divide. Retrieved from https://www.itu.int/hub/publication/d-pref-them-28-2022/
Goodwin, L. D., & Goodwin, W. L. (1999). Measurement myths and misconceptions. School Psychology Quarterly, 14(4), 408–427. https://doi.org/10.1037/h0089017
Gordon, A. (2015). "Paper Based Testing vs. Mobile Device Based Testing in an EFL Environment: What’s the Difference?". Culminating Projects in English. 38. https://repository.stcloudstate.edu/engl_etds/38
Huynh Thi Trang, Tran Kha Han, Phuong Thi Cam Loan. (2022). Cheating Behaviours in Online Exams, International Journal of Science and Management Studies (IJSMS), v5(i6), 32-37. DOI: 10.51386/25815946/ijsms-v5i6p103
Mazzeo, J., & Harvey, A. L. (1988). The equivalence of scores from automated and conventional educa-tional and psychological tests: A review of the literature (College Board Rep. No. 88-8, ETS RRNo. 88-21). Princeton, NJ: Educational Testing Service. https://onlinelibrary.wiley.com/doi/epdf/10.1002/j.2330-8516.1988.tb00277.x
Mead, A. D., & Drasgow, F. (1993). Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis . Psychological Bulletin , 114 (3), 449–458. https://doi.org/10.1037/0033-2909.114.3.449.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-104). New York, NY: American Council on Education and Macmillan.
Naznean, A. (2021). Cheating during Online Examinations– Literature Review. Journal of Pedagogy. LXVIII(2):7-21 DOI:10.26755/RevPed/2021.2/7
Nikou, S., Economides, A.. (2013). Student achievement in paper, computer/web and mobile based assessment. CEUR Workshop Proceedings. 1036.
Nikou, S., Economides, A. (2018). Mobile-based assessment: A literature review of publications in major referred journals from 2009 to 2018. Computers & Education, vol. 125, 2018, Pages 101-119, ISSN 0360-1315, https://doi.org/10.1016/j.compedu.2018.06.006.
Pommerich, M., & Burden, T. (2000). From simulation to application: Examinees react to computerized testing. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA. https://eric.ed.gov/?id=ED442813
Wang, S., Jiao, H., Young, M. J., Brooks, T., & Olson, J. (2008). Comparability of computer-based and paper-and-pencil testing in K-12 reading assessments: A meta-analysis of testing mode effects. Educational and Psychological Measurement, 68(1), 5-24.
Whattananarong, K. (2004). An experiment in the use of mobile phones for testing at King Mongkut’s Institute of Technology North Bangkok, Thailand. In Proc. International Conference on Making Education Reform Happen: Learning from the Asian Experience & Comparative Perspectives, Bangkok, Thailand.
Yu, W., Iwashita, N.(2021). Comparison of test performance on paper-based testing (PBT) and computer-based testing (CBT) by English-majored undergraduate students in China. Lang Test Asia 11, 32. https://doi.org/10.1186/s40468-021-00147-0
