Crowdsourced vs. Traditional Usability Testing: A Comparative Usability Evaluation of Learning Management Systems

Sadiq Khan; Muhammad Nasir; Zakia Jalil; Qaisar Afzal

Authors

Sadiq Khan Faculty of Computing & Information Technology, International Islamic University, Islamabad, 44000, Pakistan.
Muhammad Nasir Faculty of Computing & Information Technology, International Islamic University, Islamabad, 44000, Pakistan. https://orcid.org/0000-0003-0035-192X
Zakia Jalil Faculty of Computing & Information Technology, International Islamic University, Islamabad, 44000, Pakistan. https://orcid.org/0000-0002-4992-4189
Qaisar Afzal Faculty of Computing & Information Technology, International Islamic University, Islamabad, 44000, Pakistan.

Keywords:

Usability, User Experience, User Testing, Crowd Sourced Usability , Learning Management Systems

Abstract

Introduction/Importance of Study: Usability and User Experience (UX) evaluation is essential for improving Learning Management Systems (LMS) because these platforms directly influence students’ learning effectiveness and satisfaction.

Novelty statement: This study uniquely compares traditional user testing with a crowdsourced usability testing method on prominent global LMS platforms (Google Classroom, Canvas, and Edredo), addressing a critical research gap in large-scale benchmarking and resource-efficiency.

Material and Method: An experimental research design involved 15 traditional and 15 crowdsourced testers who performed identical tasks. Sessions were recorded via OBS software and analyzed using the Molich and Dumas coding scheme. To determine statistical significance between the two methods, the Mann-Whitney U test was employed to evaluate effectiveness (issue counts), efficiency (time), and cost-effectiveness.

Result and Discussion: The evaluation identified a total of 310 usability issues. The traditional method discovered 170 (55%), while the crowdsourced method identified 140 (45%). In terms of effectiveness parity, the traditional method detected 48 key issues compared to 45 in the crowdsourced approach (p=0.23014), proving no significant difference in identifying critical failures. However, crowdsourced testing demonstrated extreme economic efficiency, reducing costs by 6.4x—expending $150 compared to $960 for traditional testing (p=0.00001). Regarding time efficiency, crowdsourced sessions were completed in 4.88 hours compared to 6.07 hours for traditional methods (p=0.11123), indicating comparable operational speeds.

Concluding Remarks: Crowdsourced usability testing provides a statistically validated alternative to traditional methods, achieving comparable issue detection effectiveness while significantly reducing resource expenditure.

References

S. Lehong, J. van Biljon, and I. Sanders, “Usability Requirements for Learning Management Systems in Open Distance Electronic Learning Environments: Considering Lecturers’ Views on Students’ Needs,” Int. J. Human–Computer Interact., vol. 40, no. 3, pp. 567–583, 2024, doi: 10.1080/10447318.2022.2121205.

Farhan Hanis Muhmad Asri, Dalbir Singh, “A Review of Cross-Cultural Design to Improve User Engagement for Learning Management System,” KSII Trans. Internet Inf. Syst., vol. 18, no. 2, 2024, doi: 10.3837/tiis.2024.02.007.

Abdulaziz Aldiab, Harun Chowdhury, “Utilization of Learning Management Systems (LMSs) in higher education system: A case review for Saudi Arabia,” Energy Procedia, vol. 160, pp. 731–737, 2019, doi: https://doi.org/10.1016/j.egypro.2019.02.186.

N. Adzharuddin, “Learning Management System (LMS) among University Students: Does It Work?,” Int. J. e-Education, e-Business, e-Management e-Learning, 2013, doi: 10.7763/IJEEEE.2013.V3.233.

D. G. S. Júnior, R. Hernández-Ramírez, and J. Estima, “Systematic Mapping of Methods Used to Evaluate the Usability and UX of Learning Management Systems,” Springer Ser. Des. Innov., vol. 19, pp. 122–133, 2022, doi: 10.1007/978-3-030-89735-2_11/SAVE-RESEARCH.

Adolph Sedem Yaw Adu, Judy Van Biljon, Ronell van der Merwe, “The Influence of Perceived Usability on Learning Management System Acceptance of Lecturers in Ghana: A PLS-SEM Disjoint Two-Stage Approach,” Adv. Human-Computer Interact., 2025, [Online]. Available: https://onlinelibrary.wiley.com/doi/10.1155/ahci/9694629

A. Stockless, “Acceptance of learning management system: The case of secondary school teachers,” Educ. Inf. Technol. 2017 233, vol. 23, no. 3, pp. 1101–1121, Oct. 2017, doi: 10.1007/S10639-017-9654-6.

Khaleel Mershad, Pilar Wakim, “A Learning Management System Enhanced with Internet of Things Applications,” J. Educ. Learn., vol. 7, no. 3, p. 23, 2018, doi: 10.5539/jel.v7n3p23.

Reem Sualiman Baragash, Hosam Al-Samarraie, “Blended learning: Investigating the influence of engagement in multiple learning delivery modes on students’ performance,” Telemat. Informatics, vol. 35, no. 7, 2018, [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0736585318303848

T. G. Sakova and S. A. Chevereva, “Fundamentals of Choosing an LMS Platform for Distance Learning,” Lect. Notes Networks Syst., vol. 161 LNNS, pp. 348–353, 2021, doi: 10.1007/978-3-030-60926-9_45.

N. W. E. A. I Made Ardi Sudestra, “Improving Digital Learning: Evaluating The U Learn LMS with the System Usability Scale,” JIPI (Jurnal Ilm. Penelit. dan Pembelajaran Inform., vol. 94, pp. 2325–2332, 2024, doi: 10.29100/jipi.v9i4.6910.

M. Beasley, “Practical Web Analytics for User Experience: How Analytics Can Help You Understand Your Users,” Elsevier Sci. Inc, 2013, [Online]. Available: https://www.sciencedirect.com/book/monograph/9780124046191/practical-web-analytics-for-user-experience

“The Definition of User Experience (UX) - NN/G.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.nngroup.com/articles/definition-user-experience/

A. Abran, A. Khelifi, W. Suryn, and A. Seffah, “Usability meanings and interpretations in ISO standards,” Softw. Qual. J., vol. 11, no. 4, pp. 325–338, 2003, doi: 10.1023/A:1025869312943/METRICS.

“Usability Inspection Methods: Book by Jakob Nielsen - NN/G.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.nngroup.com/books/usability-inspection-methods/

S. Rosenbaum, “Usability Evaluations Versus Usability Testing: When and Why?,” IEEE Trans. Prof. Commun., vol. 32, no. 4, pp. 210–216, 1989, doi: 10.1109/47.44533.

“Usability engineering at a discount | Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems (2nd ed.).” Accessed: Jun. 09, 2026. [Online]. Available: https://dl.acm.org/doi/10.5555/92449.92499

Di Liu, Randolph G. Bias, Matthew Lease, Rebecca Kuipers, “Crowdsourcing for usability testing,” Proc. ASIST Annu. Meet., 2013, [Online]. Available: https://asistdl.onlinelibrary.wiley.com/doi/10.1002/meet.14504901100

Anders Bruun, Jan Stage, “New approaches to usability evaluation in software development: Barefoot and crowdsourcing,” J. Syst. Softw., vol. 105, pp. 40–53, 2015, doi: https://doi.org/10.1016/j.jss.2015.03.043.

Muhammad Nasir, Naveed Ikram, “Usability inspection: Novice crowd inspectors versus expert,” J. Syst. Softw., vol. 183, p. 111122, 2022, doi: https://doi.org/10.1016/j.jss.2021.111122.

M. Nasir, D. N. Ikram, D. Z. Jalil, W. Muhammad, and T. Mikkonen, “A Framework (Crowduix) for Novice Usability Inspection Using Crowdsourcing: A Multi-Case Study,” 2025, doi: 10.2139/SSRN.5369276.

“Development and validation of a virtual learning management system usability questionnaire: A case study - PubMed.” Accessed: Jun. 09, 2026. [Online]. Available: https://pubmed.ncbi.nlm.nih.gov/39177635/

Patricia D. Simon, Juming Jiang, Luke K. Fryer, Ronnel B. King & Cherry E. Frondozo, “An Assessment of Learning Management System Use in Higher Education: Perspectives from a Comprehensive Sample of Teachers and Students,” Technol. Knowl. Learn., vol. 30, 2025, [Online]. Available: https://link.springer.com/article/10.1007/s10758-024-09734-5

Javier Mella-Norambuena, Andres Chiappe & Maria Graciela Badilla-Quintana, “Theoretical and empirical models underlying the teaching use of LMS platforms in higher education: a systematic review,” J. Comput. Educ., vol. 12, pp. 997–1023, 2025, [Online]. Available: https://link.springer.com/article/10.1007/s40692-024-00336-9

A. Ertürk and M. B. K. Önaçan, “Usability Evaluation of Learning Management System in A Higher Education Institution: A Scale Development Study,” J. Glob. Strateg. Manag., vol. 2, no. 10, pp. 73–73, Dec. 2016, doi: 10.20460/JGSM.20161024357.

Ilia Maslov, Shahrokh Nikou, “Exploring user experience of learning management system,” Int. J. Inf. Learn. Technol., vol. 38, no. 4, 2021, doi: 10.1108/IJILT-03-2021-0046.

S. M. Jafari, S. F. Salem, M. S. Moaddab, and S. O. Salem, “Learning Management System (LMS) success: An investigation among the university students,” 2015 IEEE Conf. e-Learning, e-Management e-Services, IC3e 2015, pp. 64–69, Feb. 2016, doi: 10.1109/IC3E.2015.7403488.

Dimah Al-Fraihat, Mike Joy, “Evaluating E-learning systems success: An empirical study,” Comput. Human Behav., vol. 102, pp. 67–86, 2020, [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0747563219302912

“(PDF) Usability Evaluation of Learning Management Systems in Sri Lankan Universities.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.researchgate.net/publication/272747770_Usability_Evaluation_of_Learning_Management_Systems_in_Sri_Lankan_Universities

“(PDF) Investigating the Use of Learning Management System (LMS) for Distance Education in Malaysia: A Mixed-Method Approach.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.researchgate.net/publication/352524591_Investigating_the_Use_of_Learning_Management_System_LMS_for_Distance_Education_in_Malaysia_A_Mixed-Method_Approach

“(PDF) Evaluating The Usability And Accessibility Of LMS ‘Blackboard’ At King Saud University.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.researchgate.net/publication/298332435_Evaluating_The_Usability_And_Accessibility_Of_LMS_Blackboard_At_King_Saud_University

R. J. M. Ventayen, K. L. A. Estira, M. Guzman, C. M. M. Cabaluna, and N. N. Espinosa, “Usability Evaluation of Google Classroom : Basis for the Adaptation of GSuite E-Learning Platform,” 2017.

F. Guaiani and H. Muccini, “Crowd and Laboratory Testing, Can They Co-exist? An Exploratory Study,” Proc. - 2nd Int. Work. CrowdSourcing Softw. Eng. CSI-SE 2015, pp. 32–37, Jul. 2015, doi: 10.1109/CSI-SE.2015.14.

J. Scholtz, “Adaptation of traditional usability testing methods for remote testing,” Proc. Hawaii Int. Conf. Syst. Sci., p. 134, 2001, doi: 10.1109/HICSS.2001.926546.

“Why You Only Need to Test with 5 Users - NN/G.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

“(PDF) Evaluating LMS Usability for Enhanced eLearning Experience.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.researchgate.net/publication/252615996_Evaluating_LMS_Usability_for_Enhanced_eLearning_Experience

“The LMS moodle: A Usability Evaluation.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.researchgate.net/publication/241339520_The_LMS_moodle_A_Usability_Evaluation

Nhu Ty Nguyen, “A study on satisfaction of users towards learning management system at International University – Vietnam National University HCMC,” Asia Pacific Manag. Rev., vol. 26, no. 4, pp. 186–196, 2021, doi: https://doi.org/10.1016/j.apmrv.2021.02.001.

“Evaluating Learning Management System Usage at a Small University | Proceedings of the 2019 3rd International Conference on Information System and Data Mining.” Accessed: Jun. 09, 2026. [Online]. Available: https://dl.acm.org/doi/10.1145/3325917.3325929

“(PDF) Evaluating the Usability of an E-Learning Platform within Higher Education from a Student Perspective.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.researchgate.net/publication/338619470_Evaluating_the_Usability_of_an_E-Learning_Platform_within_Higher_Education_from_a_Student_Perspective

W. T. Nakamura, E. H. T. De Oliveira, and T. Conte, “Usability and user experience evaluation of learning management systems a systematic mapping study,” ICEIS 2017 - Proc. 19th Int. Conf. Enterp. Inf. Syst., vol. 3, pp. 97–108, 2017, doi: 10.5220/0006363100970108.

M. Borys and M. Laskowski, “Expert vs novice evaluators: Comparison of heuristic evaluation assessment,” ICEIS 2014 - Proc. 16th Int. Conf. Enterp. Inf. Syst., vol. 3, pp. 144–149, 2014, doi: 10.5220/0004970901440149.

Y. (Chris) Zhao, J. Tang, X. Ye, and J. Lian, “Crowdsourcing in UX Research,” Human-Computer Interact., p. vol3:266-vol3:284, Sep. 2024, doi: 10.1201/9781003584292-41.

F. Anggriantaka and D. S. Kusumo, “The Development of an Asynchronous Crowdsourcing-Based User Acceptance Testing Framework and Its Comparison with Conventional Synchronous UAT,” pp. 1–7, Dec. 2025, doi: 10.1109/ICOSEIT67010.2025.11290993.

T. Kim and Y. M. Choi, “A Comparison of Laboratory and Synchronous Remote Usability Testing Methods Using AR,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 13302 LNCS, pp. 252–263, 2022, doi: 10.1007/978-3-031-05311-5_17/SAVE-RESEARCH.

G. Desolda, R. Lanzilotti, D. Caivano, M. F. Costabile, and P. Buono, “Asynchronous Remote Usability Tests Using Web-Based Tools Versus Laboratory Usability Tests: An Experimental Study,” IEEE Trans. Human-Machine Syst., vol. 53, no. 4, pp. 731–742, Aug. 2023, doi: 10.1109/THMS.2023.3282225.

Yonas Leguesse, Mark Vella, “A Metric to Assess the Reliability of Crowd-sourced SUS Scores: A Case Study on the PoPLar Authentication Tool,” ACM Int. Conf. Proceeding Ser., 2024, [Online]. Available: https://dl.acm.org/doi/10.1145/3688459.3688470

Y. Wang, X. Chen, and X. Zhou, “A New Method for Identifying Low-Quality Data in Perceived Usability Crowdsourcing Tests: Differences in Questionnaire Scores,” Int. J. Human–Computer Interact., vol. 40, no. 22, pp. 7297–7313, Nov. 2024, doi: 10.1080/10447318.2023.2263694.

R. Molich and J. S. Dumas, “Comparative usability evaluation (CUE-4),” Behav. Inf. Technol., vol. 27, no. 3, pp. 263–281, May 2008, doi: 10.1080/01449290600959062.

“Classroom Management Tools & Resources - Google for Education.” Accessed: Jun. 09, 2026. [Online]. Available: https://edu.google.com/intl/ALL_pk/workspace-for-education/products/classroom/

“Canvas by Instructure: World Leading LMS for Teaching & Learning.” Accessed: Jun. 09, 2026. [Online]. Available: https://www.instructure.com/canvas

E. Gamboa, R. Galda, C. Mayas, and M. Hirth, “The Crowd Thinks Aloud: Crowdsourcing Usability Testing with the Thinking Aloud Method,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 13094 LNCS, pp. 24–39, 2021, doi: 10.1007/978-3-030-90238-4_3/SAVE-RESEARCH.