In 2021 and 2022, distance learning activities continued to play a very important role in the service portfolio of the Centre and quality-assuring these distance learning activities is of paramount importance for the sustainability of the organization. The 2022 external evaluation of the Centre therefore focused again on the online learning activities of the Centre.
This re-evaluation exclusively focused on training activities that were fully delivered in an online format and covered a sample of twenty training activities offered in 2021. In relation to the previous evaluation in 2021, the re-evaluation additionally incorporated a comparative perceptive, a technological focus, and specific instructional design elements.
The purpose of the re-evaluation was to provide the leadership and management of the Centre with evidence of the relevance, validity of design, effectiveness, efficiency, impact and sustainability of its fully online training activities, to assess which modalities of online training are most effective and efficient, to explore good practices, lessons learned, and to derive recommendations for the improvement and further development of the ITCILO's online training activities.
The evaluation was undertaken using a mixed-methods approach. A desk review of available data and reports, including systematic analysis of the instructional design of the sampled courses, was initially conducted. Quantitative data was then collected using a survey with a sample of 792 participants from 130 countries. Finally, qualitative evaluation methods were employed, including semi-structured interviews with the Centre's staff involved in the design and delivery of the twenty online training activities, semi-structured interviews with institutional partners, focus group discussions with former participants, and impact case studies development.
To evaluate the participants’ learning experiences in ITCILO’s online training activities, an instrument was used to measure the three dimensions of the Community of Inquiry (CoI) framework, i.e. teaching presence, social presence, and cognitive presence.
Teaching Presence is the design, facilitation, and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes.
Social presence is the ability of participants to identify with the community, communicate purposefully in a trusting environment, and develop inter-personal relationships by way of projecting their individual personalities.
Cognitive Presence is the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse.
The conclusions and recommendations are drawn along the lines of the five course evaluation criteria (relevance, outreach, validity of instructional design, effectiveness, efficiency, and impact) and the three performance dimensions defined in ITCILO’s strategic plan for 2022-25 (technical performance, financial performance, and institutional performance).
1. It is recommended that ITCILO develop a strategic plan on how to best reach their target groups in different regions with appropriate educational technologies and media to get the right mix of synchronous and asynchronous, blended and fully online distance learning delivery that allows for maximum accessibility and outreach.
2. The Centre should carefully analyze the procedures and data pertaining to technical support.
3. It is recommended to review the expected duration, learning hours and number of required tasks to avoid an overwhelming workload for course participants. A clear timetable should always be provided, and distance learners should be given time to catch up in case of falling behind.
4. It is recommended that all courses include a recorded welcome message to introduce the course tutors and facilitators and provide an introduction and overview of the course content. Communication in asynchronous forums needs to be monitored and moderated by the tutors on a regular basis. Recordings of synchronous sessions should always be provided.
5. It is recommended that collaborative learning opportunities be implemented wherever possible.
6. At least for the open courses, ITCILO should consider publishing learning materials under a Creative Commons license. It is recommended that the Center develops an OER Policy to support the development and use of open content.
7. It is recommended that ITCILO develop a more long-term mechanism to evaluate its financial performance in terms of technological innovations.
8. It is recommended that the Centre review the staff workload involved in online training activities.
9. The Centre can re-think and re-design its staff development mechanism.
10. It is recommended that ITCILO focus on translating the “idea” or “ideal” of digital inclusion into online training practice. To do so, it is necessary to start by developing a solid understanding of specific circumstances and diverse challenges that restrict both the “access” and “success” of participants’ online learning experiences.
11. For educational data mining and profiling, it is essential to avoid privileging dominant participant groups at the expanse of diverse and marginalized participant groups that do not fit the mainstream learner image. We also suggest moving from "learner analytics" to "learning analytics" to develop a deeper understanding of how different learner groups engage with learning activities and interact with other course participants and their tutors.
12. It is still recommended that the Centre develop a coherent training framework taking into account the full spectrum of target groups, content areas, technological tools, and pedagogical methods—including corresponding instructional design templates.