External evaluation 2022 report brief

External evaluation carried out by: (View full report)

In 2021 and 2022, distance learning activities continued to play a very important role in the service portfolio of the Centre and quality-assuring these distance learning activities is of paramount importance for the sustainability of the organization. The 2022 external evaluation of the Centre therefore focused again on the online learning activities of the Centre.  

This re-evaluation exclusively focused on training activities that were fully delivered in an online format and covered a sample of twenty training activities offered in 2021. In relation to the previous evaluation in 2021, the re-evaluation additionally incorporated a comparative perceptive, a technological focus, and specific instructional design elements.

The purpose of the re-evaluation was to provide the leadership and management of the Centre with evidence of the relevance, validity of design, effectiveness, efficiency, impact and sustainability of its fully online training activities, to assess which modalities of online training are most effective and efficient, to explore good practices, lessons learned, and to derive recommendations for the improvement and further development of the ITCILO's online training activities. 

The evaluation was undertaken using a mixed-methods approach. A desk review of available data and reports, including systematic analysis of the instructional design of the sampled courses, was initially conducted. Quantitative data was then collected using a survey with a sample of 792 participants from 130 countries. Finally, qualitative evaluation methods were employed, including semi-structured interviews with the Centre's staff involved in the design and delivery of the twenty online training activities, semi-structured interviews with institutional partners, focus group discussions with former participants, and impact case studies development. 

To evaluate the participants’ learning experiences in ITCILO’s online training activities, an instrument was used to measure the three dimensions of the Community of Inquiry (CoI) framework, i.e. teaching presence, social presence, and cognitive presence.

3.1 Relevance and outreach

• The Centre has successfully served its target groups during 2021. 

• the Centre has effectively played its role in providing ILO constituents with specialised training on different aspects of the Decent Work Agenda via offering online training activities. 

• There was no further evidence that the Centre’s online training activities have failed to reach out to its target groups or provide the training demanded by its beneficiaries, partners, and donors.

• The Centre played a critical role during the Covid-19 pandemic in helping partner organizations and participants across the globe cope with the rapid changes and associated challenges, where each unit, based on its long-established knowledge and teaching expertise, promptly developed online training activities relevant to the pandemic situations, which was perceived as valuable and critical by the participant learners.

• Due to the growth of online and distance learning in ITCILO'S training portfolio, the Centre can reach more than 50,000 learners per year—twice the number achieved before the 2020 COVID-19 pandemic.

• Face-to-face training activities have increased again, but on a lower level as compared to the baseline of 2018-19 biennium.  However, distance learning courses remain by far the most frequent modality in 2021.  

• With online training, a wider and more diversified audience can be reached. Especially participants from low and middle-income countries.

• Based on their experiences with online learning, the participants were asked what kind of format they would prefer in the future, choosing between three different modes of delivery. almost 70 % of the participants continue to prefer a digital format of some kind in the future. 

• As compared to the previous year's evaluation, the blended learning modality remains the first choice of participants, slightly increasing on the same level (from 39.6 to 40.9 %), while the demand for fully online learning decreased (from 35.7 to 27.2 %), and the desire for face-to-face training increased (from 28.8 to 32.0 %). 

• The results of preferences per region are congruent with those of the previous year. The differences between countries and regions might be associated with issues related to technical infrastructure, Internet connectivity, and access to digital devices.

3.2 Validity of the instructional design

• Overall, the participants of the twenty selected courses seem to be well equipped with technical devices and tools to access ITCILO’s online courses.

• In contrast to the previous evaluation, the participants mentioned no difficulties in accessing the online courses via mobile devices apart from general connectivity issues that are not related to ITCILO's e-campus services.

• 93.2% of the participants reported being able to freely choose and use different devices (PCs, laptops, mobile phones, tablets) to pursue online learning.

• Access to and navigation in the online learning system, e-Campus, is not an issue, with average ratings of above 4.0. 

• Technical issues in participating in online training activities on a regular basis remain a problem. Major problems were reported by participants from African countries. 

• As in the previous evaluation, the mean scores for the items related to technical support and guidance are both below four. Thus, there is still room for improvement regarding the information for participants on where to find help and the response time of technical support. 

• However, the Centre has added more explicit technical support to e-Campus users. A separate tutorial on how to navigate e-Campus is being offered to new users at their first log-in occasion; more positively, half of the reviewed activities offered an additional introduction to their own e-Campus sites, and in many cases, the activity tutors recorded a short introductory video

• A robust Internet connection is a prerequisite to participating in online learning. However, the quality and reliability of Internet connectivity vary across countries and regions.

• In areas with low bandwidth and unstable connections, asynchronous communication and content delivery tools and media are preferable because participants can log in, communicate and download learning material at a convenient time when the Internet is available. In contrast, synchronous video-conferencing requires much more bandwidth and a stable connection.

• The digital media and tools used for synchronous and asynchronous interaction in the various online learning formats play an important role in improving the accessibility and scalability of online learning programmes. When participants were asked if asynchronous computer-conferencing, asynchronous video content as compared to synchronous video-conferencing were used too often, just enough, or not often enough, the scores indicated that the frequency of use of asynchronous and synchronous tools was just right on average.  

• Synchronous video-conferencing requires higher bandwidth and a good Internet connection. Many participants from Africa and the Middle East—the regions with the slowest Internet connectivity—said that they spent too much time in synchronous video-conferencing. 

• The three dimensions of the Community of Inquiry (CoI) framework were measured using a 5-point scale. The results indicated that ITCILO’s course designers and facilitators managed to deliver highly engaging, interactive, and supportive online courses that provided opportunities for rich and deep learning experiences.

• Among the twenty reviewed courses, seven were effectively designed to provide a good and reasonable sense of teaching presence to their participants, with additional two activities in an excellent sense. 

• All reviewed activities provided regular or frequent webinars during which most of the direct teaching and instruction was delivered to the participants. 

• Six courses provided participants with a welcome message. Half of the courses provided a separate introduction. Eleven courses integrated a Q&A forum or specific contact information for technical support. Two activities provided participants with an opportunity to directly interact with their tutors.

• Nineteen activities included synchronous teaching elements, through which participants could meet their tutors and experience direct teaching. Thirteen among those provided recordings of the webinars.

• Among the twenty reviewed courses, seven  were effectively designed to provide a good and reasonable sense of social presence to their participants, with additional two activities in an excellent sense. 

• Most activities provided regular or frequent webinars during which participants were likely to see other participants.

• Among the twenty courses, more than half provided participants with an opportunity to formally meet their peers and introduce themselves to other participants, with a Google map feature indicating participants’ geographical locations most frequently employed. Only six activities offered collaborative learning opportunities. Not all of the group works were, however, well-facilitated. 

• Eleven activities integrated a discussion forum. Most of those forums without guided discussion topics were not actively utilized by participants.

• Among the twenty reviewed courses, six activities were effectively designed to provide an excellent sense of cognitive presence to their participants, with additional eight activities in a good and reasonable sense. We could not find much evidence for a good sense of cognitive presence in five activities. Two of those five activities seem to have provided cognitive overload or burden to their participants. One activity has an underdeveloped e-Campus site, which did not provide adequate information to evaluate its participants' potentially perceived cognitive presence.  

• Among the twenty reviewed training activities, three lasted less than a week, one for two weeks, and two for three weeks. The rest lasted longer than a month. The expected learning hours varied among those activities ranging from three to 20 hours a week with a medium of five hours a week. Given that most training participants have full-time jobs and other social commitments, it seems reasonable to expect them to dedicate no more than 10 hours a week to the training. Four activities did not indicate the expected learning hours which would not help participants develop a good sense of cognitive presence. On the other hand, several activities indicated the expected learning hours too specifically.

• Two among the twenty reviewed courses seem to have an exceptionally heavy workload that students are likely to spend far more time than the indicated learning hours.

• Most reviewed courses had a clear division of the activity with a number of sub-units consistently and regularly occurring daily and weekly. The rest have irregular sub-unit distribution across the activity period.

• Nine activities provided a pre-activity assessment in different formats; among the rest, seven activities have a duration longer than four weeks; thus, it could have been useful to provide such a feature for participants to develop a stronger sense of their cognitive and intellectual development throughout the activity period. On the other hand, most activities provided a post-activity assessment.  

• While most of the reviewed activities provided a clear timetable, only six provided a comprehensive handbook to their participants.

• Nine activities provided additional resources; among them, two provided additional courses that participants could take which is particularly recommendable. Another two among the nine, however, seem to have provided too much additional information, which can decrease a perceived sense of cognitive presence among participants.  

• The Centre has increased the design and structural consistency across online training activities offered on e-Campus, which could help learners more conveniently and easily navigate different course sites, potentially reducing cognitive overload.

3.3 Effectiveness

• The reviewed courses effectively achieved their immediate objectives. The learner survey results (as well as course evaluation results) suggest that many have found their engagement with the activities beneficial, contributing to their professional practice and development. The simple formula, good objectives lead to good outcomes, was observed in this review process.  

• The average participant evaluation results of seventeen online courses were 4.3 out of 5.

• The participant survey outcome demonstrated that participants made good progress in their knowledge and skills development during the course period. 

• The overall quality of the online training activities has been evaluated as 4.12, with 78% of the participants having responded either “very good” or “good”. The effectiveness of the training format has also been evaluated relatively positively at 4.01, with 72% of the participants have chosen either “very good” or “good.” 

• Followed by the last year’s impressive result (98.3%), 97.4% of the participants have responded that they would recommend the concerned training activities to their colleagues.  

• More than half of the survey participants have given a concrete example to articulate how participating in the training activities has been of practical use for achieving results in their work.

• Two Focus Groups with six learners also supported the positive evaluation in this regard. Learner perspective on the effectiveness of their online training activities was highly positive. 

• Compared to the enormous sense of uncertainty and serendipity that had emerged during our last year’s staff interviews, staff’s descriptions of their activity design and the rationale for such design were very clear, confident, and in most cases, convincing. 

• It is clear that the Centre has successfully achieved the important milestone in terms of its online training activities as the staff seem to be in general agreement that online training is an essential part of the Centre’s business. 

• Staff members seem to establish a good understanding of the advantages of blended training approaches not only in terms of blending both face-to-face and online activities but blending synchronous and asynchronous activities. 

• There is a clear desire among some staff members to have a “systematic” mechanism for them to receive feedback on how they are doing and “personalized” support for improving their online teaching practices.

• While the increased diversity among activity participants is worthwhile to note and celebrate, it also suggests the increased diversity among the needs of the participants, posing subsequent challenges to the activity managers and tutors. 

• The technological expansion achieved by the Centre within such a short period is positively noted. Also, the way in which the VR technology was employed in this year’s reviewed training activities is certainly much more advanced and sophisticated. 

• Most reviewed training activities employed basic accessibility principles. However, the accessibility issues still need to be discussed more in a practical, mundane, and specific sense.

• Learner data analytics is under fast development as one of the Centre’s strategic priorities for the coming years. Such future directions ambitiously embrace the possibilities of analyzing a more extensive learner data set collected from different platforms using multiple tools. 

3.4 Efficiency

• All twenty reviewed online training activities created revenues, covering direct costs. 

• Similar to the previous evaluation results, the Centre’s online training activities are highly efficient based on a relatively simple inputs-and-outputs formula. 

• The Centre has effectively re-distributed and circulated some of its revenues in advancing its technological infrastructure required for developing and delivering quality online training activities. Most of the financial investment in online training is typically made up-front as one-time fixed expenses. once the necessary technological infrastructure and operational mechanisms are effectively set up, there tends to be a significant decrease in the on-going expenses of online training activities. 

• The positive outcome of the Centre’s ongoing investments from the staff development fund to improve the staff’s use of e-Campus is well-noted. The Centre is continuously improving the pedagogical functions of its e-Campus, by inserting various technical tools and applications. Such investment can be generally considered efficient. 

• The Centre has also put a significant level of resources into a series of innovations for its digital infrastructure and organizational structure to better support its staff and (partner organizations) to improve the quality of online training provision. 

• A growing number of online training activities with advanced VR/AR technology being effectively integrated or fully focused on also suggests the successful outcome of the Centre’s investment in digital innovation.  

• In 2021, the Centre continuously developed its “physical space dedicated to learning innovation”, which includes a range of advanced technological tools on the Turin campus, executing the Centre-wide Learning Innovation Action Plan 2018-21. 

• It is rather evident that the Centre has successfully developed its reputation, playing important roles in achieving digital transformations nationally and internationally—with an Innovation Lab currently under development.

• The Centre has also made major investments in its organizational structure to better support online training provisions. 

• Staff knowledge and expertise have noticeably increased in 2021, which can also be seen as evidence of the efficiency of the Centre’s overall financial operation regarding staff development. However, a majority of staff members preferred having some direct feedback on their teaching practices to participating in official and one-off staff development sessions.

• It is not necessarily true that online training is cheaper than face-to-face training if human labour is calculated as the main “cost” of effective delivery of online training activities. It is unclear from the review how much resources have efficiently been invested in such staff organization issues.  

3.5 Impact

• The vast majority of survey participants agreed or strongly agreed (90.4 %) that the courses they attended provided many examples that translated theory into practice.

• A larger proportion of the participants agreed or strongly agreed that they could apply the knowledge created in this course to their work settings (94.1 %).

• 52.2 % of the survey respondents provided concrete examples of their application of knowledge after online training.

• Participants reported that they made large or very large improvements in terms of their competencies (68.2 %) and job performance (60.8 %) as a result of the training activities.

• Based on the interviews and focus group discussions, three case studies that effectively and vividly capture the positive impacts of the Centre's online training activities were written. 

The conclusions and recommendations are drawn along the lines of the five course evaluation criteria (relevance, outreach, validity of instructional design, effectiveness, efficiency, and impact) and the three performance dimensions defined in ITCILO’s strategic plan for 2022-25 (technical performance, financial performance, and institutional performance).

View ITCILO's management response