External Evaluation 2025 Report Brief

Photo by Ian Bathan on Unsplash

• This report presents the findings of the external evaluation of the ITCILO's training activities on the thematic area of Social Protection for the year 2024.

• This evaluation forms a crucial part of the Centre's commitment to continuous quality improvement and strategic development, aligning with its 2022-25 Strategic Plan emphasizing a quality-focused, data-driven approach to monitoring and evaluation.

• The ITCILO's capacity development efforts in Social Protection are aimed at empowering ILO constituents to address pressing challenges, and contribute directly to the global agenda for social justice and decent work.

• The evaluation focused on a sample of 20 training activities delivered by the Social Protection, Governance and Tripartism (SPGT) programme in 2024, encompassing a mix of online, face-to-face, and blended courses conducted in the field or at the Turin Campus, ensuring diversity across regions and participants.

• The purpose of this evaluation was to provide the Centre with evidence of the relevance, coherence, validity, effectiveness, efficiency, impact, and sustainability of its 2024 Social Protection training activities. It also aimed to identify which training modalities are more impactful in terms of effectiveness and efficiency, and to extrapolate good practices, lessons learned, and recommendations for improvement or scale-up of relevant training activities.

• The evaluation adopted a mixed-methods design, combining quantitative and qualitative techniques to generate robust, triangulated evidence.

• It was guided by internationally recognised evaluation and quality standards, including the evaluation criteria of the Organisation for Economic Co-operation and Development – Development Assistance Committee (OECD DAC), the United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation, and the International Organisation for Standardisation (ISO) standard 29993:2017 for learning services outside formal education.

• Data collection and analysis methods included a systematic document review, quantitative dataset analysis, key informant interviews with ITCILO staff and partners, focus group discussions with former participants, and five illustrative case studies.

• This section presents the main findings of the evaluation, structured according to the eight evaluation questions derived from the OECD DAC criteria and ITCILO's strategic evaluation framework.

• The evaluation of the ITCILO’s Social Protection training portfolio for 2024 revealed a programme that is strategically relevant, technically robust, and highly valued by its participants and institutional partners. It shows tangible contributions to capacity development at individual, institutional, and, in some cases, policy levels.

EQ1: To what extent are the objectives and design of the ITCILO's Social Protection trainings aligned with the needs of participants, institutional clients, and the strategic priorities of the ITCILO and ILO?

• The evaluation found that the design and objectives of the ITCILO’s Social Protection training activities are well aligned with the strategic priorities of both the ITCILO and the ILO. This alignment is most clearly reflected in the thematic focus of the portfolio, which directly supports the ILO’s 2022–2025 Strategic Plan and its Programme and Budget for 2024–2025.

• Active efforts are made to align the Centre’s biennium planning processes with those of the ILO. This alignment is not only strategic but also operationalised at the portfolio level. Collaboration with strategic partners like the Social Protection Department of the ILO (SOCPRO), and other strategic partners, such as the International Social Security Association (ISSA).ensured policy coherence and technical robustness.

• The evaluation found that the objectives and design of the ITCILO’s Social Protection training portfolio demonstrate a high degree of alignment with the needs and priorities of institutional clients. This alignment was evident in the Centre’s strategic partnerships, planning processes, and course development practices.

• The inconsistency of documenting learning needs assessments, especially in open courses, represented a gap in aligning course design with participant expectations and starting points. 

• Across formats and modalities, participant feedback consistently indicated high level of satisfaction with ITCILO’s Social Protection trainings. Both evaluation survey data and qualitative responses pointed to strong perceived relevance, practical applicability, and overall positive learning experiences. 

• Participants consistently rated courses as highly relevant to their needs, with 93.6% agreeing or strongly agreeing on this point in the online evaluation survey. 

EQ2: To what extent are the Social Protection trainings complementary to other (non-training) ITCILO initiatives supporting social protection, and to what extent do they reinforce the broader mandate of the ILO and its constituents?

• Document review confirmed that ITCILO's Social Protection training portfolio contributes to the ILO's social protection agenda by building institutional and individual capacities, facilitating policy coherence, leveraging partnerships, and driving innovation and knowledge sharing.

• The evaluation found evidence that the Social protection  trainings are closely coordinated with ILO's technical cooperation projects and advisory services.

• While Social Dialogue and Tripartism is strongly mainstreamed, the integration of Gender and International Labour Standards related topics was less prominent, indicating opportunities for enhancement.

• Key informant interviews confirmed that for all social protection training activities, the content and requests were consistently validated with ILO SOCPRO. This process ensures coherence with ILO policy messages, particularly when dealing with sensitive policy issues or new content. From SOCPRO’s perspective, the training offer is well-aligned with their mandate and is considered a useful mechanism to reinforce ILO messages. The integration of policy and training efforts ensures a coherent approach to capacity development.

• ISSA's partnership with ITCILO was considered "highly valuable", with ITCILO being referred to as "the most important partner" for the delivery of ISSA guidance training. All trainings involving ISSA are "100% aligned with their mandate and three-year plan". As such, ISSA played a significant role in defining course content, aligning offerings with its three-year plan and contributing to curricula, objectives, and syllabi, often recommending guideline
  
  authors as trainers to ensure relevance.

EQ3: To what extent are the Social Protection trainings logically designed to achieve their stated objectives, and supported by appropriate tools to monitor learning outcomes and progress?

The Community of Inquiry (COI) framework was assessed through a dedicated online survey administered as part of this evaluation. The COI framework provides a well-established model for understanding and evaluating learning experiences in online and blended environments. Across all ITCILO training modalities, whether in-person, blended, or self-guided online, the COI model received consistent empirical support. Item-level response distributions from the COI survey confirmed the alignment between theoretical constructs and participant perceptions.

Teaching Presence

• Teaching Presence received particularly strong validation. Responses were especially positive across its three subdimensions: design and organization, facilitation, and direct instruction. Items measuring instructor clarity, course structure, and guidance routinely saw over 85% of respondents selecting “Agree” or “Strongly Agree,” with some surpassing 90%.

• Learning design was further supported by strong satisfaction with instructional clarity and relevance. The evaluation survey indicated that 86.6% of participants felt they received timely and effective support from tutors, and 90.2% confirmed they were provided with all necessary learning resources. These findings align closely with data from the broader post-course satisfaction questionnaire.

• This convergence of evidence reinforces the construct validity of teaching presence, as also supported by Arbaugh et al. (2008), who found teaching presence to be a key predictor of both satisfaction and learning outcomes.

Social Presence

• Social Presence demonstrated favorable patterns, though with greater variability across delivery formats. COI items related to affective expression and open communication were rated positively, with over 70% agreement on participants’ ability to express opinions and feel part of a learning community. However, items measuring group cohesion showed a higher frequency of neutral responses or “Not Applicable,” particularly in asynchronous courses.

• Supporting this, results from the post-course satisfaction questionnaire show slightly lower, though still high, means for learning methods (4.19) and materials appropriateness (4.28), which may indicate subtle differences in how interactivity and collaboration are experienced across formats. These nuances are in line with Garrison and Arbaugh (2007), who observed that modality significantly shapes the manifestation of social presence.

• In response to engagement-related questions, 85.3% of participants rated the level of engagement and interaction as either “Very Good” or “Good.” Additionally, over 91% reported that the course offered sufficient opportunities for participation, collaboration, and discussion. When asked whether the delivery method enhanced their understanding and application of the course content, 95.4% responded positively, and 98.2% stated they would recommend the course format to others. These results clearly indicate that interactivity and learner engagement were well integrated across delivery types, contributing to both satisfaction and practical learning outcomes.

• Overall, participants clearly valued delivery features that fostered interaction, collaboration, and structured support. These findings underscore the importance of maintaining a balance between flexibility and guided learning, while also tailoring course features to promote meaningful engagement and application.

Cognitive Presence

Cognitive Presence was robustly validated through high agreement on items measuring critical thinking, curiosity, and knowledge integration. Over 80% of participants responded positively to prompts about triggering exploration and deep understanding. This perception of intellectually engaging content is echoed in the post-course satisfaction questionnaire results.

EQ4: To what extent have the Social Protection trainings achieved their intended results, and how do these vary across different stakeholder groups, course types, or modalities?

• The ITCILO evaluates effectiveness using a four-level approach inspired by the Kirkpatrick Model.

• The training outcomes of the ITCILO Social Protection courses - measured by certificate issuance - show strong evidence of achievement across the evaluated sample. Out of 692 participants (excluding Masterclass attendees), 635 received certificates, corresponding to a high certification rate of 91.8%.

• The sampled trainings had a very high satisfaction rate. With a mean overall satisfaction of 4.55 out of 5. The balance of theory and practice was perceived to be satisfactory (4.21), but perhaps more importantly, participants were very confident to apply the acquired knowledge (4.42), which can be considered a prerequisite for impact. 

• The majority of respondents to the survey (79.2%) reported that, looking back, their satisfaction with the course had increased. In addition to that, 94.4% stated that the training had contributed to their motivation for further developing their skills and competencies, which highlights another positive effect that the trainings had on participants.

• Knowledge acquisition shows variability, with only about half of participants demonstrating measurable improvement between pre- and post tests. Knowledge acquisition also declined consistently with age, and a performance gap was observed across participants from different countries of origin. However, the performance gap between venues is relatively modest. Gender-based differences in learning outcomes were minimal, supporting the conclusion that gender was not a significant determinant of learning performance in the assessed courses.

• Factors influencing effectiveness include high-quality facilitation, intellectually engaging content, and learner motivation. Delivery mode (online vs. in-person) had no statistically significant impact on either overall satisfaction or training effectiveness. Gaps remain in practical application, contextualisation, language accessibility, and post-course interaction.

EQ5: To what extent have the management arrangements, including roles, responsibilities, and coordination, supported the effective delivery of Social Protection training activities?

• The Centre’s training coordination is supported by well-defined roles, quality management systems, and collaborative mechanisms, with participant feedback largely positive regarding organisation and administrative support. However, practical ambiguities in roles, informal knowledge sharing, ad hoc cross-departmental collaboration, and procedural burdens with support services persist.

• Coordination between SPGT and corporate services is formally structured, yet operational challenges persist.

• The ITCILO maintains several collaborative arrangements that support alignment with institutional mandates and reinforce the technical integrity of its Social Protection training offer. These examples illustrate the Centre’s commitment to joint programme development and alignment with broader ILO themes. 

• Findings from both the evaluation survey and the post-course satisfaction questionnaire reveal a consistent and favorable perception of ITCILO’s instructional clarity, and overall delivery. Participants widely recognized the training as well-structured and responsive to their learning needs, with both technical and pedagogical components contributing to the positive experience. 

• From the evaluation survey, 92.8% of respondents agreed that the training was logically and consistently organized. A similar proportion felt that the mode of delivery effectively addressed their learning needs (95.4%) and provided the flexibility or structure appropriate to their learning style (92.7%). Moreover, 91.7% affirmed there were sufficient opportunities for participation and interaction, while 95.4% stated that the delivery mode enhanced their understanding and application of the course content. Participants also noted high satisfaction with the clarity of tutor guidance and the quality of learning resources, reflecting well-coordinated implementation and support.

• These perceptions are strongly reinforced by the post-course satisfaction questionnaire data, where participants rated key dimensions of training organization and clarity with uniformly high scores. Specifically, coherent content received a mean of 4.51 out of 5, clear objectives 4.50, well-organized delivery 4.46, and administrative support 4.53, all indicating over 89% satisfaction. These indicators not only align with the evaluation survey findings but also validate the effectiveness of pre-course planning, communication, and coordination mechanisms.

EQ6: To what extent have financial, human, and time resources been used efficiently in the delivery of Social Protection trainings, and how do participants and clients assess their value for money?

• ITCILO’s Social Protection training portfolio demonstrates a deliberate balance between cost-efficiency, pedagogical quality, and strategic outreach.

• Turin-based residential courses generate the highest CFCs per course and per participant, driven by both scale and higher pricing structures. In contrast, tailor-made and field-based activities generally return lower CFC values due to smaller group sizes and more decentralised delivery models. However, these courses often meet other strategic goals, such as institutional relevance or customisation for priority audiences, which justifies their inclusion in the broader portfolio.

• Strategic measures like early publication of courses and leveraging local expertise are perceived to enhance efficiency. While participants and clients generally expressed high satisfaction and perceived value for money, the evaluation identified operational inefficiencies in participant enrolment and internal workflows, with a clear need for greater automation and standardisation. Staff capacity is also reportedly stretched, particularly for tailor-made requests.

EQ7: To what extent have the Social Protection trainings contributed to meaningful changes for participants and their organisations, and what evidence exists of broader or lasting impact?

• The Social Protection trainings have driven notable change at individual, organisational, and policy levels. Evaluation survey results indicate that participants not only valued the ITCILO training programmes but also actively applied the acquired knowledge in their professional contexts. According to the survey, 94.4% of respondents either agreed or strongly agreed that they could apply the knowledge gained to their work or other activities. Furthermore, 84.8% reported that they had already applied this knowledge in practice, reflecting a strong immediate transfer of learning, with 72.8% providing concrete examples.

• Evidence shows improvements in job performance, professional growth, and influence on organisational practices, including legislative reforms and improved governance. The trainings also contribute to broader systemic changes by facilitating professional networks and dialogue. However, the absence of systematic post-training follow-up limits the Centre's ability to reinforce learning and document longer-term results. 90.4% reported at least a slight improvement in their job performance. This underscores the training’s practical utility and impact on day-to-day professional activities.

• The trainings also appeared to have a motivational effect on participants’ professional development. A combined 94.4% indicated that the course had contributed to their motivation for further learning or growth. This suggests that the programmes not only addressed immediate skill gaps but also inspired ongoing engagement with the field of social protection.

• Looking beyond individual outcomes, 79% of survey respondents reported that the training had already influenced or had the potential to influence their country’s social protection system. While 36.8% noted future potential, 56.8% observed actual influence, ranging from legislative reform and policy development to enhanced actuarial modelling and improved governance practices.

• Evidence from interviews, the evaluation survey, and the desk review suggests that SPGT training has contributed to broader policy-level changes in several countries. Training participation has led to concrete policy outcomes.

EQ8: To what extent are the results and benefits of Social Protection trainings likely to be sustained or scaled, and how well are current and future trainings aligned with the evolving needs of social protection systems?

• There is evidence that the effects of ITCILO’s Social Protection trainings are likely to be sustained and, in many cases, upscaled beyond the immediate training period. Evaluation survey results indicate that 79% of respondents believed the training had already influenced or had the potential to influence their country’s social protection system.

• Qualitative evidence from interviews and focus group discussions reinforces these findings. Participants and institutional clients described the trainings not as isolated events but as part of longer learning and change processes.

• The evaluation also identified examples of institutional uptake and programme evolution that reflect sustained engagement.

• ITCILO’s Social Protection training portfolio is designed to evolve alongside the rapidly shifting global, national, and institutional priorities of its constituents. SPGT’s responsiveness to emerging demands further contributes to the likelihood of sustained results. The team regularly updates its training portfolio to address topical issues. These revisions are based on ongoing consultation and feedback, allowing the team to offer relevant and future-oriented training.

• Participants often replicate what they have learned within their organisations, using ITCILO materials to design and deliver new programmes for colleagues, constituents, or national audiences. In some cases, entire training systems have been established around this content, enabling scale-up beyond the initial group of learners. Ministries, trade unions, and national training institutes have been particularly active in this regard.

• Moreover, the evaluation notes that training outcomes are often embedded in institutional strategies or reforms, including updated policy guidelines, newly adopted international labour standards, and internal operational changes.

• Alumni networks and peer exchange mechanisms also contribute to sustained application. Participants mentioned informally maintaining contact with peers, trainers, and ITCILO staff after the course, exchanging updates and to some extent continuing collaborative efforts. The Centre encourages this through alumni engagement and online communities of practice.

• A recurring theme from interviews with both participants and institutional partners is the value of structured follow-up. Several informants recommended formal post-training sessions, ideally three to six months after course completion, as a space to reflect on implementation progress, troubleshoot challenges, and exchange solutions.

• Participants also highlighted the potential of sustained alumni engagement through thematic communities of practice or peer networks. While such networks do exist, they remain under-utilised, mostly informal and insufficiently supported by institutional mechanisms.

• Participants pointed to the value of follow-up support, particularly for developing and funding implementation projects. Suggestions included providing more face-to-face time in blended courses, increasing flexibility and multilingual delivery, and integrating more region-specific content to improve applicability and uptake.

• The evaluation confirms that ITCILO’s Social Protection training portfolio is strategically relevant, technically robust, and highly valued by its stakeholders, contributing tangibly to capacity development.

• While strong performance was observed in relevance, design validity, and management effectiveness, opportunities for improvement include addressing inconsistencies in knowledge acquisition, enhancing practical application and language accessibility, and strengthening the integration of cross-cutting themes (Gender Equality and International Labour Standards in particular).

• Operational efficiencies could be improved through automation of some processes, while formalising long-term post-training follow-up mechanisms will further bolster quality, inclusiveness, and long-term impact, ensuring continued relevance and contribution to global social justice.