Begin with the end: RTO practices and views on independent validation of assessment

Description

Validation of assessment tools, processes and outcomes is an important element of the quality of assessment in vocational education and training (VET). This research explored how registered training organisations (RTOs) conduct independent validation of assessment (IVA) and what industry involvement they have in these processes. Interviews with RTOs reveal that IVA practices are diverse. In terms of industry involvement, IVA tends to be incorporated into ongoing RTO-employer relationships, rather than employers having a formal role. Employers, especially SMEs, have neither time nor interest in validation and tend to see training and assessment as the responsibility of the RTO.

Summary

About the research

RTO staff can see the value of independent validation of assessment as part of a training system striving for quality. Those interviewed in this project would welcome reforms that foster excellence and professionalism. Such change would not, however, require abandoning the current systems. What is favoured is bringing assessment and certification to the fore of training package development, implementation and adjustment, with less onerous validation processes helping to inform improvements.

This report seeks to deepen understanding of the persistent issues registered training organisations (RTOs) encounter when conducting independent moderation and validation of assessments, and of the nature of industry involvement in these processes. The report achieves this by drawing on the findings from semi-structured interviews with public, private and community RTOs in urban and regional settings.

The research is accompanied by two support documents:

• a literature review containing an annotated timeline, spanning 2001 to 2020
• a desktop investigation of validation and moderation approaches in the United Kingdom, Europe and New Zealand.

The interviews reveal that independent moderation and validation entail consideration of the diverse clients and business models of the various types of providers, and that a one-size-fits-all approach is not a workable solution. A desire for more professional development and greater opportunities to share experience in the field of assessment, moderation and validation was frequently expressed.

Key messages

• Independent validation of assessment is driven by regulatory requirements, which can generate a compliance mentality, leading to over-assessment, but not necessarily better assessment practices or improved training.
• Validation can play a constructive role in RTO governance and continuous improvement, although the associated terminology is not universally understood or used consistently, and the reporting burden is considered onerous.
• Compliance and good business practice drive ongoing relationships between RTOs and employers. Integrating validation into these relationships is viewed as potentially more constructive than any formal validation role for employers, especially since many, particularly those in small-to-medium enterprises (SMEs), see validation as the responsibility of the RTO.
• Validation has the potential to bridge the gap between training package requirements and industry realities, given the latter often evolves more quickly than the training package.
• Where moderation does occur, it is primarily used to benchmark assessments across an organisation rather than as a measure of consistency with other organisation’s assessment results.

Executive summary

Getting assessment right is not easy, but it is important. The quality of assessment in vocational education and training (VET) has implications for the credibility of VET qualifications and the competence of the graduates who hold these qualifications. Validation of assessment tools, processes and outcomes is therefore an important element of quality assurance, albeit not one that is universally understood in the same way or practised in a consistent manner. This research was undertaken in order to better understand how registered training organisations (RTOs) conduct independent moderation and validation of assessment (IVA), as well as industry’s involvement in these processes.

The Australian Skills Quality Agency (ASQA; 2020a) website explains these two terms:

Validation is the quality review of the assessment process and is generally conducted after assessment is complete. Validation involves checking that your assessment tools have produced valid, reliable, sufficient, current and authentic evidence, enabling your RTO to make reasonable judgements about whether training package (or VET accredited course) requirements have been met.

Moderation is a quality control process aimed at bringing assessment judgements into alignment. Moderation is generally conducted before the finalisation of student results as it ensures the same decisions are applied to all assessment results within the same unit of competency.

A timeline setting out the policy approaches and refinements to independent validation and moderation of assessment over the last twenty years, as well as reviews and the results of pilots (see support document 1), reveals recurring issues, ones also confronting VET systems in Europe and New Zealand (see support document 2). These are:

• the desirability of a change in the culture of compliance into one of continuous improvement
• greater clarity about what should be independently validated: assessment of tasks in units of competency, qualifications, occupational competency or work-readiness
• the requirement to define ‘independent’, given that the work of validation requires experts who understand the industry, as well as training and assessment processes
• the need to incorporate validation of assessment tools and outcomes into the cycle of training package implementation and to ensure that the language used in the system is clear to trainers, learners, employers and auditors
• more emphasis on the continuous professional development of trainers, and particularly assessors, as part of quality assurance
• a lack of unanimity about the extent to which industry should become involved in moderation and validation
• tensions between reconciling national standards with local contexts, and juggling individual employers’ judgment of competency with broader industry requirements
• frequent reviews and changes to policy settings in the national training system.

The loudest message to emerge from the interviews with a range of RTOs was that independent validation (the 2015 RTO Standards[1] do not mention moderation) of assessment is driven by regulatory requirements. This compliance mentality can lead to over-assessment, for example, the repeated assessment of elements common to more than one unit of competency, but not necessarily better assessment practices or improved training.

That said, RTO managers do acknowledge the role of validation in good governance and continuous improvement. This aspect would be much enhanced if less paperwork were involved: the evidence-collecting and reporting burden is onerous. Where moderation does take place, it is used to benchmark assessments across an organisation (and much less externally) and rarely in response to a student querying a result. Interpretation of these terms, including the variants ‘pre- and post-validation’, is idiosyncratic.

RTOs do fully understand the importance of industry engagement to help them produce competent graduates. In terms of validation, engagement with industry partners is more usually incorporated into ongoing business relationships and encounters. This may in fact be more constructive than formal validation arrangements, given a common perception among those interviewed that most employers, especially SMEs, have neither the time for, nor an interest in, validation, because training and assessment are seen as the responsibility of the RTO.

Understanding of training package requirements is not uniform. Not all trainers and assessors know what is mandated and get lost in the voluminous documentation. Others rely on experts within their organisations to interpret what they must do. Internal rules are often determined with an eye on the auditor not the trainer or student. The result can be impractical assessment and validation rules. The pendulum appears to be swinging, again, towards receiving more guidance on assessment, with some arguing for centrally produced and regulated assessment tools. This could signal a move away from a system that sets out to RTOs what is required of a competent student but not how to train and assess.

At this time of flux in the policy and funding environment, this research is intended to enhance understanding of the persistent problems in current practice. The interviews raised, inter alia, the following questions, although these were not always viewed from the same standpoint. Indeed, the interviews, which ranged across public, private and community RTOs operating in urban and regional settings around the country, strongly suggest that finding workable solutions to independent moderation and validation will entail consideration of the business models and clientele of different types of training providers. As so often in VET, a one-size-fits-all approach is unlikely to achieve the goal of having nationwide credible assessment and certification practices.

Does validation of assessment aim to ensure competence and industry relevance or compliance with RTO Standards?

The compliance mentality can lead to a separate assessment of the skills and knowledge set out in a unit of competency. This obviates against the cumulative experience of learning that results in competent application of skills and knowledge in the workplace. There is a widespread sense that professional judgment is no longer sufficiently valued as part of the assessment process.

The language of trainers and standards is not that of the workplace. This gives rise to ambiguity and a sense of unrealistic expectations. It also poses the question of the degree to which validators can be independent of the training, given they need still to understand the fundamentals of training, assessment and industry realities. Further, workplace conditions and industry requirements evolve more quickly than the training packages, sometimes making, from the RTO’s point of view, compliance difficult, if not impossible.

Is there an adequate feedback loop?

Industry is involved in the formulation of occupational standards but is out of the loop at the training package review stage. Validation should try to bridge that gap. This might be achieved by having assessment appear more prominently in the training packages and incorporating audit findings into the training package processes, with quality assurance involving experts in assessment and industry. This might lead to better articulation of what a graduate can do and how an employer assesses work-readiness, as well as draw attention to the obstacles trainers face in tailoring their programs and assessment practices. For example, some qualifications are wide-ranging (construction) or generic (business), and the rules are not seen to accommodate local contexts and student cohorts in all instances.

In the United Kingdom, a new system of end-point assessment for apprenticeships involves an employer-set assessment plan, introduced to add clarity about what is expected of assessment tasks. A quality assurance, or validation, exercise examined 50 end-point assessments in the first wave of this reform. That exercise showed a considerable amount of what was required for valid assessment had been identified upfront by employers in the assessment plan. The exercise also identified flaws in assessment processes, such as inconsistencies between assessments and the plan, missing elements in assessment tools and poorly explained assessment tasks (see support document 2, p.11). These findings point both to the advantages of employer involvement in shaping assessment and verification processes to improve the process of assessment.

Can new technologies help?

Lack of access to actual or simulated workplaces is a problem. Virtual reality (VR) teaching environments can help but these require investment. Online learning and the use of cameras and recording devices etc. can assist in assessment and its validation, as can greater clarity — but also flexibility — in what are acceptable forms of assessment and validation.

Digital communication via platforms such as Zoom is making industry consultation easier and should help reduce duplication of evidence collection. With the move to this kind of industry engagement and by using the recording functions of the technology, it may be possible for RTOs to more easily gather evidence for assessment and its validation. Some providers are wary of these technologies; some think the training package rules or auditor will not accept them as substitutes. With the rapid shift to digital solutions necessitated by COVID-19, it would be timely to issue (and possibly revise) guidelines on the place of technology in assessment and validation.

How can professional development be better supported?

The interviews revealed an appetite for more opportunities to share experience in the field of assessment, moderation and validation. This could perhaps be facilitated by peak bodies in annual validation/quality assurance summits. In many interviews, the desire for more professional development was emphasised, with some individuals suggesting a specialisation in assessment at diploma level was needed. Those involved in digital solutions indicated that their uptake would not be widespread unless accompanied by a concerted effort to upskill trainers. More centralised assessment resources, or at least benchmarks, might create greater certainty about what is expected of a competent, work-ready graduate.

Validation is part of RTO practice and has a valuable place in the system: it makes good business sense to check with stakeholders that the training product is working. Formally, it often operates on a risk-based approach (high-volume, higher-risk qualifications) or is triggered by the release of an updated training package. Validation systems vary significantly depending on the size of the RTO and its connection to the community. These systems are constructed primarily to comply with regulatory standards. Documenting the validation of assessment tools and the outcomes of knowledge assessments is more straightforward than recording evidence of practical assessments either in the workplace or in simulated work environments. This is a particular challenge for most, as is formally involving employers in the validation of training outcomes. The lack of systematic external validation across the sector means that it remains difficult for employers to judge whether the qualifications issued by different RTOs are of equivalent quality.

There is a mood for a change that values excellence and professionalism while preserving the national system of defining occupational standards. At the RTO level, this could mean, in addition to deeming graduates competent, issuing awards or badges to signal a student’s achievement beyond the benchmark (and as a form of advertising an RTO’s expertise) — validation in the sense of affirmation or recognition of excellence — accompanied by an increase in external arrangements to certify qualifications. In some industries, it will see the continuation of licensing arrangements or the introduction of external examinations or capstone projects. It might even lead to the responsibility for verification of assessment/graduate outcomes to be assumed by an independent person (a master assessor) or body.

COVID-19 has demonstrated that the sector can be nimble and innovative. Embedding these qualities into assessment and its validation needs support for upskilling and professional practice. All this has implications for the architecture of the system and funding regimes and would represent a significant cultural shift that would entail investment and careful management.

1. <https://www.legislation.gov.au/Series/F2014L01377>.