Measuring the quality of vocational education and training using the Student Outcomes Survey

By Wang-Sheng Lee, Cain Polidano Research report 4 March 2011 ISBN 978 1 921809 51 4 print; 978 1 921809 50 7 web

Description

With the Australian VET sector moving towards a more competitive model of provision, prospective students should have greater choice in where they study and what they study. A competitive market also means that VET providers should make available information about course outcomes. This report looks at the potential use of information from the Student Outcomes Survey, including the use of student course satisfaction information and post-study outcomes, as a means of determining markers of training quality. The main recommendation is that a 'scoreboard' approach of post-study outcomes is adopted providing average outcomes by provider and field of education for a number of employment- and training-related variables.

Summary

About the research

Currently, as noted in the review on the Australian vocational education and training (VET) system undertaken by the Organisation for Economic Co-operation and Development (OECD), there is a dearth of information available to students to help them make decisions about which course and provider will best meet their needs.

This report by Lee and Polidano examined the potential use of information from the Student Outcomes Survey, including the use of student course satisfaction information and post-study outcomes, as a means of determining markers of training quality. This project was undertaken independent of the work of the National Centre for Vocational Education Research (NCVER) in this area and is a welcome complement to it.

The main recommendation is that a 'scoreboard' approach of post-study outcomes is adopted as a means of measuring quality. The scoreboard would provide average outcomes by provider and field of education for a number of variables related to employment and training. While this approach has merit, it would necessitate a larger sample than that currently obtained for the Student Outcomes Survey in order for robust estimates to be provided.

In addition to the scoreboard approach, the authors recommend other changes to the Student Outcomes Survey to ensure the data are better used. Coincidentally, NCVER has a number of projects currently underway that align with these recommendations:

  • Publish individual provider information: NCVER is reviewing the data protocols which currently proscribe the release of identified provider information.
  • Collect more information on students and their labour market outcomes: NCVER reviews the survey instrument regularly and welcomes Lee and Polidano's suggestions.
  • Expand the survey to include information on private fee-for-service courses and all ACE (adult and community education) courses: NCVER has commenced a three-year project to address this data gap.
  • Add a panel dimension to the survey: others have also identified the need for longitudinal data that allow for the pathways of students to be tracked. The main issues with this proposal are the cost and the likely response rate in subsequent waves.

In addition to the recommendations listed above, a further challenge now in the quest for greater
transparency is to design a survey framework that applies across the entire tertiary sector.

Tom Karmel
Managing Director, NCVER.

Executive summary

The vocational education and training (VET) sector in Australia is moving towards a more competitive model of provision. In theory this will deliver more efficient outcomes by making training providers more responsive to the needs of students. However, in practice, realising such efficiency gains depends upon prospective students being able to determine how well each course meets their needs so that responsive providers are rewarded with higher demand for their courses. At present, there is little information available for students to make such decisions, a key point raised in the review on the Australian VET system undertaken by the Organisation for Economic Co-operation and Development (OECD) (Hoeckel et al. 2008). Without information on the quality of training, there is the risk that providers will compete on fees alone, to the detriment of quality. Anderson (2005) found evidence of such 'commoditisation' occurring in the VET sector in response to market reforms.

The aim of this report is to examine the potential use of information from the Student Outcomes Survey, including the use of student course satisfaction information and post-study outcomes, as a means of determining markers of training quality. In an analysis of the student course satisfaction measures, we found there are very small variations in reported average student satisfaction across providers, with and without controls for factors that differ among providers unrelated to training quality, such as differences in student intake. There are several possible reasons for this, including the sample used for the survey not being representative of all VET participants.

We argue that outcome measures from the Student Outcomes Survey, such as further study and labour market outcomes, are more meaningful for students making choices on courses and providers, given that such outcomes are the main motivations for study. Further, differences in labour market outcomes also signal how valuable the skills acquired are to employers. All else being equal, the more favourable the graduate employment outcomes relative to competitors, the better a provider is in meeting the needs of students.

We recommend the collation of outcome measures from the Student Outcomes Survey, along with other relevant course and provider information, to be made available as part of a 'scoreboard' of information on courses, similar to the Good universities guide for prospective higher education students. Such a depository of information makes it easy for students to compare and contrast courses and providers. However, we recognise that using outcomes for comparison has its drawbacks. In particular, differences in the outcomes across providers may not only reflect differences in quality, but also differences in the regions and in student clientele, which may create perverse incentives for providers to bias their student intake, shift their location, or pressure poor students to exit prematurely. For this reason we suggest that raw outcome measures are validated against measures that control for differences in student characteristics and student opportunities across providers, such as output from regression models.

To ensure that data from the Student Outcomes Survey are better used, including as part of a 'scoreboard' of information, we recommend a number of changes to the survey, listed in order, from what we consider to be easiest to hardest:

  • publish individual provider information
  • collect more information on students and their labour market outcomes
  • increase the sample size and survey response rates
  • expand the survey to include information on private fee-for-service courses and all adult and
    community education (ACE) courses
  • add a panel dimension to the survey.

Download

TITLE FORMAT SIZE
Measuring-the-quality-of-VET-2327 .pdf 692.7 KB Download
Measuring-the-quality-of-VET-2327 .docx 291.2 KB Download

Related items

Using data from the Household, Income and Labour Dynamics of Australia surveys, the report finds tha… Show more

Increasing the number of people with vocational qualifications at the certificate III level or highe… Show more

When the skills workers have do not match the skills that jobs require, a number of negative labour… Show more

Having a choice of transition pathways from school to work, further education or training gives youn… Show more

Amongst recent initiatives aimed at lifting Australia’s productivity has been a push for a greater n… Show more