Preliminary stocktake of VET survey activity in Australia

Background

The National Vocational Education and Training Data Strategy Action Group (Action Group) commissioned the Social Research Centre to undertake a preliminary stocktake of VET-related survey research. While this stocktake was originally envisaged as part of a larger strategic review of VET-related survey activity, it was seen as a valuable compendium of survey research across the sector in its own right and is presented here as a standalone report.

The stocktake was compiled from:

• desk research, drawing heavily on NCVER's 'VET Data Strategy: current issues paper' (2008), internet search engines and academic databases
• discussions with NCVER (Adelaide) about the 'VET Data Strategy: current issues paper' and other survey research
• consultations with DEEWR (Canberra) to discuss their VET-related survey research program
• limited liaison with NCVER's Survey Network Group (under the auspices of the Action Group) to identify and collect information about state and territory research being undertaken in this area.

Analytical approach

A taxonomy of VET-related surveys was developed by the authors to identify and categorise the variety of survey activity across the sector. The intention of the broader VET Enhancing Survey Data Project was a reduction in the number of surveys and the burden of survey participants. Five tiers of survey activity were identified:

• Tier 1A: surveys which provide input into established national accountability measures and agreed quality and performance frameworks
• Tier 1B: ad hoc and repeat national surveys undertaken by DEEWR, which are often used for program evaluation, policy development or contract management purposes
• Tier 1C: ad hoc or repeat national surveys which are used for policy development, scoping, or environmental scanning
• Tier 2: state and territory surveys. This tier comprises state and territory level surveys with a primary focus on measuring the performance of the VET sector in that state or territory, as well as surveys which inform and contribute to VET research and policy development
• Tier 3: institute-level surveys or other small-scale specific-purpose surveys.

The main focus of this report is the identification of Tier 1A, Tier 1B, Tier 1C and Tier 2 surveys.

Main findings

VET survey data in context

The report identifies the VET administrative data collections, key VET survey data stakeholders, survey data requirements and relevant VET survey populations in order to make clear the role of VET surveys as one component of the evidence base that supports VET decision-making.

Surveys constitute one of the major methods used to measure a range of aspects of the current and future performance of the VET system that are not captured in the administrative data collections; however, there is a concern that there may be considerable overlap in the surveys being undertaken. Many of these surveys are administered by different bodies, yet target the same respondents. These concerns increase with the number of VET-related surveys, as do issues relating to the ambiguity surrounding survey populations contributing to VET-specific and VET-related data collections.

Tier 1A surveys

The nine Tier 1A survey programs identified by this report assess the nature, quality and outcomes of VET training in Australia to varying degrees of detail and specificity. Broadly speaking, this occurs through the ABS surveys; in more detail from the employers' and learners' perspectives in NCVER's surveys; and in a very specific, localised manner through VET providers in the Australian Quality Training Framework (AQTF) Quality Indicators surveys.

Tier 1B surveys

Twelve Tier 1B surveys were identified in the public domain by this research. This number is an unconfirmed estimate of total survey activity in this field. It could be argued that many of the DEEWR VET surveys are out of scope for this review because they focus on program evaluation or contract management. Departmental programs relevant to the VET sector are large, ongoing and national in focus, which suggests that any survey data required will be on a similar scale. If it is not the Jobs and Training Compact that is being evaluated, there will be another relevant program of a comparable order of magnitude. Similarly, there will always be a requirement to manage the performance of VET-related service providers and it is extremely likely that surveys of clients and customers receiving these services will be a continuing part of DEEWR's work program. Coordinating VET survey activity within the department and across the sector to minimise overlap of data collection periods and respondent burden appears to present an ongoing challenge.

Tier 1C surveys

This research identified ten Tier 1C surveys relevant to the scoping exercise, undertaken by or on behalf of industry skills councils (ISCs), the Australian Flexible Learning Framework and the Australian Chamber of Commerce. This number is an unconfirmed estimate of total survey activity in this field. It is difficult to assess the true volume of industry-related VET survey activity since, while some industry groups openly publish survey outcomes, others appear to be less forthcoming or engage in less primary quantitative data collection. Industry surveys, while often narrowly focusing on a specific skills area or sector are typically national in scope and have the potential to overlap or coincide with other VET-specific or VET-related surveys. Consultation and discussion with ISCs and other industry stakeholders would be required to fully understand the scale and scope of survey activity in this area.

Tier 2 surveys

Twenty-nine Tier 2 surveys were identified by this research. This number is an unconfirmed estimate of total survey activity in this field.

Based on information available in the public domain, it is difficult to assess the scale, scope and currency of VET surveys. State and territory surveys listed in this section are indicative of the 'best available' information and will need to be confirmed as part of the consultation phase of the research.

Jurisdiction-level VET surveys appear to collect data about VET issues from both a training perspective and an education perspective. Surveys of school students or school leavers typically contain a broad range of questions about VET outcomes. It is possible that these recent school leavers are also answering similar questions about training when surveyed by registered training organisations (RTOs) or other VET agencies.

Tier 3 surveys

The institute-specific surveys included for review here are indicative only of the total survey activity undertaken by institutions. The precise number of surveys undertaken at this level could not be clarified by this research. The huge task of identifying and assessing all surveys conducted by RTOs would perhaps require a census of this survey activity or the development of administrative record-keeping practices to report or compile such activity.

Early findings

The ability of this research to adequately assess the scope and relative impact of the issues identified in relation to VET-specific and VET-related survey activity was hindered by a lack of stakeholder consultation. Early findings of the desk-based scoping exercise are listed in brief below:

• Changes in reporting and compliance regimes for the Australian Government and the state and territory governments during 2010 and 2011 create some uncertainty with respect to the identification of future survey data needs.
• There is a very limited amount of information in the public domain about VET-specific and VET-related surveys beyond the Tier 1A national surveys and any work undertaken by NCVER.
• Based on the volume and scope of current survey activity, there appears to be a clear role for VET-specific survey coordination, as distinct from rationalisation, both within and across tiers.
• A thorough assessment of the VET survey population and sample frames needs to be completed before the issues of respondent burden, sample size and longitudinal surveying can be explored. At this early stage, there is little evidence of content gaps but there are indications of 'blind spots' in population coverage, most notably in relation to private providers.

Issues for future consideration

Impact of the national regulator on VET surveys

The establishment of the Australian Skills Quality Authority (ASQA) in July 2011 has the potential to substantially change survey activity at both national and jurisdictional levels. It is currently difficult to determine the nature of this potential impact on VET surveys.

Availability and accessibility of VET survey information

There is a distinct lack of published publicly available information about the scope and content of survey activity across the VET sector (due to, for instance, a lag between research conduct and results release, or a reluctance of jurisdictions and stakeholders to provide detailed information). Again, direct consultation with VET survey stakeholders will be required to substantially analyse relevant issues.

Coordination of VET survey activity

Ultimately, information-sharing amongst stakeholders about survey activity could minimise overlapping periods of data collection and create opportunities for cost-sharing, improved knowledge about survey research and data-sharing.

The Statistical Clearing House (SCH) for Business Surveys website once acted as a key source of information for survey managers, but has not been updated since 2007. It is anticipated that it will be updated in the 2011–12 financial year. For more information, go to this page.

Survey populations and sample frames

Given that information about survey execution was difficult to obtain via desk research, a key challenge encountered was attempting to recognise the VET population of interest and also who had been surveyed from within that sample frame. Obtaining this information accurately rather than inferring from available documentation is a key survey parameter that should be investigated further if a consultation phase of the research is conducted.