International benchmarking of vocational education and training

Aim

The aim of this project is to present a number of issues in relation to benchmarking the performance of the Australian vocational education and training (VET) system against that of other countries. It is not a project which attempts to provide benchmark data, but inform about the benchmarking process.

The report critically examines two approaches to benchmarking VET. The first of these approaches is that offered by the collection and publication of performance indicator data; the second approach is that represented by comparative case studies.

What is benchmarking?

While comparative studies have long been a source of influence on educational policy-making in Australia, benchmarking as a specific form of study is a much more recent phenomenon. Underpinning the concept of benchmarking is an understanding of how organisations achieve a certain level of performance and the subsequent application of these approaches to another organisation.

The term 'benchmarking' is used to describe a large variety of different measurement and evaluation technologies which have been collected with one single aim: the improvement of organisational performance. Some define benchmarking as a technique similar to process mapping (for example, analysing work processes and comparing them to 'best practice'). Others see benchmarking as the activity of comparing outcomes or results of similar organisations.

Benchmarking generally involves the measurement of key performance criteria, identification of entities which may have similar performance data, comparison of the performance of the organisations, and analysis of the reasons for the differences in performance. Benchmarking as a tool for organisational improvement offers a range of benefits for organisations. These include:

• changing the culture of organisations from being inward-looking to being outward-looking
• improving the quantity and quality of performance information within an organisation
• making monitoring of agency performance by executive government and other stakeholders easier, thus improving accountability.

Benchmarking involves far more than the ad hoc collection of statistics. Benchmarking is a purposeful activity in which the processes used by an agency to deliver its products and services are compared with similar processes elsewhere. It is an ongoing and systematic process to search for and then introduce best practice into an organisation. It is by understanding the reasons for differences or gaps in performance that many organisations improve or grow.

The lessons from international experience are clear: benchmarking is a participative exercise, which needs to be well focused and sensitive to the needs of the organisations and stakeholders involved. Benchmarking is therefore something which is not readily applied to an organisation (or activity), but something which is applied by organisations.

In examining the differences in the performance of the VET sector in different countries, it is natural to attempt to identify the factors which have promoted a higher level of performance. Are the differences in performance due to differences in legislative requirements, structural organisation, resource levels, demographic profile and so on, or are they due to less tangible factors such as motivation of students or morale of the workforce, competing demands from other forms of education and training, or the 'efficiency' of teaching?

The VET sector in Australia is, of course, not one organisation, but a complex set of public and private institutions and service providers. Some are institution-based, while others are based in the workplace. It would be wrong to speak of the Australian VET sector as a single entity, and there may well be as much variation within the VET sector in Australia as between Australia and other countries.

Benchmarking through performance indicators

The first method for benchmarking the performance of Australia's vocational education and training system considered in this project draws on the experience of international organisations in developing performance indicators. The particular example used to illustrate this approach is the Organisation for Economic Co-operation and Development (OECD) publication, Education at a glance (OECD 1998, 2000a).

What distinguishes indicators from sets of statistics is not only the deliberate focus on issues of policy relevance, but also the emphasis placed on conceptual coherence. Considerable effort in the international indicator development process has been devoted to identifying the most appropriate model for organising how the indicators 'fit together'.

Assessing organisational or activity performance is usually not done using only one indicator; rather, a whole suite of indicators is necessary to give a more comprehensive picture of organisational performance. In assessing performance against this suite of indicators, the interrelationships between the indicators need to be taken into account.

Practical issues for benchmarking in the VET sector

The experience gained in developing the indicators in Education at a glance has highlighted a number of key practical issues which have emerged and which need to be considered in developing education benchmarks (particularly in a cross-national context), including problems of definition, data availability, and data quality. There are clearly significant gaps in international data sets at this time in regard to VET. The most serious of these is that the OECD data classifications do not have a separate category for VET.

Given the potential influence of these kinds of data, we need to identify the consequences which arise if the data are 'wrong'. What would the policy consequences be if we continually overestimate our completion rates from VET, or under-estimate funding levels, relative to other countries?

The comparative case study approach

A second approach to benchmarking makes use of what might be termed 'comparative case studies' to highlight similarities and differences between the performance of the education and training systems in different countries. The comparative case study approach gives priority to the analysis of circumstances (and changes in them) within countries rather than between them.

Instead of looking for correlations between standardised indicators in VET, qualitative data are used to identify trends within each country.

The case study approach allows a more thorough analysis of particular issues and can take greater account of the complexity and diversity of countries' institutional structures. This approach also accommodates the complex linkages between education, the labour market and other social institutions. The conceptual framework for the case study approach should therefore provide a tool which is useful for a more comprehensive understanding of an individual country's VET system and associated cross-national comparisons. The case study approach also has the practical advantage over the indicators approach in that it is less constrained by the varying coverage and availability of statistical data for each country.

More importantly, this approach provides valuable insights into how each system works. The individual country analyses enrich formal accounts of systems. They may challenge a country's image of its education system as open or flexible, or may draw attention to pathways which have more symbolic than actual substance.

There are three ways in which we can learn more by examining case studies alongside each other; that is, by making the study a comparative one. We can learn more about each individual country, we can learn from the similarities in the experience of different countries and we can learn from the differences between countries. Hypotheses developed on the basis of the comparisons can be put to a double test: can they explain differences between countries, and can they explain trends and patterns within individual countries?

The case study approach is by its nature, a time-consuming and therefore expensive exercise. Case studies cannot be developed effectively from a distance. They demand the participation of those with significant experience within the VET system of each country to locate, explain and contextualise appropriate data. Unless the case studies are constructed with a specific purpose in mind, it is possible that the data examined will have no relevance to the issue at hand. The usefulness of the case study approach can be enhanced by more clearly focusing on issues of particular policy relevance, rather than on simplistic descriptive overviews of national systems.

Conclusions

An examination of the various approaches to benchmarking enables identification of the ways in which each type of data might usefully contribute to a better understanding of Australia's VET system. While it is clear that there is a range of significant issues attached to each approach in terms of the availability of data, further attempts to initiate serious benchmarking studies would appear to be worthwhile.

The analysis would suggest that contributing to the international comparative indicator efforts of organisations such as the OECD should continue to be supported. However, while they are necessary for the implementation of effective benchmarking, this level of data is not sufficient to be useful in driving improvement.

There is still much work to be done in developing the vocational and training statistical collections, both in Australia and other countries, in terms of developing common definitions, common language to describe the systems and common criteria for judging best practice and so on. However, we need to question whether making simplistic comparisons always makes good sense, particularly if we know that the data are flawed. We also need to avoid drawing conclusions about causes and effects where the data do not support this kind of analysis. The assumption that reforms successfully implemented in one country can be easily transplanted into another also needs to be challenged. Structural and cultural conditions can combine to provide a different context for reform in different countries.

Developing sets of indicators which have a deliberate policy focus may provide a more strategic approach to benchmarking and thus highlight Australia's international standing in relation to VET. To make benchmarking most useful, there is a need to ensure that individual benchmarking studies are clearly linked to the Australian VET policy agenda, and answer specific research needs.