EMBARGOED UNTIL 1.00 PM THURSDAY 7 AUGUST 1997

WAIKATO FORUM ON EDUCATION

UNIVERSITY OF WAIKATO

THE NATIONAL QUALIFICATIONS FRAMEWORK

WHERE TO NOW?

MICHAEL IRWIN

POLICY ANALYST HAMILTON

NEW ZEALAND BUSINESS ROUNDTABLE 7 AUGUST 1997

THE NATIONAL QUALIFICATIONS FRAMEWORK

WHERE TO NOW?

Introduction

In this paper I provide a brief description of the 'original' National Qualifications Framework (NQF), the types of criticisms that it has encountered, and the recent attempt to 'rescue' it by 'broadening' it to allow registration of 'provider' qualifications.

In my view, the NQF designers are faced with a dilemma which has not been sufficiently addressed in official papers including the recent Green Paper. The dilemma, in brief, lies in the tension between:

ï a framework emphasising uniformity and thus interchangeability; and

ï a framework emphasising comprehensiveness and inclusiveness.

Both types of framework have their problems and limitations. The original NQF was of the former type: the rigorous specification of unit standards as the common building block was aimed at maximising comparability, portability, 'seamlessness', and the like. The 'broadening' policy announced in April 1996 and elaborated in the recent Green Paper proposes the latter type but is represented as a natural evolution of the former (i.e. original framework) and one that maintains all its assumed advantages.

The problem is that one can not have it both ways: there are trade-offs to be taken into account. The present proposal is that qualifications designed for different purposes, varied in componentry, and employing a variety of assessment approaches should be eligible for registration. But the more diverse the qualifications, the less comparable they will be, the less cross-crediting can occur, and the less 'seamless' the resulting education system. The Green Paper discusses one aspect of this dilemma in an Appendix, but the body of the text largely submerges it by adhering to the language of unit standards (with the implications of uniformity) while emphasising inclusiveness and comprehensiveness. Thus any redesign of the NQF along the lines of the Green Paper will be unstable and unsatisfactory.

An enduring problem in official documents on the NQF - at least in those made public - has been the failure to adequately address fundamental analytical problems to do with a qualifications framework such as the possible advantages and disadvantages of various models (including the original unit standards-based model), inherent tensions and limitations, and the technical assessment and epistemological issues. The resulting costs to the education system have been considerable. It remains to be seen whether those responsible for NQF design will be prepared to 'go back to the drawing board' in the context of their consideration of submissions on the Green Paper. If they do not there will be ongoing educational and financial costs.

The 'Original' Framework

Before April 1996, the National Qualifications Framework seemed very straightforward. The basic building block of the NQF was the unit standard, and qualifications on the NQF were to consist of various combinations of unit standards. Essential components of each unit standard were - and, of course, still are - outcomes and performance criteria and the level and number of credits.

This all looked very satisfactory. As a framework across all levels and subject areas using a common building block, with its own standards-based assessment philosophy and level and credit characteristics, the NQF seemed to offer great advantages in terms of:

ï recognising existing knowledge and skill wherever and however acquired;

ï ensuring some success for all through standards-based assessment, and enabling assessment as and when the student is ready;

ï encouraging progression. Unit standards could start at school and students could earn credits which would count towards qualifications to be acquired in tertiary education and training or 'on-the-job'. 'Seamless' education would become a reality; and

ï providing coherence across a very wide range of qualifications and facilitating cross-crediting. Since all the outcomes of all education and training at F5 and after would be recorded in unit standards, it would maximise opportunities for 'mixing and matching' of unit standards. Many unit standards would be common to two or more qualifications thus enabling students to change to different pathways without losing the benefit of relevant credits already gained.

Moreover, supporters of the NQF had wider ambitions - to radically change the nature of much of New Zealand education. Establishing 'clear and transparent' standards would explicate what had to be taught and learnt; internal assessment would bring the classroom teacher into the summative assessment process; and putting all qualifications on one standards-based framework would break the perceived elitist stranglehold of the examinations system with its alleged academic/vocational divide and the inbuilt failure rate seen by many as intrinsic to norm-referenced assessment. Certainly David Hood could say in the context of the NQF that "We are still on track towards an education system with the competitive edge in the global marketplace" - as if the education system and the qualifications system were, if not synonymous, at least very closely related.

Of course, the NQF was based on assumptions about the nature of knowledge and the limits of assessment methods which simply could not be sustained. Notwithstanding many excellent intentions, which can only be commended, the NQF as a framework covering all qualifications of every type and level from F5 upwards was sooner or later bound to collapse - it was being built on sand. This is not the place to examine those assumptions - they have been analysed by a number of New Zealand's assessment experts, and I have drawn on their work in my own writings on the subject.

The point I wish to make is that if one could accept the underlying epistemological and assessment assumptions (which I don't) the NQF made a lot of sense. Given those assumptions, it was very attractive, and it was promoted with considerable enthusiasm by the New Zealand Qualifications Authority (NZQA) and others. The key ingredient of the NQF, which held it together and gave it its appeal and a certain elegance, was the common building-block, the unit standard, employing common design characteristics including a common assessment methodology. It was this feature, I suggest, which allowed comparisons in terms of level and credit and encouraged progression in learning, cross-crediting, and so on. Indeed without the common building block it would have been difficult to talk of a framework in anything like the sense in which that word was employed - at least until April 1996.

The 'Broadening' of the Framework - April 1996

By early 1996, considerable pressure had built up for a change to the framework. The Tertiary Lead Group (TLG) had recommended the idea of registering whole 'provider' degrees on the framework - in particular to accommodate degrees without breaking them down into unit standards. The Tertiary Action Group (TAG) was developing this recommendation and the mechanisms for achieving it. Also, some in the secondary schools sector wanted to retain School Certificate and Bursary and, if necessary, have their results recognised in some way on the NQF. And of course the underlying assumptions on which the NQF itself was based had been under severe criticism for some time.

Something had to give, and it came as no particular surprise that the government gave notice that changes were to be made. These were announced in a statement signed by the chief executives of the Ministry of Education, the NZQA and the Education and Training Support Agency (ETSA) on 4 April 1996 and entitled "Realising the Goals of the National Qualifications Framework". The minister sent a memorandum on the same day to the chief executives of the Industry Training Organisations (ITOs). In brief, the main points in the announcement were:

ï "natural evolution" of the NQF suggested it was time to "broaden" it by the inclusion of provider qualifications i.e. those not consisting of unit standards;

ï such broadening should not be seen as altering the nature of the NQF as a "single comprehensive and integrated qualifications structure" but as a long foreseen stage in its "evolution";

ï the relationship between provider and unit standard-based qualifications would be ensured by requiring all qualifications to meet three essential criteria:

- the specification of learning outcomes which "clearly explain what the learner knows and can do";

- adherence by providers and assessors to a "strict quality assurance programme"; and

- defined levels and credits to provide a basis for flexibility and portability for learners.

I wrote to the chief executive of the ministry expressing support for the move away from the monolithic, procrustean structure but pointed out that much needed to be done to clarify what "clearly defined relationships" and "clearly specified outcomes" might mean in practice, and how qualifications involving varying assessment methodologies and covering vastly different content and skills could be assigned to a common structure of levels and credits. I proposed an approach to these issues and, further, suggested that the introduction of unit standards in the schools (to which there was no reference in the announcement) be put on hold pending a thorough and independent review. In reply, the ministry advised that most of the further work on the NQF was a responsibility of the NZQA and that it did not agree that the introduction of school-based unit standards be put on hold pending a review as, in its view, "the end point [was] in sight".

I obtained the officials' paper which preceded the 4 April announcement. In my view, that paper is notable for, inter alia:

i its failure to draw the attention of ministers to the fact that accepting non-unit-standards-based qualifications for registration on the NQF would entail a very major shift in policy, and one requiring a fundamental rethink of the purpose and design of the framework;

ii the representation of current difficulties with the NQF as being "practical" and arising from a "degree of confusion and uncertainty about the final shape of the [NQF]". No doubt these difficulties were, and still are, real but, in my view, the far more important aspects of current difficulties arise from basic NQF design features which were not discussed at all and which were and remain the principal cause of confusion about the NQF's final shape. By contrast, officials stated that the NQF concept is "broadly accepted" and dismissed questions about its "conceptual feasibility" and any challenge to "fundamental concepts", although it is precisely the NQF's lack of conceptual feasibility that has led to demands for its broadening;

iii the absence of any discussion about what is meant by "clearly specified outcomes, an assurance of quality, and level and credits specificity". The lack of quality analysis on critical issues such as these has led to the present ad hoc and totally unsatisfactory approach to policy development. In assessment issues - and in much other policy work - 'the devil is in the detail';

iv the lack of any explicit discussion about the applicability of the concept of an expanded NQF to the schools' sector (other than a brief reference to "some school exams" and a few references to Bursary but without mention of the 'prerequisites' proposal). The implication, possibly unintended, was that all non-unit-standards-based school qualifications would be eligible for registration if they met the criteria;

v frequent references to the NQF as being 'competency' based which carried the implicit assumption that such a concept applies to all education and training whereas it clearly does not; and

vi frequent assertions that the NQF involved significant "gains" which were never substantiated except in a brief mention of employee coverage by ITOs which is hardly indicative of success or otherwise in qualitative terms. The assertions also imply that the ITO system is unproblematic which is not the case.

In short, it was, in my view, an unsatisfactory paper about a very major policy issue. The paper claims to describe "the final shape of the NQF" as if all design and most major implementation problems had been solved, yet the former task has never been properly addressed and until it has been the latter task will be unending.

To those who had followed the development of the NQF, there were features in the April 1996 announcement and in the preceding officials' paper that were, to put it mildly, surprising. First, the 'broadening' of the NQF to include non-unit-standards-based qualifications was described as a stage in a long anticipated "evolution" and necessary to "realising [its] goals". Of course it was always appreciated that such an enormous undertaking as the construction of the NQF would involve progressive implementation. But the idea of the NQF as evolutionary seemed to be in sharp contrast with the revolutionary nature of the NQF and the zeal with which it was promoted as a totally new and ground-breaking concept by the NZQA. The NQF was promoted as leading the world and I would agree with that, but the question is, of course, whether it is leading us forwards or backwards.

I do not recall any suggestions in the early documents, dating from 1990, that did not contemplate a common building block for all NQF-registered qualifications. Far from incorporating other types of qualifications (except possibly as a transitional measure), existing qualifications were to be reformatted to fit the unit standard design. As far as I know, the first serious suggestion that the NQF might depart from the exclusive unit-standards-based approach was in connection with 'provider' degrees, and a recommendation on this was made by the Tertiary Lead Group (TLG) in its report published in November 1994. There seemed to be, in short, a certain re-reading of past events to fit a present requirement.

The second feature I found extraordinary was the implicit assumption that we could do away with the common building block requirement and still talk about the NQF as if its nature was essentially unchanged. If my view that the appeal of the NQF was based on the common building block incorporating a common assessment methodology is correct, the relegation of that building block to the status of an optional ingredient for NQF-registered qualifications was very significant indeed. Thus talk about 'evolution' was inaccurate from a conceptual point of view as well as an historical one. Moreover, the broader claims for the NQF as radically changing the face of New Zealand education could not be sustained - at least not to the same extent.

It seems to me that by April 1996 official thinking on the NQF was impervious to any fundamental critique. Officials had clearly dismissed any possibility that there might be significant weaknesses in the NQF that should be carefully considered and of sufficient seriousness to be brought to the attention of ministers. The work of assessment experts within New Zealand and the overseas studies of competence-based assessment, including those of the Scotvec and NVQ systems for vocational qualifications in the United Kingdom, were, presumably, dismissed as faulty or irrelevant. All problems were those of implementation and, having nothing to do with the basic design characteristics of the structure, could be resolved piecemeal and by allowing "natural evolution" to occur.

However, there was considerable uncertainty as to how exactly the new arrangements would work. In particular, it was quite possible for the three criteria for registration to be interpreted and applied in such a way (for example by applying a strict 'competency' approach to assessment) that very little broadening took place and the "overly restricted approach", which officials stated they wished to avoid, would be maintained. Moreover, the officials' case is complicated by several tensions and contradictions in their arguments which they did not identify, let alone resolve.

First, it was assumed that even within a broadened NQF all qualifications could and should specify clear learning outcomes setting out what students know and can do. This is, of course, the language of standards-based assessment and of unit standards. A phrase used frequently by the previous minister of education when advocating the move to unit standards was the importance of setting 'clear and transparent standards'. It would seem that this is still very much part of ministerial and official thinking. Thus 'outcomes' would seem to mean both curriculum content to be covered and the levels at which students should perform, i.e. the 'elements' and 'performance criteria' of the unit standard. The NZQA's chief executive, David Hood, in a letter to The Independent of 17 May 1996, denied, in reference to the proposed registration of 'provider' degrees, that any breach of the "founding principles" of the NQF would result and stated that "... a learning outcome clearly defined is a standard." How does this maintenance of the unit standard approach fit with the concern to 'broaden' the NQF? Presumably, for a start, all norm-referenced qualifications are to be excluded.

It is instructive at this point to look at the TAG report which was nearing completion at the time of the officials' report and the direction of which was described by officials as "consistent with the [broadened] framework described in [their] paper." At page 10, the TAG state that "Objectives [of a university programme] indicate the planned journey. Achievement of learning outcomes indicates arrival at the destination." At the course level, outcomes should be "meaningful" and "provide an indication of what people who successfully complete the course or paper are able to demonstrate they know and can do." Assessment criteria are also envisaged which "state the evidence and quality of evidence needed so a judgment can be made that the outcome has been achieved." While it is not entirely clear what is intended, the language and concepts are very much those of unit standards - clear outcomes (elements) of a 'know and can do' kind and performance criteria.

A second area of potential contradiction lies in the claim that broadening the NQF will preserve the goals of the NQF to be "inclusive" and "a single comprehensive and integrated qualifications structure". There are at least two problems here. First, as I have just pointed out, the requirement for clear specification of learning outcomes and performance criteria would appear to exclude rather than include. Secondly, there is a tension between comprehensiveness and integration. The more comprehensive the NQF becomes by the inclusion of qualifications with diverse componentry and assessment methodologies the less it can be said to be integrated. The best way of being both comprehensive and integrated is to force all national qualifications into the same mould which was, and perhaps remains, the intention.

Thirdly, it was claimed that the 'broadening' would not endanger flexibility and portability. Again there is a trade-off: the more diverse the qualifications registered on the NQF, the less scope there will be for 'mixing and matching' to form a range of qualifications with common components, and the more difficult it will be to promote credit transfer and hence portability. Again, the best way of maximising flexibility and portability is to force all qualifications to adopt a common design and common assessment methodology - the unit standard approach.

The April 1996 decision to broaden the NQF left many questions unanswered. Its aim appears to have been to retain all the perceived advantages of an NQF based only on unit standards and to meet some of the objections to the NQF by allowing registration of non-unit-standards-based qualifications. Unfortunately, it is impossible to do this: there are trade-offs which should have been identified and evaluated. However, some of the language and concepts applied to the 'broadening' decision suggest that a very restrictive form of 'broadening' was in mind which would maintain a unit standards approach, though not necessarily by that name, across the whole range of the 'broadened' NQF. Whether this was intentional or not was unclear.

These and other issues remained to be addressed and resolved by the Green Paper originally expected in late 1996, but finally published on 5 June 1997.

The Green Paper - June 1997

Although described as a Green Paper, the document is in reality a draft government policy paper. In effect it is the first public draft of a government policy statement, a White Paper, to be published later this year. As a policy statement it provides little analysis; rather it asserts. It states the conclusions with little reference to whatever analysis led to them. While it acknowledges the existence of some criticisms, it still does not address the fundamental concerns raised about a unit standards-based framework or the empirical and theoretical work critiquing similar qualifications in the United Kingdom.

In brief, the Green Paper says that:

ï NQF registration will remain voluntary;

ï however, the government's overall goal for the NQF is that all major qualifications will be registered on it regardless of how they are designed, taught or assessed;

ï for registration qualifications will have to meet or exceed "a clearly specified quality benchmark" (p. 7, emphasis in original), key attributes of which are (pp. 7 and 19-20):

- credibility to interested groups, portability, durability, and structural soundness;

- skills and knowledge are clearly stated by employers and other interested parties;

- valid assessment; and

- opportunities for students to exit, enter and transfer between programmes leading to qualifications;

ï all NQF-registered qualifications will have a common currency, which enables comparisons and cross-crediting, and consisting of:

- statements setting out clearly what students "know and can do", and

- common level and credit characteristics,

as well as being required to meet the quality benchmark;

ï an 'excellence' scale will be developed for unit standards in school subjects;

ï national school examinations that meet the criteria can be registered; and

ï the NZQA will be the overall guardian of the quality of NQF qualifications, will be impartial between different types of qualifications and assessment approaches, and may delegate approval to register to agencies meeting certain criteria.

There is at least some recognition of the trade-offs involved. For example, Appendix B notes, correctly, that there are trade-offs between 'inclusiveness' and ease of credit transfer, and hints of this and other trade-offs are to be found elsewhere (e.g. pp. 11 and 21). On credit transfer, the Paper considers its proposals to be mid-way on the "continuum" between an exclusive unit standards-based approach and one that lets individual awarding authorities decide what credit recognition should be allowed. This may sound a happy compromise, a 'half-way house', but it is far from clear that it is an optimal position and even less clear how and whether this position would be maintained, or even recognised, in practice. In any case, elsewhere the document insists that "outcomes [are] comparable" (p. 21), that NQF qualifications have a "common currency of outcomes, level and credit" (e.g. pp. 7 and 16), and have "logical and obvious relationships with other qualifications" (p. 20) and "the potential to offer credit towards other qualifications" (p. 20), all of which would seem to point to a position nearer the unit standards end of the "continuum".

Further, the NQF is to be "broad" and "inclusive", yet the criteria for registration could be interpreted in a way that is narrow and exclusive. Advantages such as portability and progression are to be maintained, yet embracing all design and assessment approaches will clearly reduce the scope for so doing. The Paper adds to the confusion by the manner in which "quality" is elevated to the position of a "key" issue. In all, the treatment of trade-offs is cursory and unsatisfactory, raising more questions than are answered.

As regards school subjects, the development of an excellence "scale" would seem likely to involve an enormous increase in administrative burdens. Assessing and moderating against all the elements in one standard is difficult enough, yet assessing and moderating against two (or possibly more) standards for each element would seem to be envisaged.

No doubt there are many in the school sector who were relieved that SC, Bursary and other examination-based qualifications can be registered. But it is unclear to me how "clear outcomes" of a "know and can do" variety can be developed for norm-referenced, and scaled qualifications. Nor is it at all clear that they would meet several of the other proposed quality "attributes" without substantial redesign which might lead those who welcomed the announcement to reconsider their support.

The quality criterion is stated to be the "main criterion for NQF qualifications" (p. 7) and the "key element" (p. 10). There are frequent references to this criterion, for example to "a clearly specified quality benchmark" (p. 7, emphasis in original) and "quality threshold", as if it was, in fact, clear. However, the attributes (especially those at p. 20) are numerous, vaguely specified and open to widely varying interpretations. In short, on present proposals there is not, and cannot be, one clear quality "benchmark" or "threshold" which all qualifications must meet or exceed for registration. What is reasonably clear is that we will have a highly intrusive bureaucracy trying to second guess providers and users of qualifications about such matters as their relevance, value and durability. Such issues can only be ultimately determined by the users of qualifications. Again, those who welcome the Paper's advocacy of inclusion and its proposal that the NZQA should be "impartial with regard to different kinds of qualifications [and] assessment methods" (p. 28) may have cause to revise their views.

The introduction to the Paper assures readers that "NQF registration is and will remain voluntary" and goes on to advise that "those who seek to register will gain the benefits of expert audit and endorsement of their qualifications" (p. 4). Those who consider that the national and international reputation of their own institution's qualifications would not benefit significantly from such audit and endorsement may be inclined to read no further. If they were to glance at the required quality attributes on page 20, any lingering inclination to seek registration would, I suspect, promptly disappear as they contemplate possibly protracted and costly negotiations with NZQA officials (or those of an approval agency to which the NZQA has delegated this task) on such matters as their qualifications' value to their students, whether the outcomes establish what their students "know and can do", and whether ongoing "relevance" can be assured. Universities and other tertiary institutions may, in any case, question whether some of the more obviously utilitarian criteria are relevant to much of the education they offer.

I do not think, however, that even the most prestigious of our tertiary institutions should be complacent. The Paper says that the government's overall policy goal in regard to qualifications will only be achieved if all major types of qualifications, at all levels and across all subject areas, regardless of how designed taught and assessed, are included in the NQF (p. 6). In context this appears to mean that all major qualifications should be registered. The Paper does not say what the government would do if major institutions or types of institution stood aside from the NQF, thus frustrating the achievement of its "overall goal". But it is not difficult to imagine that financial incentives could be used to provide powerful incentives to register, for example by setting lower levels of EFTS funding, or no funding at all, for courses leading to non-registered qualifications. It would thus be wise, I suggest, to view the voluntary nature of the NQF with some caution. It seems very possible that the issue will be raised in the context of the tertiary review.

The Paper proposes that the NZQA be the overall guardian of the quality of NQF qualifications and as such should be impartial as regards different kinds of qualifications (p. 28-29). Clearly impartiality in the administration of criteria is required in any state registration authority, but it is pertinent to ask whether it is fair to ask the NZQA to take on this role given the zeal with which it has hitherto promoted an exclusive, unit standards-based, framework. Moreover, the NZQA or any other gatekeeping agency will need substantial and highly experienced and skilled resources if it is really going to evaluate evidence on matters such as the "internal links" within a qualification and whether the outcomes relate to a "coherent body of skills and knowledge" and at all levels from F5 to post graduate work (p. 20).

These highly intrusive criteria raise very considerable concern about who is to be ultimately responsible for professional matters relating to content, coverage and design of qualifications - the awarding body or the NZQA. Even where the NZQA delegates authority to register qualifications to a body such as the NZVCC, it would retain, presumably, ultimate responsibility. In the case of a university's programmes and qualifications, rigorous audit by its peers within and without New Zealand is required, and it is difficult to see how a government agency could perform such tasks. The standing of a university's degrees will also depend on the evaluation of the employers of its graduates who will do their own ranking of university departments and faculties. Thus it is not clear to me that the proposed NZQA quality control will enhance the reputation of our university degrees. It could do the opposite. Certainly an external audit on the effectiveness of an institutions' own quality controls (including external peer review) could be valuable (and perhaps insisted upon as a condition for public funding), but the Paper seems to propose much more than this.

The need to separate the design and award of qualifications from the gatekeeping function raises the question of who is to award the school examinations and the New Zealand vocational certificates, presently administered and awarded by the NZQA. The Paper does not provide the answer.

So where have we got to?

The papers I have discussed are in the long tradition of official papers on the NQF, not one of which has, in my view, adequately addressed such basic questions as:

(i) What real advantages and disadvantages might there be in establishing a national qualifications framework across a broad range of qualifications (e.g. in terms of any existing problems it might resolve or intensify and the positive or negative flow-on effects on the education system)?

(ii) How would these advantages and disadvantages compare with those that might result from the development of existing qualifications frameworks and structures and alternative ways of addressing existing problems by, for example, facilitating credit transfer across existing qualifications and frameworks?

(iii) What are the trade-offs - for example between the advantages of comprehensiveness and inclusiveness and the disadvantages of a less diverse range of qualifications and the loss of innovation; between the advantages of integration and the disadvantages of straying beyond the proper limits of various assessment methods; and between the benefits of portability and the costs of the loss of the intellectual integrity of individual qualifications? And how might these tensions best be resolved?

(iv) If a national qualifications framework were to be constructed, what possible models are there (in terms of, for example, the range of qualifications to be covered and the criteria for inclusion), how should the pros and cons of each be evaluated, and which one would seem likely to offer the greatest net advantage?

As long as these questions remain unaddressed, we are in danger of making matters worse by dealing with symptoms of problems and not their underlying causes. And this is what we have been doing. The unit standards based-NQF was seriously flawed. However, the response to criticism of unit standards has not been to go back to the drawing board but to meet the objections by 'broadening' the NQF to include a wider range of qualifications as if this was part of a natural evolution which would leave the framework unchanged in every other respect. Inevitably contradictions, uncertainties and tensions remain. The model now proposed talks of inclusion but appears to retain some of the language and concepts of unit standards. Registration is to be voluntary, yet the government sees the achievement of its overall goal as necessitating extensive registration, and the question therefore arises about the extent to which registration will be voluntary in practice. The Paper wants the NQF to be all things to all people, but this is impossible.

On the basis of recent official papers, the risks as I assess them are, on the one hand, of yet again trying to force all major qualifications into a narrow, exclusive model with highly intrusive quality control and muddled accountability or, on the other, the model becoming so inclusive as make the concept of a framework virtually meaningless. However, in the latter case, something will have been gained if awarding authorities are clearer about the coverage of their qualifications, the expected outcomes and the basis for assessment; but obviously such gains could be made by far less costly means than the establishment of a comprehensive framework.

Where do we go now?

The Education Forum with which I am associated will be publishing shortly a report by Professor Alan Smithers of Brunel University. The Forum will be making its own submission on the Green Paper drawing on Professor Smithers' report. At this stage I think I can best outline an approach to the issue which I hope will contribute to the further work of officials.

The basic questions such as I have just outlined must be addressed. There are no short-cuts if we want an educationally satisfactory outcome - one which retains what is worthwhile and discards or redesigns the rest. In doing so, I suggest the following four principles might be kept in mind:

(i) We should avoid excessive expectations from a framework. Some of the earlier promotional material suggested that the NQF would inaugurate an educational equivalent of a Second Coming when all that is wrong in our education system would be swept aside. We must be realistic about what a framework can and cannot offer.

(ii) Similarly, we need to accept that there are trade-offs such as I have outlined: the more a framework encourages portability, the less inclusive of diverse qualifications it can afford to be, to repeat just one. We need to work our way through such dilemmas, and one way to do so is to concentrate on those parts of the educational and training sector in which a framework might yield the highest educational benefits. This might, for example, suggest the private training establishments (PTE) area rather than, say, the university sector which already has its own 'framework'.

(iii) There are other dilemmas often raised by ideological ambitions, for example by the perception that an inclusive framework can remove or reduce 'elitist' vocational/academic divides or that internal assessment must be pursued because it is more 'child-centred' than national examinations. We need to face up to the fact that the achievement of such ambitions would not be without educational costs which we should compare with expected educational benefits.

(iv) Let us be very careful not to undo what is already working well or could readily be improved simply because of some commitment to an untried vision. I am glad that at least School Certificate and Bursary are to remain under present proposals, which is not to say that their retention and/or improvement should not be debated as a separate exercise. It does worry me that some of the New Zealand Certificates are being reformatted into unit standards without, as far as I can see, adequate prior analysis and discussion.

(v) A framework will only be as effective as those who implement it. Teachers, tutors and academics must be convinced that it makes sense, that it doesn't lead to curriculum distortion, that tests lead to valid judgment, that time on assessment is commensurate with the information about student achievement that is obtained, that registered qualifications have real value, and so on.

Finally, I would note that a qualifications framework seeks to provide some linkages between the certification of student achievement across a range of content areas. The method of assessment must suit the purpose of the qualifications and the knowledge and skills to be tested. Because such purposes vary and the range and level of subject and skill is so vast, the notion of a framework, even across a quite small range of knowledge, is intrinsically problematic. The problem of making comparisons of achievement across subject areas is compounded by the inevitable human element in assessment, and the results will usually be approximate to a greater or lesser degree. Statements of the educational outcomes expected from educational programmes and courses are almost invariably going to be tentative and incomplete, and their assignment to levels open to dispute. Given all this, I wonder whether our preoccupation with a qualifications framework isn't putting matters back to front. Perhaps we should concentrate on the best possible range of courses and programmes, then decide what summative assessment process is best for each, and only then consider to what extent the results can be linked in some sort of qualifications framework or frameworks.

I am not saying that the construction of a framework should never be attempted. I am saying that a qualifications framework must always be concerned to preserve the intellectual integrity of what is to be taught, learnt and then tested. In other words, intellectual humility is a necessary attribute in any would-be framework designer or, to use the words of the Green Paper, the 'key' quality element.

ACKNOWLEDGEMENTS

I acknowledge with thanks helpful comments made on an earlier version of this paper by Wendy Dodd, Warwick Elley, Cedric Hall, Roger Kerr, David Lythe, Alan Smithers and Warwick Terry. However, responsibility for the views expressed remain my own.

BIBLIOGRAPHY

The following list includes items mentioned in the text of this paper and a selection of other papers and reports written about the NQF and vocational qualifications in the United Kingdom. Numerous other references can be found in some of the texts listed below including Irwin 1994a and Qualifications Framework Inquiry, 1997.

Beaumont, G. (1996), Review of 100 NVQs and SVQs, a report to the Department for Education and Employment, London, January.

Channel Four Commission on Education (1991), Every Child In Britain, Channel 4 Television, London. [The Commissioners were A H Halsey, N Postlethwaite, S J Prais, A Smithers and H Stedman.]

Codd, J., McAlpine, D. and Poskitt, J. (1991), Designing the Framework - A Position Paper on Problems and Issues in the Restructuring of National Qualifications, Massey University, Palmerston North.

Croft, C. (1993), 'The Conflicting World of Standards-Based Assessment', a paper given to the Fifteenth National Conference, New Zealand Association for Research in Education, Hamilton, 2-5 December.

Croft, C. (1994), 'Standards-based Assessment - What do we Know? What should we Know?', a paper given to the Sixteenth National Conference of the New Zealand Association for Research in Education, Christchurch, 1-4 December.

Elley, W.B. (1992), Is Standards-based Assessment Feasible? Some Blunt Words about our Blunt Instruments, National Workshop on Professional Development in Educational Assessment, Educational Assessment Secretariat, Ministry of Education, Wellington.

Elley, W.B. (1994), 'Fundamental Flaw at the Heart of the new Qualifications Plan', The Press, 17 October (reprinted in the Education Digest, Vol. 3 No. 2, August 1995).

Eraut, M., Steadman, S., Trill, J., and Parkes, J. (1996), The Assessment of NVQs, Research Report No. 4, Institute of Education, University of Sussex.

Fitzsimons, P. (1997), What Difference does the NQF Make for Young People? A Longitudinal Study Tracking 98 Students, NZCER, Wellington, February.

Gilmore, A. (1991), An Evaluation of the Sixth Form Certificate English Moderation Trials, Department of Education, University of Canterbury, July.

Gilmore, A. (1996), Assessment Issues in the Qualifications Framework, a paper presented at the Otago-Southland Principals Conference, Cromwell, 22-23 August.

Gilmore, A. (1997), "New Zealand Literature Review" in Te Tiro Hou - Report of the Qualifications Framework Inquiry, PPTA, Wellington, June.

Hall, C. (1994), Obstacles to the Integration of University Qualifications and Courses into the National Qualifications Framework, Higher Education in New Zealand Occasional Paper Number 1, Syndicate of Educational Development Centres of New Zealand Universities, Wellington.

Hall, C. (1994), Unit Standards and Related Matters: Some Observations and Reflections, unpublished mimeo, Victoria University of Wellington, Wellington, 18 October.

Hall, C. (1995a), 'Why Universities do not want Unit Standards', NZVCC Newsletter, August.

Hall, C. (1995b), Following Principles which Accord with the National Qualifications Framework, but without Unit Standards, University Teaching Development Centre, Victoria University of Wellington, Wellington, May.

Hall, C. and Klijn, F. (1995), Academic Staff Perceptions of the Meaning of Competence and Excellence, University Teaching Development Centre, Victoria University of Wellington, Wellington.

Irwin, M.D.R. (1994), Curriculum, Assessment and Qualifications - An Evaluation of Current Reforms, Education Forum, Auckland, May.

Irwin, M.D.R. (1995), The National Qualifications Framework - Notes for Meeting with the New Zealand Council for Teacher Education, 26 September.

Irwin, M.D.R., Elley, W.E., and Hall, C. (1995), Unit Standards in the National Qualifications Framework, Education Forum, Auckland.

Laking, R., Foulkes, A. and Kyrke-Smith, N. (1996), The New Zealand Qualifications Authority - Strategic Risks and Opportunities - A Review, NZQA, Wellington, December.

Neyland, J. (1994), Mathematics on the Unit Standards Framework: What can we Learn from Research?, a paper presented to NZARE Conference, December.

Qualifications Framework Inquiry (1997), Te Tiro Hou, PPTA, Wellington, June.

Robinson, Peter (1996), Rhetoric and Reality: Britain's New Vocational Qualifications, Centre for Economic Performance, London School of Economics, London.

Smelt, S. (1995) Industry Training Organisations, Education Forum, Auckland, June.

Stedman, G. (1997), 'Assessment, Accountability and the 1995 School Certificate Science Examination', New Zealand Annual Review of Education, 6:1996, Victoria University of Wellington, Wellington.

Tertiary Action Group (1996), Report to the Board of the NZQA on the Implementation of a Harmonised Qualifications Framework, NZQA, Wellington.

Tertiary Lead Group (1994), A Single Harmonised Qualifications Framework, November.

Wolf, A. (1995), Competency-based Assessment, Open University Press, Buckingham.