ERIC ED354330: Designing a Moderation System. Developing a Qualifications Framework for New Zealand.

48 Pages·1992·1.4 MB·English

by  ERIC

DOCUMENT RESUME
CE 063 034
ED 354 330
Designing a Moderation System. Developing a
TITLE
Qualifications Framework for New Zealand.
New Zealand Qualifications Authority, Wellington.
---INSTITUTION
REPORT NO
ISBN-0-908927-28-2
PUB DATE
92
48p.; For related documents, see CE 063 028-031 and
NOTE
CE 063 035-037.
New Zealand Qualifications Authority, P.O. Box 160,
AVAILABLE FROM
Wellington, New Zealand.
PUB TYPE
Reports
General (140)
EDRS PRICE
MF01/PCO2 Plus Postage.
Accountability; *Certification; Education Work
DESCRIPTORS
Relationship; Employment Qualifications; *Evaluation
Methods; Foreign Countries; *Government School
Relationship; Labor Force Development; Licensing
Examinations (Professions); Outcomes of Education;
Postsecondary Education; Self Evaluation (Groups);
*Standards; State Licensing Boards; *Student
Evaluation
Moderation System (National Qualif Framework);
IDENTIFIERS
*National Qualifications Framework (New Zealand);
*New Zealand
ABSTRACT
A "moderation system" is a system intended to help
ensure uniform interpretation and application of standards within New
Zealand's National Qualifications Framework, which consists of all
nationally registered academic and vocational qualifications and the
nationally registered unit standards from which they are derived. The
process of designing such a system has three stages. In the first
stage, designers are given information depending upon which option is
chosen: a centrally established national system, a national system of
local networks. or internal, local moderation systems. During stage
a management structure is established; functions of moderation
2,
activities are determined; components such as timing, extent,
materials, and personnel are added depending on the characteristics
of unit standards, participating providers, and workplaces. A
moderation method is then selected from among such techniques as the
following: exemplars and benchmark materials, expert assessors,
external written examinations, statistical moderation, common
assessment tasks and reference tests, item banks, distance
moderators, external assessors, external moderators with site visits,
external moderators with panel meetings, consensus panels, and
consensus panel networks. In the third stage, the system is reviewed
to ensure that proposed arrangements are efficient, resource
effective, and accessible. (The appendices offer seven case scenarios
of national moderation systems and a glossary.) (CML)
***********************************************************************
Reproductions supplied by EDRS are the best that can be made
*
from the original document.
***********************************************************************
4
DEVELOPING A
121
QUALIFICATIONS FRAMEWORK
'WV
INW
FOR NEW ZEALAND
Cenr
V
"74!
---:
3
3 3
V
01111111MM161
11,4
3
Tv;stff -r!
II
II
II
2
2 I
Tirr
vilrf
w
.
v`
AI
II
II
DESIGNING A
APIIIIF
wprivar
ODERATION
N61.7
v"lir
-17*...
IliFk
'
SYSTE
7-
7=
"4",..4.,
.
'47!V
"PERMISSION TO
REPRODUCE THIS
U.S. DEPARTMENT OF EDUCATION
MATERIAL HAS
Office or Educational: Research
BEEN GRANTED
arid Improvement
BY
EDUCATIONAL RESOURCES
INFORMATION
CENTER (ERICI
ihrs
dOeument has been reproduced
as
received from the person Or Orpliditat.On
ongenaling ii
Mmor changes have been made
to .morove
reproduction outlay
TO THE EDUCATIONAL
RE-OURCES
Points of new or ommons stated
.n this docu.
INFORMATION
went dO not necessarily represent
official
CENTER (ERIC)."
OERI portion Or
poliCy
7 .1' A1.1
.
NEW ZEALAND QUALIFICATIONS AUTHOURITY
Mana Tohu Matauranga o Aotearoa
2
BEST COPY AVAILABLE
W _
The New Zealand Qualifications Authority will
promote improvement in the quality of education and
training in New Zealand through the development
and maintenance of a comprehensive, accessible and
flexible National Qualifications Framework.
The Authority's main functions are to:
W
ff,
coordinate all qualifications in post-compulsory
education and training (from upper secondary to
degree level) so they have a purpose and
relationship to one another that the
iblic and
.
W
students can understand
set and regularly review standards as they relate to
qualifications
MI
ensure New Zealand qualifications are recognised
overseas and overseas qualifications are
in New Zealand
reco
administer national examinations, both secondary
and tertiary
W_
© New Zealand Qualifications Authority 1992
All rights reserved No part of this publication may be
reproduced by any means without the prior permission of the
New Zealand Qualifications Authority.
71/
.
3
TABLE OF CONTENTS
2
FOREWORD
4
BACKGROUND: Assessment and moderation
6
BACKGROUND: Unit registration
BACKGROUND: Accreditation and re-accreditation
7
8
OVERVIEW OF THE DESIGN PROCESS
STAGE 1: Provide the designers with a brief
9
10
STAGE 2: Design the moderation system
A. Establish a management structure
10
11
B. Give the moderation system a function
13
C. Add the components of the system
13
Timing
Extent
15
17
Materials
18
Personnel
19
D. Select a moderation method
20
Exemplars and benchmark materials
21
Recognition of 'expert' assessors
22
External written examinations
Statistical moderation
24
Common assessment tasks and
25
reference tests
27
Item banks
28
Distance moderators
29
External assessors
External moderators with site visits
30
External moderators with panel meetings
31
32
Consensus panels
33
Consensus panel networks
34
STAGE 3: Review the system
35
RELATED PUBLICATIONS
.
APPENDIX ONE: National moderation systems -
36
Case Scenarios
41
APPENDIX TWO: Glossary
December 1992
4
FOREWORD
7
-
Holders and users of national qualifications must have confidence that
jr
-17r47N.IF
t"./
-
different assessors throughout New Zealand have assessed to the same
standards.
nor
Moderation is a broad term that covers activities that help to ensure there is
-;
uniform interpretation and application of standards.
EXTERNAL MODERATION is an option available to individual national
standards bodies. If a standards body decides it requires external
moderation, it has two further options:
Either
a centrally established and directed national
option 1
moderation system
option 2
a national moderation system of local networks.
Or
In option 1 the moderation systems will be designed centrally by individual
national standards bodies and/or their agents. Agents could, for example, be
any one or a combination of the following:
the Qualifications Authority,
a national professional association,
a national industry organisation,
an individual provider or consortium of providers,
a private consultant.
In option 2 moderation systems will be designed locally by providers and
work-places. Local user group representation may be included.
The main purnse of this bookl et is to assist those given responsibility for
designing moderation systems. Designing a moderation system is not a
simple task and many of the issues covered in this booklet are inescapably
complex.
All providers and work-places are required to have internal arrangements for
ensuring that local assessments are within acceptable national limits. This
means that they will carry out INTERNAL MODERATION of assessment.
This booklet will also assist in the development of internal moderation
systems.
An additional option available to standards bodies is to conclude that
expectations can be met through internal moderation without needing to add
external moderation requirements.
In addition, the booklet will be of interest to those involved in assessing for
the recognition of prior learning.
5
BACKGROUND: Assessment and
Moderation
Fair, valid and consistent assessment
The broad goal for all assessors is to produce assessments that are fair, valid
and consistent. This requires:
appropriate assessment activities, and
accurate assessment decisions.
It is important to note that there are a number of related issues (such as
providing candidates with fair re-assessment opportunities) which are not
within the scope of this booklet.
The technical issues
valid
A
assessment activity is one which assesses what it sets out to assess,
and not something else.
Another way of thinking about validity is that the assessment is 'fit for
purpose' - it is an appropriate way to assess the specific learning outcomes
that are under consideration.
An assessment activity will be valid if its results accurately represent
achievements in the knowledge, skills, attitudes and values that are to be
assessed.
reliable
A
assessment gives results that are a consistent and accurate picture
of what is measured. In other words, the results can be relied upon.
WI/ W
Variation
Any assessment includes a margin of error. Methods of assessment and
applications of standards will vary. The challenge is to limit the variation to
acceptable proportions.
A shoal of fish has been used as a way of explaining the issue. Note that the
fish represent assessors, not learners.
Individual fish might be to the left or right, high or low, forward or
laggard, each following its own path, each with its own motive power.
Yet the shoal as a whole can veer in one direction or another. There are
apparent mechanisms at work within each individual which keep it
close to its neighbour.
Through these mechanisms, deviant fish are returned to the
mainstream. The relationship between individuals adjusts constantly
but the shoal remains within appropriate bounds and retains a constant,
steady progression. Some variations are considered acceptable)
w
This illustration does not mean that learners should not always be encouraged
'
to excel by achieving beyond expectations.
Legislation gives tertiary institutions the freedom 'to teach and assess
students in the manner they consider best promotes learning'. This is a
freedom they practice within the need to maintain 'the highest ethical
standards and the need to permit public scrutiny to ensure the maintenance
of those standards', as well as the need for general accountability.
Legislation gives the Qualifications Authority responsibility for ensuring
there are mechanisms in place to guarantee that assessment procedures are
fair, equitable, consistent, and in keeping with the required standard.
The rights and responsibilities of tertiary institutions and the responsibility of
the Authority require fine balance.
Although this booklet does not directly address funding implications, this is
nonetheless a critical consideration when designing a moderation system.
Costs will most often be borne by providers, and these will usually be passed
on to candidates. The Authority has the responsibility to reject proposed
external moderation systems that have undesirable resource implications for
providers, candidates or Government.
Moderation
Wherever it occurs, moderation of assessment is usually carried out through
sampling This involves targetting selected, representative points to check out
the quality of the whole.
As individual assessors move through the assessment process, the main
function of moderation activities is to ensure that different applications of
standards remain within acceptable limits. In other words, moderation
ensures that assessors remain within the national ,hoal.
1
8
BACKGROUND: Unit
Registration
v
.777
Ntyv."-.,
Details on external moderation requirements are included in the information
that a national standards body submits to the Authority at the time of unit
registration.
The criterion used to determine the acceptability of the proposal is:
arrangements for
There are
efficient, resource-effective and accessible
ensuring the consistency of assessment with the required standards.
If any one of the three components in this criterion is not met, the unit will
not be registered. This would mean that the design process would have to be
re-activated.
The Authority has the responsibility to reject proposed systems that are
technically flawed or have undesirable resource implications for providers,
candidates or Government.
Isorior
vow
"'My
it
01
Tr*
:
'34
17 74/
;'
9
V/ Air
BACKGROUND: Accreditation
and Re-accreditation
Accreditation
An application for accreditation will need to show that for the scope of the
accreditation there is a system for ensuring that assessment is fair, valid and
consistent.
There are three possible routes:
If the registered unit standards indicate that the national standards body
1.
does not require external moderation, an application will focus solely on
internal moderation arrangements.
Where the standards body has decided it will centrally establish and
2.
direct a national moderation system (option 1), the application will
outline internal moderation arrangements and will also indicate intentions to
participate in the external moderation system.
Where the standards body preferred a national moderation system of
3.
local networks (option 2), the applicant will outline internal moderation
arrangements and will also explain how external moderation arrangements
meet the moderation criteria stated in the registered unit standards.
Re-accreditation
At the time of re-accreditation, all providers will be expected to report on
matters such as:
moderation and evaluation activities carried out during the period of
accreditation;
any changes in delivery that have been made as a result of the
moderation and evaluation activities;
any further changes which are being considered.
The re-accreditation process ensures that providers have been meeting their
obligations.
10