MANUAL TEST OF ABSTRACT REASONING - Psychometrics Canada

66 Pages·2006·1.49 MB·English

TAR 1/11/06 3:21 PM Page 1
T
C
A
R
T
S
B
A
F
O
T
S
E
T

G
N
I
N
The ACER Test of Abstract Reasoning (TAR)is a quick and easily administered test of O
S
abstract reasoning in adults. Abstract reasoning is linked to the ability to think A
E
strategically, analyse information, learn new information quickly, engage in creative R

problem-solving and make logical decisions. TARis suitable for use as a selection or CT
A
development tool for graduate, managerial and other positions that require critical thinking
R
skills. The test is free from language, cultural and gender biases. ST
B
A
TARis available online and in paper-based formats, and is easy to administer and F
O
interpret. It provides immediate results using Australian norms and can be used in
T
individual or group testing situations. S
E
T

G
N
I
N
O
S
A
E
R
TAR can be used for various courses and occupations, including: ACE CT
Managers and executives R A ACER
R
TE T
Business managers S S
T B
Professionals and specialists O A
F F Test of Abstract Reasoning
Para-professionals (e.g. technicians) AB O
S T
Arts professionals TR ES
Finance and data analysts ACT R G T M A N U A L
Medical professionals E N
A I
S N
Scientists O O
N S
IN A
G E
R

T
C
A
R
TAR TAR
T
S
B
A

F
O

T
S
ISBN 10: 0-86431-733-6 E
T
ISBN 13: 978-0-86431-733-9
G
N
I
N
O
S
A
E
R
9 780864 317339 T
C
A
R
T
S
B
Australian Council for Educational Research A

F
O

T
S
E
T
ACER Press
First published 2007
by ACER Press
Australian Council for Educational Research Ltd
19 Prospect Hill Road, Camberwell, Victoria, 3124
Copyright © 2007 Australian Council for Educational Research
All rights reserved. Except under the conditions described in the Copyright Act 1968
of Australia and subsequent amendments, no part of this publication may be reproduced,
stored in a retrieval system or transmitted in any form or by any means, electronic,
mechanical, photocopying, recording or otherwise, without the written permission
of the publishers. Only material contained in the photocopy masters may be reproduced,
and then only in quantities sufficient for the purchaser’s own proper use and not for
resale to, or use by, any other person or organisation.
Edited by Elisa Webb
Cover design by R.T.J. Klinkhamer
Typeset by Cannon Typesetting, Melbourne
Printed by Print Impressions
National Library of Australia Cataloguing-in-Publication data:
Australian Council for Educational Research
ACER test of abstract reasoning manual.
Bibliography.
For secondary students.
ISBN 9780864317339.
ISBN 0 86431 733 6.
1. Reasoning (Psychology) – Testing – Handbooks, manuals,
etc. I. Title.
153.43
Visit our website: www.acerpress.com.au
Contents
List of tables .................................................................................................................................................v
List of figures ................................................................................................................................................vi
List of appendices ............................................................................................................................................vi
About ACeR test of Abstract Reasoning (ACeR tAR) ............................................................................1
Background .............................................................................................................................................1
Description of the tests ............................................................................................................................1
Typical uses .............................................................................................................................................2
Occupational uses ..............................................................................................................................2
Educational and vocational guidance uses ..........................................................................................2
test options ...............................................................................................................................................3
Short or long form ...................................................................................................................................3
Short Form (TAR45) .........................................................................................................................3
Long Form (TAR60) ..........................................................................................................................3
Paper or online administration ................................................................................................................3
Paper .................................................................................................................................................3
Online ...............................................................................................................................................3
scoring and Reporting .............................................................................................................................4
Paper administration ...............................................................................................................................4
ACER Test Scoring Services ...............................................................................................................4
Online Response Entry System ..........................................................................................................4
Online administration .............................................................................................................................5
Automated email reporting ................................................................................................................5
ACER Online Testing System ............................................................................................................5
Direct request ....................................................................................................................................5
Interpretation of ACeR tAR Reports .......................................................................................................7
Interpreting norm scores .........................................................................................................................7
Percentile ranks .......................................................................................................................................7
Stanine scores ..........................................................................................................................................7
Rasch scores .............................................................................................................................................8
Comparison with Raven’s Standard Progressive Matrices and Advanced Progressive Matrices ...............8
Case studies .............................................................................................................................................11
Example 1: Recruitment for a public sector department .......................................................................11
Example 2: Call centre selection ............................................................................................................11
Example 3: Promoting floor staff to store manager ...............................................................................11
Directions for Administration ................................................................................................................13
General information ..............................................................................................................................13
General information for online testing ..................................................................................................13
Minimum system requirements ........................................................................................................14
Specific instructions...............................................................................................................................14
Instructions for paper administration ...................................................................................................15
Instructions for online administration ..................................................................................................19
technical Information .............................................................................................................................22
Summary of research studies .................................................................................................................22
Study 1 – ACER TAR Short Form (TAR45) .........................................................................................23
Study design ....................................................................................................................................23
Study sample ...............................................................................................................................23
Testing procedure .........................................................................................................................24
iii
Contents
Item analysis ....................................................................................................................................25
Reliability ........................................................................................................................................25
TAR45 subsets .................................................................................................................................26
Correlations .....................................................................................................................................26
Set correlations ............................................................................................................................26
Set reliability ..............................................................................................................................27
Validity ............................................................................................................................................27
Correlations with Raven’s Standard Progressive Matrices ................................................................27
Scale scores ......................................................................................................................................28
Demographic variables .....................................................................................................................28
Gender .......................................................................................................................................28
Education level ...........................................................................................................................29
Occupational classification ...........................................................................................................29
Language background ..................................................................................................................30
Age groups ..................................................................................................................................30
Test order ....................................................................................................................................31
Conclusion ......................................................................................................................................31
Study 2 – ACER TAR Long Form (TAR60) ..........................................................................................32
Study design ....................................................................................................................................32
Study sample ...............................................................................................................................32
Testing procedure .........................................................................................................................33
Item analysis ....................................................................................................................................33
Reliability ........................................................................................................................................34
TAR60 subsets .................................................................................................................................34
Correlations .....................................................................................................................................34
Set correlations ............................................................................................................................34
Set reliability ..............................................................................................................................35
Validity ............................................................................................................................................36
Correlations with Raven’s Advanced Progressive Matrices ...............................................................36
Set correlations ............................................................................................................................36
Calculating the scale scores ...........................................................................................................37
Equating TAR60 to APM ............................................................................................................37
Background variables .......................................................................................................................38
Gender .......................................................................................................................................38
Education level ...........................................................................................................................38
Language background ..................................................................................................................39
Study type ...................................................................................................................................39
Test order ....................................................................................................................................40
ACER TAR Short Form (TAR45) ....................................................................................................40
Background variables .......................................................................................................................40
Gender .......................................................................................................................................40
Education level ...........................................................................................................................41
Language background ..................................................................................................................42
Study type ...................................................................................................................................42
Test order ....................................................................................................................................43
Conclusion ......................................................................................................................................43
Study 3 – ACER TAR Short Form (TAR45) 2006 ................................................................................44
Study design ....................................................................................................................................44
Reliability ...................................................................................................................................44
Gender differences .......................................................................................................................45
Conclusion ......................................................................................................................................46
References ...............................................................................................................................................47
Appendices ...............................................................................................................................................48
iv
LIst of tAbLes
Table 1 Source of items in ACER TAR ..........................................................................................1
Table 2 Example occupation groups ..............................................................................................2
Table 3 Examples of appropriate forms for selected jobs ................................................................3
Table 4 Stanine groups and descriptors ..........................................................................................7
Table 5 Simple conversion table between ACER TAR45 and Raven’s SPM ...................................9
Table 6 Simple conversion table between ACER TAR60 and Raven’s APM .................................10
Table 7 Key statistical findings in Study 1 ...................................................................................22
Table 8 Key statistical findings in Study 2 ...................................................................................22
Table 9 Key statistical findings in Study 3 ...................................................................................22
Table 10 Demographics – Gender .................................................................................................23
Table 11 Demographics – Highest level of education completed ...................................................23
Table 12 Demographics – Occupational classification ...................................................................24
Table 13 Demographics – Age group .............................................................................................24
Table 14 Demographics – Language spoken at home ....................................................................24
Table 15 Demographics – Order of tests .......................................................................................25
Table 16 TAR45 set descriptions ...................................................................................................26
Table 17 TAR45 internal set correlations .......................................................................................26
Table 18 TAR45 set correlations with total score ...........................................................................26
Table 19 TAR45 set reliabilities .....................................................................................................27
Table 20 TAR45 mean and standard deviation by gender ..............................................................28
Table 21 ANOVA TAR45 scores on gender ...................................................................................28
Table 22 TAR45 mean and standard deviation by education level .................................................29
Table 23 ANOVA TAR45 scores on educational level....................................................................29
Table 24 TAR45 mean and standard deviation by occupational classification ................................29
Table 25 ANOVA TAR45 scores on occupational classification .....................................................30
Table 26 TAR45 mean and standard deviation by language ...........................................................30
Table 27 ANOVA TAR45 scores on language ................................................................................30
Table 28 TAR45 mean and standard deviation by age groups ........................................................30
Table 29 ANOVA TAR45 scores on age groups .............................................................................31
Table 30 TAR45 mean and standard deviation by test order ..........................................................31
Table 31 ANOVA TAR45 scores on test order ...............................................................................31
Table 32 Demographics – Study types ...........................................................................................32
Table 33 Demographics – Highest level of education completed ...................................................32
Table 34 Demographics – Gender .................................................................................................33
Table 35 Demographics – Language spoken at home ....................................................................33
Table 36 Demographics – Order of tests ......................................................................................33
Table 37 TAR60 set descriptions ...................................................................................................34
Table 38 TAR60 internal set correlations .......................................................................................35
Table 39 TAR60 set correlations with total score ...........................................................................35
Table 40 TAR60 set reliabilities .....................................................................................................35
Table 41 TAR60 set correlations with APM total score ..................................................................36
Table 42 Correlations of TAR60 items with APM total score ........................................................36
Table 43 TAR60 mean and standard deviation by gender ..............................................................38
Table 44 ANOVA TAR60 scores on gender ...................................................................................38
Table 45 TAR60 mean and standard deviation by education level .................................................38
Table 46 ANOVA TAR60 scores on educational level....................................................................39
Table 47 TAR60 mean and standard deviation by language ...........................................................39
Table 48 ANOVA TAR60 scores on language ................................................................................39
Table 49 TAR60 mean and standard deviation by study types .......................................................39
Table 50 ANOVA TAR60 scores on study types ............................................................................40
vv
List of Tables
Table 51 TAR60 mean and standard deviation by test order ..........................................................40
Table 52 ANOVA TAR60 scores on test order ...............................................................................40
Table 53 TAR45 mean and standard deviation by gender ..............................................................41
Table 54 ANOVA TAR45 scores on gender ...................................................................................41
Table 55 TAR45 mean and standard deviation by education level .................................................41
Table 56 ANOVA TAR45 scores on education level ......................................................................41
Table 57 TAR45 mean and standard deviation by language ...........................................................42
Table 58 ANOVA TAR45 scores on language ................................................................................42
Table 59 TAR45 mean and standard deviation by study types .......................................................42
Table 60 ANOVA TAR45 scores on study types ............................................................................43
Table 61 TAR45 mean and standard deviation by test order ..........................................................43
Table 62 ANOVA TAR45 scores on test order ...............................................................................43
Table 63 Number of males and females in the reference group ......................................................44
Table 64 Descriptive statistics for TAR45 ......................................................................................44
Table 65 Reliabilities of TAR45 subscales ......................................................................................45
Table 66 Correlations between subsets in TAR45 ..........................................................................45
Table 67 TAR45 scores for males and females ...............................................................................45
Table 68 Statistics from independent sample t tests .......................................................................46

LIst of fIguRes
Figure 1 Example of a group report from ACER Test Scoring Services ............................................4
Figure 2 Example of an automated email report ..............................................................................5
Figure 3 Example of a group report from the Practitioner’s site .......................................................6
Figure 4 Bell-shaped distribution ....................................................................................................8
Figure 5 95% confidence belt for equating between TAR45 logits and SPM logits .......................27
Figure 6 95% confidence belt for equating between TAR60 logits and APM logits .......................37

LIst of AppenDICes
Appendix A QUEST item analysis for all 60 TAR items (Study 1) .....................................................48
Appendix B QUEST output for TAR45 (Study 1) .............................................................................50
Appendix C Detailed conversion table between TAR45 and SPM (Study 1) .......................................52
Appendix D QUEST item analysis for all 60 TAR items (Study 2) .....................................................53
Appendix E QUEST output for TAR45 (Study 2) .............................................................................55
Appendix F Detailed conversion table between TAR60 and APM (Study 2) ......................................57
vi
About ACeR test of AbstRACt ReAsonIng
(ACeR tAR)
background
ACER Test of Abstract Reasoning (ACER TAR) has been developed in response to demand for an abstract reasoning
assessment with Australian adult data.
The items from ACER TAR have been sourced from two existing, well-researched ACER tests of abstract reasoning:
the Middle Years Ability Test (MYAT) and the Aptitude Profile Test Series (APTS). Items from these two existing tests
have been combined to create the ACER TAR Long and Short forms:
Table 1 Source of items in ACER TAR
Test form Item source
ACER TAR Short Form (TAR45) 45 abstract reasoning items from MYAT
ACER TAR Long Form (TAR60) 45 abstract reasoning items from MYAT
15 abstract reasoning items from APTS
Items were chosen to reflect an appropriate range of difficulty and task sub-type.
The data samples detailed in this manual are based on:
(a) a study conducted on 151 adults in Australia who completed both ACER TAR and the Raven’s Standard
Progressive Matrices (SPM);
(b) a study conducted on 295 adults at an Australian university who completed both ACER TAR and the
Raven’s Advanced Progressive Matrices (APM); and
(c) a study conducted on 329 adults seeking work who had completed ACER TAR online.
Further research studies and analyses of ongoing data collection projects will be reported in future editions of this
manual.
Description of the tests
ACER TAR is a test of abstract (or non-verbal) reasoning.
The MYAT Manual defines abstract (non-verbal) reasoning as ‘the extent to which [a person] can discover
principles and rules, and apply them to solve problems using abstract visual patterns rather than numbers and
words’ (p. 1).
The APTS Manual defines abstract reasoning as the ability to ‘discover the principle/s [that] determine… the
rule or rules that govern the progression of a pattern from one diagram to another in a series, or to identify the part
which is missing from a diagram’ (p. 3).
Tests of abstract reasoning such as the ACER TAR are nearly language-free and so may avoid problems asso-
ciated with the use of verbal reasoning tests in some contexts where candidates’ written English skills are not well
developed, such as where candidates are from non-English speaking backgrounds or have specific language learning
difficulties. Statistical analyses detailed in this manual (see Technical information, p. 22) confirm that language
background is not a significant source of variance in performance on the ACER TAR.

ACER Test of Abstract Reasoning
typical uses
Occupational uses
ACER TAR is typically used as a measure of general ability for selection into occupations which involve a moderate
to high level of demand on reasoning ability and for other purposes where the ability to think clearly, analyse infor-
mation, solve problems and make rational decisions are important.
Examples of selection criteria linked to measures of abstract reasoning are:
• Planning and coordinating activities
• CCrreeaattiinngg iinnnnoovvaattiivvee ssoolluuttiioonnss ttoo pprroobblleemmss
• MMaakkiinngg ddeecciissiioonnss iinn aa rraattiioonnaall mmaannnneerr
• LLeeaarrnniinngg nneeww mmaatteerriiaall qquuiicckkllyy
• CCoolllleeccttiinngg aanndd aannaallyyssiinngg iinnffoorrmmaattiioonn
(Power, 2004, p. 24)
Table 2 provides examples of occupational groups for which the ACER TAR can be used.
Table 2 Example occupation groups
Main occupation group Examples of occupations
Managers and administrators General and specialist managers, managing supervisors, business
managers
Professionals Scientists, architects, engineers, medical practitioners, school
teachers, social and business professionals
Para-professionals Medical and science technical officers, technicians, technical
workers, registered nurses, police officers
Tradespersons Fitters, mechanics, electrical and electronic tradespersons
Artistic and creative Graphic artists, photographers, pianists, actors, dancers, desktop
publishers
Figures and computational Accountants, clerks, data analysts, computer programmers,
mathematicians, physicists
Literary Authors, journalists, editors, script writers
Medical Surgeons, general practitioners, nurses
Technical and engineering Electrical technicians, mechanical engineers, architects
Scientific Biologists, chemists, physicists, mathematicians, surveyors, teachers
(APTS, 2000, pp. 6–7)
Educational and vocational guidance uses
The ACER TAR tests may also be used in conjunction with a record of attainment and other data, in counselling
a candidate in selecting an appropriate course of study, and in the selection of occupations where further study
and training are involved. (See Nelson Bolles, 2002; Holland, 1997; Kummerow, 1991; Anastasi, 1976; Cronbach,
1970.)

test optIons
short or long form
ACER TAR has two forms: a Short Form (TAR45) and a Long Form (TAR60).
Short Form (TAR45)
TAR45 is the easier of the two tests and is intended for use with candidates who have completed at least Year 11
and who are applying for a variety of technical, clerical and administrative positions where in-service training, or
part-time or after work study is involved.
Long Form (TAR60)
TAR60 includes the 45 items in the Short Form, plus an additional 15 items. TAR60 is the more challenging test
and is appropriate for use with candidates who have completed at least Year 12 and are applying for positions where
the work requires a high level of reasoning ability or where these qualities will become increasingly important as
staff are promoted in a career system.
Table 3 provides examples of appropriate forms for selected jobs.
Table 3 Examples of appropriate forms for selected jobs
Short Form (TAR45) Long Form (TAR60)
Personal assistant Journalist/copywriter
Market research interviewer Computer programmer/analyst
Accounts clerk Market research analyst
Insurance investigator Financial advisors
Customer service operator Graduates
Call centre operator Managers
Workplace trainer Executives
Legal clerk
paper or online administration
ACER TAR is available for online administration or administration in paper and pencil format. In both cases ACER
TAR should always be conducted under appropriate supervision. In no circumstances should ACER TAR be
administered in an unsupervised environment. Candidates should not be provided with login codes to access the
online test from home or other remote locations unless appropriate supervision has been put in place.
Paper
Test administrators should provide one ACER TAR reusable test booklet and one ACER TAR answer sheet for each
candidate. Completed ACER TAR OMR answer sheets should be sent to ACER Test Scoring Services for scoring
and report generation. Alternatively, responses on non-OMR answer sheets can be keyed into the Online Response
Entry System by the test administrator for immediate access to results.
Online
Test administrators should provide each candidate with a unique login code, which allows candidates to access the
tests online. On completion of the online test, results will be automatically generated and a report emailed to the
test administrator. Further details regarding online testing are provided in Directions for Administration on p. 13.
Contact ACER Press Customer Service on 1800 338 402 for more information about setting up an online testing
account.