ERIC EJ1146560: Using Oral Exams to Assess Psychological Literacy: The Final Year Research Project Interview

2015·0.23 MB·English

by  ERIC

Using oral exams to assess psychological
literacy: The final year research project
interview
Mark Turner & Marina Davila-Ross
The ability to reason scientifically and communicate research appropriately is central to psychological literacy.
Scientific research has little value unless scientists are able to convey results and their consequences clearly
to others. In this study, we outline a method of assessing the development of psychological literacy in
undergraduate students. Data from three cohorts undertaking assessed interviews as part of a final year
research project unit are examined. This assessment evaluated students’ ability to explain the purposes and
findings of their research to a lay audience, to articulate the conceptual basis and methodological background
to their approach, and to reflect on their own development during the research process. Analysis of marks and
feedback from the assessment suggests it provides a reliable means of evaluating skills that contribute to
psychological literacy in a manner that is both educationally acceptable to students and a valid indicator of
their general level of course performance.
Keywords: oral assessment; final year project; student employability.
Introduction
T
HIS STUDY presents and evaluates the (Morrison, 2014). growth in participation
use of oral examinations as a means of rates has placed an increased expectation on
assessing undergraduate psychology stu- higher education institutions to supply grad-
dent learning during their final year empir- uates who are ready for the demands of the
ical research project. Learning how to modern workplace and possess the profes-
communicate scientific findings to others in sional skills that employers want (e.g.
speech as well as writing, is an important part Docherty & Fernandez, 2014; Tomlinson,
of the development of undergraduate psy- 2010). An additional challenge for the psy-
chologists which can contribute to their chology subject area is that its graduates are
future employability. In this study, we known to take longer to progress into grad-
examine the reliability and validity of this uate careers following university and less
form of assessment and consider some of the than 20 per cent are thought to eventually
educational and practical implications of become professional psychologists (e.g.
using oral examinations to assess psycho- Trapp et al., 2011; Van Laar & Udell, 2008).
logical literacy. In a survey of four cohorts of psychology
Psychology remains one of the most pop- graduates between one and seven years post-
ular degree choices with over 100,000 appli- graduation, Coulthard (2013, 2015) found
cations made annually to study the subject at that only 40 per cent of psychology students
UK universities (Universities and Colleges were in full-time employment 12 months
Admissions Service [UCAS], 2015). The con- after graduating increasing to 67 per cent
tinued expansion of higher education, to four years after graduation. Moreover, only
which the subject area of psychology con- 60 per cent of psychology graduates reported
tributes, is founded on the premise that uni- that having a degree in psychology had been
versity study will reward students in the long necessary for obtaining their current employ-
term through the opportunity to progress ment, confirming that a large proportion of
into better paid, highly skilled employment psychology graduates may not explicitly make
4488 PPssyycchhoollooggyy TTeeaacchhiinngg RReevviieeww VVooll.. 2211 NNoo.. 22,, AAuuttuummnn 22001155
© The British Psychological Society
Assessing psychological literacy via project intervTieiwtlse
use of subject knowledge from their degree edge, concepts and practice of psychology;
following graduation (Lantz, 2011). It is, (ii) use of scientific inquiry and critical
therefore, a growing concern within higher thinking; (iii) ethical and socially responsible
education to provide opportunities for actions and behaviour; and (iv) professional
students to gain work-related skills and expe- awareness, development and reflection.
rience during their degree (Quality Assur- A further pervasive theme evident in
ance Agency [QAA], 2014); and for most definitions of psychological literacy
psychology students in particular, to incorpo- and implicit within the sub-principles is an
rate professional development as a core stan- overarching requirement for effective
dard within undergraduate curricula (British communication. Whether presenting or
Psychological Society [BPS], 2014). reporting research findings, communicating
One approach important to the develop- psychological beliefs to the public, or inter-
ment of the employment potential of psy- acting successfully with others, someone who
chology students is the concept of is ‘psychologically literate’ should be capable
‘psychological literacy’. Psychological lit- of expressing relevant information in a
eracy is a broad concept that is assumed to manner appropriate to their intended audi-
encapsulate employability skills, global citi- ence. As Trapp et al. (2011) have indicated,
zenship and scientific understanding central to the notion is communicating the
(Cranney et al., 2013). By emphasising to value of psychology in everyday contexts,
students the scientific, evidence-based rea- such that when engaging with communities,
soning implicit within the study of psy- employers and the media the real-life appli-
chology and its value and relevance to cability of psychological skills and knowledge
different professions, it is possible that the can be conveyed and common mispercep-
employability of graduates can be enhanced tions about psychology can be challenged.
(Mair, Taylor & Hulme, 2013). Since psycho- Since many problems within contemporary
logical understanding might be used benefi- society (e.g. obesity, work-life balance, radi-
cially by graduates to help others function calisation) are behaviourally based, the more
more effectively and ethically within the those trained in psychology are able to com-
workplace or wider society, it is necessary for municate their understanding to others, the
providers of psychology programmes to greater the net benefit to society may poten-
present students with opportunities to ‘apply tially be (Cranney, Botwood & Morris, 2012).
their skills and knowledge to authentic prob- Whilst there is common agreement that
lems in a range of contexts that demonstrate psychological literacy should be a primary
the broad application of psychological outcome of an undergraduate psychology
theory to real life and work situations.’ (Mair education both in the UK (Mair et al., 2013)
et al., 2013, p.6). The challenge faced by and elsewhere (e.g. Cranney et al., 2012;
education providers then is to consider how American Psychological Association [APA],
psychological literacy might be better inte- 2013), the abstract and multi-dimensional
grated into undergraduate degree pro- nature of the construct presents a barrier to
grammes. its integration into curricula with some
Several distinct but related interpreta- authors suggesting there is an urgent need
tions of psychological literacy exist (e.g. to more clearly specify an operational defini-
Cranney et al., 2011; Mcgovern et al., 2010; tion of the concept, its boundaries and how
Trapp et al., 2011) which indicate the con- it can be measured in order to improve its
cept to be both broad and complex (Roberts, real-world utility (Roberts et al., 2015). As
Heritage & gasson, 2015). Central to most Halpern and Butler (2011) argue, simply
definitions lie four key principles whereby calling for students to be educated about
the demonstration of psychological literacy psychological literacy may prove hollow
should include: (i) fluency in the core knowl- unless an effective means of assessing how
Psychology Teaching Review Vol. 21 No. 2, Autumn 2015 49
AMuatrhko rTurner & Marina Davila-Ross
well students develop this collection of graduate curricula should require a diverse
attributes is developed. Assessments that range of assessment practices so that all skills
evaluate psychological literacy are, there- defined in the psychology benchmark state-
fore, required. ment (QAA, 2010) are evaluated. Moreover,
Some attempts at examining psycholog- to improve the employability of graduates,
ical literacy in undergraduates have students should be encouraged to communi-
explored its links to more generic graduate cate the skills they develop during their
skills and abilities. Morris et al. (2013) evalu- degree to others, rather than assuming the
ated a cross-section of undergraduates, abilities that psychology graduates possess
finding that whilst the importance of psy- will be publically recognised. It follows that
chological literacy was rated highly by the development of assessments that pro-
students, the extent to which students felt mote an evaluation of the multi-dimensional
they had developed these skills during their components of psychological literacy and
studies was rated only moderately. Using also allow students to practice the communi-
psychometric test responses from over 500 cation and demonstration of these attributes
students in an Australian university, Roberts will be key to successfully embedding and
et al. (2015) concluded that psychological enhancing its inclusion within undergrad-
literacy could best be characterised as con- uate curricula.
sisting of three primary components: reflec- Several authors have suggested scenario-
tive processes, general academic attributes, based approaches to assessment whereby
and the perception of psychology as a students are asked to adaptively apply psy-
helping profession, although scores on these chological concepts to explain events or pro-
dimensions did not correlate strongly with pose solutions. Such activities might include
students’ self-rated assessment of nine indi- evaluating newspaper editorials or advertise-
vidual characteristics derived from the defi- ment claims (e.g. Halpern & Butler, 2011),
nition of psychological literacy proposed by writing letters intended for a non-psycholog-
Mcgovern et al. (2010). The authors con- ical audience outlining the evidence for and
clude that further objective means of identi- against a particular course of action (e.g.
fying psychological literacy are required. Cranney et al., 2013) or case study analyses
Potential activities that might provide a which require students to implement a
means of embedding psychological literacy strategy or propose an intervention (e.g.
within undergraduate curricula have been Mcgovern et al., 2010). A common issue
considered by some authors. For example, with situated learning experiences, however,
Mair et al. (2013) identify several concepts is that they can be difficult for students to
common to undergraduate psychology accomplish in a limited time frame without
syllabuses (e.g. social influence, mental over trivialising the nature of the task. This
health, resilience) where students might be makes the standardisation of such tasks for
able to use their knowledge to benefit the purpose of assessment challenging given
others, whilst Taylor and Mair (2013) suggest the ambiguity which exists in the potential
three different ways of encouraging students approaches to each topic and the variable
to reflect on social psychological aspects of outcomes that may be obtained. Cranney
their learning including tasks exploring envi- et al. (2013) advocate a portfolio-based
ronmental behaviour, teamwork-based activi- approach, where students are required to
ties and mock interviews. Trapp et al. (2011) build up evidence and reflect on their skills
proposed setting applied problems within development across several modules on their
businesses or the local community such that degree, or the use of ‘capstone’ modules
employers and students are able to see how which require students to apply the knowl-
taught materials have real-world relevance. edge and skills learned from earlier in their
The authors also recommend that under- studies. The nature of capstone units,
50 Psychology Teaching Review Vol. 21 No. 2, Autumn 2015
Assessing psychological literacy via project intervTieiwtlse
however, is known to vary widely across insti- fidence in their ability to deal with questions
tutions and may be taken to include intern- and to avoid feeling foolish in front of
ships, research projects, outreach the examiner (Butler & Wiseman, 1993;
experiences or a mixture of these options Joughin, 2003). The presence of examiner
(Weimer, 2012). Therefore, the need exists panels with whom the student interacts
for more consistent methods of assessing psy- directly, whilst providing motivation, can also
chological literacy to be developed which lead to stress in students and may impede
might readily fit into undergraduate UK their ability to perform to their best,
degree programmes and which are relevant although research evidence that might con-
to the diverse range of attributes which con- firm a direct link between anxiety levels and
tribute to the concept. students’ oral performance in viva voce
One possibility for the assessment of psy- examinations appears inconclusive (Arndt,
chological literacy which also satisfies the guly & McManus, 1986). A further factor to
related goal of contributing to the develop- consider in the use of oral examinations is
ment of employability skills in students is the the unavoidable lack of anonymity for the
use of oral (viva voce) examinations. Using student, which may contravene assessment
oral examinations to assess student compe- protocols in some universities. As a conse-
tencies is well established in many subject quence, assessments may be influenced by
areas including medicine (e.g. Evans, Inger- prior knowledge of the student or be subject
soll & Smith, 1966) and law (e.g. Butler & to other sources of bias similar to those
Wiseman, 1993). Viva voceexaminations also found in selection interviews (e.g. Arvey,
have a long tradition in the assessment of 1979). To protect against this, safeguards are
doctoral research and have been shown to required to standardise the interview
increase self-perceptions of academic com- process, assessment criteria used, and alloca-
petence in PhD students (Jackson & Tinkler, tion of examiners to students.
2001). Oral examinations have been used suc-
The pedagogic benefits of interview- cessfully to evaluate business communication
based oral examinations have been well skills in human resource undergraduates
researched. In most forms of the assessment, (Burke-Smalley, 2014), assess pharmaceutical
a set of questions are developed that cover students’ confidence in care settings (Sib-
core aspects of domain knowledge and skills bald, 1998) and evaluate understanding of
that students are expected to demonstrate. customer service relationships in marketing
Both the students’ depth of comprehension undergraduates (Pearce & Lee, 2009). How-
of this content and their ability to communi- ever, evidence regarding the extent to which
cate this effectively can, therefore, be evalu- oral assessments are indicative of a student’s
ated (Joughin, 1998). The approach allows overall level of ability appears mixed. Oakley
examiners to explore topics in direct conver- and Hencken (2005) used 30-minute
sation with students such that knowledge can assessed interviews with undergraduate
be further interrogated and questions can be sports science students, finding performance
clarified, maximising opportunities for on the assessment where six interview ques-
students to demonstrate their full potential. tions were drawn at random correlated posi-
Students are forced to rely on their own tively with end-of-year exam scores. Torke et
words and understanding strengthening the al. (2010) compared the performance of
academic integrity of the assessment and medical students on a written theory exami-
reducing the potential for plagiarism. Oral nation with their performance during a 10- to
assessment is also thought to improve depth 15-minute viva voce assessment that con-
of learning with some evidence suggesting tributed to the same module. Students were
that students prepare more thoroughly for given a ‘viva card’ of preliminary interview
this form of assessment to help improve con- topics whilst waiting their turn to take the
Psychology Teaching Review Vol. 21 No. 2, Autumn 2015 51
AMuatrhko rTurner & Marina Davila-Ross
examination in an attempt to allow students The existing literature shows that oral
to mentally prepare, although examiners examinations are used across different aca-
deviated to other topics later during the demic disciplines to improve student skills
examination. The authors found that whilst where the assessment format has clear voca-
the ratio of students passing and failing each tional relevance, although evidence
assessment did not vary, overall scores on the regarding the predictive validity of oral
two assessments did not correlate well, con- examinations as an indicator of general
cluding that whilst viva voce examinations course performance is less clear cut, and
may be suitable to differentiate between top may be dependent on the interview method-
performing students where more in-depth ology used. In addition, relatively little is
questions can be posed, they should not be known about the utility of oral examinations
used in isolation to determine whether a stu- on undergraduate courses within the psy-
dent passes or fails a subject. chology subject area, despite the obvious
An old but comprehensive study con- importance of oral communication skills
ducted by Evans et al. (1966) similarly found within the profession. The use of interviews
no correlation between 20-minute oral to assess learning from undergraduate
examinations and an objective written exam- research projects not only fits well with the
ination taken two days later by students on core aspects of the psychology subject bench-
the same module. In this study, medical mark (QAA, 2010) but also encourages
students were interviewed twice by different students to engage in social conversation
pairs of assessors. good agreement (r>0.7) about the validity of the scientific conclu-
was found between junior and senior physi- sions, a central component of psychological
cians within each interview team, and literacy (Cranney et al., 2013). Whilst most
between different teams of assessors. Oral scientists consider the reporting and visi-
examination grades also correlated positively bility of their research in broadcast and print
with the total amount of time each student media to be important (Peters, 2013) this
spent talking and the number of words process can often lead to the misrepresenta-
spoken during the interview which the tion of findings through over-simplification,
authors attributed to more fluent speakers exaggeration or omission of critical detail,
being better able to organise their thoughts. such that some researchers feel ill-prepared
However, no relationship was found between in the art of public communication (Kaye et
spoken interview performance and grading al., 2011; Wien 2014). Misconceptions about
of the same interviews from verbatim written psychology, outside of the scientific disci-
transcripts, when marked independently two pline, are common in everyday society and it
months later; described by the authors as is important for psychologists to help
‘oral parallax’ (p.654) given the shift in eval- develop public understanding by chal-
uation of the same information when con- lenging incorrect assumptions. It follows that
sidered via a different medium. More engaging undergraduates with opportunities
recently, Huxham, Campbell and Westwood to explain research findings in a manner
(2012) also found undergraduate biology that is appropriate to different audiences
students scored significantly higher when should be a fundamental goal of undergrad-
assessed by oral examination compared to uate programmes in psychology. The ability
students who answered the same questions to present scientific arguments and commu-
by written examination, which the authors nicate findings accurately will be key to
suggest may be related to an increased sense improving the scientific literacy of the
of professionalism in the oral context. This general public, and may help counter public
pattern held for questions that required scepticism about psychology and its ability to
scientific analysis and those which asked address significant issues within society. As
students to reflect on skills development. Crowe (2012, p.58) points out, ‘Psychology
52 Psychology Teaching Review Vol. 21 No. 2, Autumn 2015
Assessing psychological literacy via project intervTieiwtlse
leaders, educators, and graduates should be graduating in 2013 (N=163), 2014 (N=155)
capable of “giving psychology away” to recep- and 2015 (N=125), of whom 72 (16 per cent)
tive members of the public, many of whom were male and 371 (84 per cent) were
will have influence on their immediate and female.
wider communities.’
The current study, therefore, seeks to Project interview design and preparation
investigate the use and effectiveness of oral The project interview was designed to meet
examinations as a means of assessing the psy- three key objectives. Firstly, to allow students
chological literacy of undergraduate to demonstrate their ability to explain
students by examining their application to a aspects of their final year project as if to an
substantial core research component of all interview panel unfamiliar with their work
BPS accredited courses, the final year and with only a general awareness of the dis-
research project. The specific aims of the cipline. Secondly, to give students an oppor-
study, were: tunity to expand on the written account of
l To examine the validity of assessed oral their project with respect to the process of
project interviews to predict students’ conducting research and identifying
performance elsewhere on their psy- learning from this culminating aspect of
chology degree. their degree. Thirdly, to assess the extent to
l To examine the reliability of oral project which students’ interview performance was
interviews as an assessment method consistent with the standard of scientific rea-
where different teams of interviewers are soning and communication skills expected
used. of a graduate of psychology.
l To examine students’ views on the Project interviews lasted for 15 minutes.
acceptability and educational impact of All interviews were timed so as not to extend
oral project interviews. beyond this limit and audio recorded for the
l To consider the impact of prior mock purposes of later mark verification. Students
interviews on students’ preparedness and were advised that whilst interviewers would be
perceptions of assessed interviews. friendly and try to place them at ease, the con-
duct of the interview was a formal summative
Method assessment and they should respond accord-
Sample ingly. Interviews started and finished at a des-
Participants were 454 final year undergrad- ignated time, such that late arrivals would
uate students enrolled on two BPS accred- have time deducted from their interview.
ited undergraduate psychology programmes The interview assessment contributed
at one university within the UK. As part of 10 per cent of the student’s overall mark for
these degree programmes, all students com- the project unit, with the remaining 90 per
plete a compulsory 40-credit empirical cent being derived from a written project
research project unit, equivalent to 33 per report of up to 8000 words. This weighting
cent of the total marks available during their was chosen to strike a balance between lim-
final year. iting student anxiety arising from the credit
Data from three cohorts of students are value of the interview whilst still providing
examined of whom 443 attended an assessed students with an opportunity to improve
project interview as part of the final year upon the overall degree class of their
project unit. The remaining 11 students did project. The assessment regulations of our
not complete the assessment on grounds of institution also meant that a student could
reasonable adjustment, extenuating circum- not successfully pass the project unit without
stances on the day of the interview or non- attempting all assessments on the unit. Inter-
completion of the academic year. The final views were held approximately seven weeks
data sample, therefore, comprised those after students had completed and submitted
Psychology Teaching Review Vol. 21 No. 2, Autumn 2015 53
AMuatrhko rTurner & Marina Davila-Ross
their written project reports. However, to Operational procedures
limit the diverse effects that a good or bad Interviews were held over a three-day period
report mark may have on student anxiety or two weeks prior to the students’ end-of-year
motivation during the interview, the decision examinations. All interviews were held in the
was made not to release report marks to same location comprising six adjacent
students prior to interview. research rooms within the psychology depart-
Students were informed prior to the ment, with students first being asked to report
interviews that they should address their to a central waiting area from where they were
answers as if to an intelligent lay person or collected. To help standardise practice
professional who was not familiar with the between different interviewers, all interviewers
specifics of their study. Project supervisors attended a group training session covering
were not included on interview panels, such expected interview questioning and marking
that interviewers were naive to the nature of protocol prior to the main interviews.
each student project. Students were also Each student was assessed by a panel of
informed that they could bring a copy of two academic staff. The first interviewer
their project report to the interview if they acted as the lead interviewer, covering set
wished, but that this was not necessary and themes from a pre-defined list of compulsory
not advised. It was not the purpose of the questions such that all candidates were asked
interview to examine the detailed informa- the same core questions. The second inter-
tion in their work, but their ability to explain viewer acted as the primary marker, making
their work that was of primary importance. notes about the candidate’s answers and
For this reason, interviewers read only the scoring their responses in each question cat-
abstract of the student’s work prior to each egory as the interview progressed. If time
interview, to familiarise themselves with the permitted and where relevant to the candi-
basic nature of the study conducted. date’s earlier responses, the second inter-
As Oakley and Hencken (2005) recom- viewer would also ask follow-up questions
mend, student anxiety can be reduced by selected from a supplementary list of ques-
making students more familiar with the tions, used by all interview teams. Inter-
structure and style of the assessment before- viewers were permitted to be encouraging
hand. To help students prepare for the inter- and supportive by repeating or rephrasing
view, online tutorial materials were questions if required, but were not per-
developed consisting of a guide to the inter- mitted to assist students beyond this.
view process, three videos of full interviews Depending on cohort size, five or six
conducted with ex-students showing good interview teams were used to assess all
and bad answers to different questions, a students, such that each team undertook
pool of practice interview questions, and an between 25 to 30 interviews over a three-day
online forum to which students could post period. First and second interviewers were
questions. Additionally, a special project systematically rotated over the three-day
interview workshop was held six weeks prior assessment period such that each first inter-
to the assessment period where marking viewer marked with each second interviewer.
criteria were discussed and any further The reliability of oral assessments has been
queries could be addressed. Students were shown to increase when multiple examiners
also given the opportunity to practice are used (Wass et al., 2003). This arises since
responding to interview questions via a mock sharing perceptions helps interviewers
interview with their project supervisor after become more aware of the inferences they
completion of their project report. Super- make, information is less likely to be missed,
visors were asked to make use of the practice and bias in decision-making is reduced since
questions available within the online tutorial interviewers provide checks on each other
for this purpose. (Campion, Palmer & Campion, 1997).
54 Psychology Teaching Review Vol. 21 No. 2, Autumn 2015
Assessing psychological literacy via project intervTieiwtlse
Students were allocated to scheduled To facilitate the opening of the interview,
interview slots which exceeded the expected the first two questions were disclosed to
duration of the interview so that the inter- students before the assessment. These were:
view procedure could be explained to the ‘How would you explain your project to a non-psy-
student, to allow for overrun, and to provide chologist?’and ‘How did the idea for your project
time for the interviewers to agree marks and emerge?’Student responses to each of the five
complete a feedback sheet with comments interview themes were graded by the second
for the student, before starting their next interviewer on a discontinuous percentage
interview. Assessment and feedback was scale, with a mark being awarded for each
therefore completed instantaneously, with theme. An overall mark was then calculated
both project report and interview results by the second interviewer, determined as the
being made available to all students two mean score of these five themes plus an addi-
working days following the date of the last tional mark based on their global assessment
interview sessions. of the candidate’s performance during the
interview.
Interview questions and marking criteria The lead interviewer, blind to the marks
Students were asked questions structured awarded by the second interviewer, would
around five main themes. The five themes also independently provide an overall mark
required students to: (i) give a concise non- for the interview. A final interview mark was
specialist explanation of their project; (ii) then agreed following discussion between
explain the rationale for their study; (iii) the two interviewers, taking into considera-
demonstrate an understanding of method- tion the first and second interviewer marks
ological issues concerning their study; (iv) and interview marking criteria. Marking
provide an interpretation of their findings; criteria in the first class category for each
and (v) reflect on what they had learned interview theme plus the global assessment
from the project process. Themes were of the candidate are shown in Table 1.
chosen to reflect the nature of questions that
a recent graduate might reasonably be Results
expected to answer about their research Concurrent validity
project when attending a selection interview To examine project interviews in the context
following university. of students’ performance elsewhere on their
Each theme consisted of at least two com- degree, agreed interview marks were com-
pulsory questions asked by the lead inter- pared with project report marks, final year
viewer and a further three supplementary course averages and students’ overall degree
questions that could be asked by the second classification upon graduation. Paired-
interviewer. Using the same question pool samples t-tests were used to examine whether
and asking questions in the same order interview and project report grades differed
whilst limiting requests for elaboration is for each student. When all cohorts were
known to improve the consistency of the aggregated, no significant difference was
interview process (Campion et al., 1997). found between interview marks (M=67.2%;
However, some variety in follow-up questions SD=8.9%) and project report marks
was felt desirable to help reduce the possi- (M=66.6%; SD=7.1%), t(437 df)=1.27, p=.21
bility of questions being passed between n.s., d=0.07. This was also true when inter-
students tested on different days (Oakley & view and project marks were examined
Hencken, 2005). Interviewers attempted to separately for each cohort (Table 2).
devote around the same amount of time to Significant positive correlations, with a
each theme, with the second interviewer moderate effect size were found between
monitoring the elapsed time and moving the project interview and project report marks
discussion on, as required. (rip). In addition, strong positive correlations
Psychology Teaching Review Vol. 21 No. 2, Autumn 2015 55
AMuatrhko rTurner & Marina Davila-Ross
r
ria for the final year project interview (five main interview themes plus global candidate assessment). Marking criteria (first-class response category)= The student was able to summarise the study and its findings in a succinct and sophisticated manner. Critical analysis ofexisting literature was excellently used to provide a convincing rationale for the study and the methods. The rationale fothe study was clearly placed within an appropriate context. The responses revealed a strong understanding of thesignificance of the research. The student clearly explained the development of the study idea (e.g. from previous research) in a highly sophisticated,concise and logical manner. The responses revealed a strong intellectual interest in theory and scientific ideas. All relevant information concerning the method and procedures (including identifying issues) were expressed in a simplebut sophisticated way, without superfluous detail or repetition of information. The information was logically structuredand easy to understand, and the choice of design decisions with respect to particular methods, analyses or approachesare readily understood by the student. There may have been evidence of advanced methodological considerations (e.g.use of power analysis to determine sample size). The student is able to articulate and evaluate the strengths andweaknesses of the chosen methodology and analyses which are accurately interpreted and expressed using recognisedlanguage. The student is able to provide a convincing description and evaluation of their major findings with respect to itspractical/theoretical implications, as well as showing an awareness of the potential broader applications of their work.The origins and scientific contribution are clearly placed and understood. The responses given are coherently linked tothe issues raised by the project’s rationale. The thoughts expressed do not include undue or unsubstantiated claims orspeculations. Any criticisms expressed or suggestions for further work are genuine and insightful and naturally emergefrom the findings of the study. The response provides a sophisticated analysis of their findings –overall, theydemonstrate an excellent understanding of the strengths and weaknesses of their research. Responses indicate that the student fully understands key challenges faced within the research process. The studentcritically evaluates different stages of their study and comes up with convincing, elegant, and/or original solutions onhow to improve it. Responses show that the student is able to reflect on their plans and expectations prior to the studyand the extent to which these were met. The student is able to illustrate their skills development during the conduct ofproject using thoughtful examples (e.g. with respect to employability or research skills, etc.). S/he also provides genuineideas for improvements or future studies that build on their work and are able to explain convincingly why such futurestudies would be important to conduct. Convincing statements were provided reflecting on the broader application ofthe subject knowledge.
e
t
cri n
Table 1: Marking erview theme Explanation of topic Choice of topic Understanding of methodological issues Depth of interpretatioand scientific discussion Learning derived from study
t
In 1. 2. 3. 4. 5.
56 Psychology Teaching Review Vol. 21 No. 2, Autumn 2015
Assessing psychological literacy via project intervTieiwtlse
%,
Table 1: Continued. The student demonstrates excellent communication skills, a well informed scientific dialogue, and an overall level of6.Global assessment competence expected of a graduate. Responses to the questions are clear and well organised with little redundancy, andof candidatecontained an appropriate balance of description, critical analysis and evaluation. The purpose of their research was clearand their explanations show a clear logical structure with strong attention to detail in every aspect of the interview.Responses consistently were focused on the questions posed. The relative length of the answers provided wasappropriate. Overall, the student made a positive impression on the interviewers, which would convince them (in anemployment context) to strongly support an application from the candidate towards a graduate position or furtheracademic study. † Grades for each interview theme were awarded on a discontinuous percentage scale. To simplify grading, one of three numerical values only could be assigned to an answer in eachdegree class band, with the exception of the first class band where four numerical values were used. The numerical values used for grading were: 0%, 20%, 35%, 42%, 45%, 48%, 5255%, 58%, 62%, 65%, 68%, 74%, 79%, 85%, 95%. Table 2: Comparison of project interview marks, with project report marks and final year average marks for three student cohorts. Year ofNProjectProjectInterview vs. Validity CoefficientGraduationInterviewReportReport(paired-RangeMean (SD)Mean (SD)samples t)rripiy 201316320% –95%65.8% (8.6%)66.5% (6.9%)–0.96 n.s..38**.58** 201415555% –95%68.3% (8.1%)67.1% (6.6%)1.70 n.s..40**.50** 201512542% –95%67.7% (9.8%)66.3% (8.1%)1.53 n.s..37**.49** Notes:rPearson correlation between project interview (%) and project report mark (%)iprPearson correlation between project interview (%) with final year course average (%)iy**p<.001
Psychology Teaching Review Vol. 21 No. 2, Autumn 2015 57