2015, 1–13, Early Online
Predictors of professional behaviour andacademic outcomes in a UK medical school:A longitudinal cohort study
JANE ADAM1, MILES BORE2, ROY CHILDS3, JASON DUNN1, JEAN McKENDREE1, DON MUNRO2 &DAVID POWIS2
1Hull York Medical School, UK, 2University of Newcastle, Australia, 3Team Focus, UK
Abstract
Background: Over the past 70 years, there has been a recurring debate in the literature and in the popular press about how best
to select medical students. This implies that we are still not getting it right: either some students are unsuited to medicine or the
graduating doctors are considered unsatisfactory, or both.
Aim: To determine whether particular variables at the point of selection might distinguish those more likely to become satisfactory
professional doctors, by following a complete intake cohort of students throughout medical school and analysing all the data used
for the students’ selection, their performance on a range of other potential selection tests, academic and clinical assessments
throughout their studies, and records of professional behaviour covering the entire five years of the course.
Methods: A longitudinal database captured the following anonymised information for every student (n¼ 146) admitted in 2007 to
the Hull York Medical School (HYMS) in the UK: demographic data (age, sex, citizenship); performance in each component of the
selection procedure; performance in some other possible selection instruments (cognitive and non-cognitive psychometric tests);
professional behaviour in tutorials and in other clinical settings; academic performance, clinical and communication skills at
summative assessments throughout; professional behaviour lapses monitored routinely as part of the fitness-to-practise
procedures. Correlations were sought between predictor variables and criterion variables chosen to demonstrate the full range of
course outcomes from failure to complete the course to graduation with honours, and to reveal clinical and professional strengths
and weaknesses.
Results: Student demography was found to be an important predictor of outcomes, with females, younger students and British
citizens performing better overall. The selection variable ‘‘HYMS academic score’’, based on prior academic performance, was a
significant predictor of components of Year 4 written and Year 5 clinical examinations. Some cognitive subtest scores from the UK
Clinical Aptitude Test (UKCAT) and the UKCAT total score were also significant predictors of the same components, and a unique
predictor of the Year 5 written examination. A number of the non-cognitive tests were significant independent predictors of Years
4 and 5 clinical performance, and of lapses in professional behaviour. First- and second-year tutor ratings were significant
predictors of all outcomes, both desirable and undesirable. Performance in Years 1 and 2 written exams did not predict
performance in Year 4 but did generally predict Year 5 written and clinical performance.
Conclusions: Measures of a range of relevant selection attributes and personal qualities can predict intermediate and end of
course achievements in academic, clinical and professional behaviour domains. In this study HYMS academic score, some UKCAT
subtest scores and the total UKCAT score, and some non-cognitive tests completed at the outset of studies, together predicted
outcomes most comprehensively. Tutor evaluation of students early in the course also identified the more and less successful
students in the three domains of academic, clinical and professional performance. These results may be helpful in informing the
future development of selection tools.
Introduction
Writing in the British Medical Journal in 1946, Smyth observed
that ‘‘Existing methods of selection [of medical students] which
worked well in the past may no longer be the best possible in
changing conditions’’, further suggesting ‘‘we want . . . two
independent tests or sets of tests – the one for ability, the other
for character’’ (Smyth 1946). Though most medical students do
graduate and become professional and capable doctors, the
subsequent and continuing debate creates the impression that
medical schools are still selecting unsuitable students
(Campbell 1974; Lockhart 1981; Lancet editorial 1984; Best
1989; Barr 2010), ‘‘who, though able to pass examinations,
have not the necessary aptitude, character or staying power for
a medical career’’ (Goodenough Committee 1944).
For decades medical schools have tried to appraise the
personal qualities that might underpin students’ future
Correspondence: Professor David Powis, School of Psychology (Psychology Building), The University of Newcastle, Callaghan, New South Wales
2308, Australia. Tel: +61 2 4921 5625; E-mail: [email protected]
ISSN 0142-159X print/ISSN 1466-187X online/15/000001–13 � 2015 Informa UK Ltd. 1DOI: 10.3109/0142159X.2015.1009023
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
professional behaviour, by evaluating applicants’ personal
statements and referees’ reports, measuring cognitive skills and
conducting face-to-face interviews, in addition to assessing
their academic suitability. These approaches have not as yet
shown convincing predictive validity for medical school or
later (Gray et al. 2002; Ferguson et al. 2002, 2003; Groves et al.
2007; Lynch et al. 2009; Poole et al. 2012; Kelly et al. 2013). No
previous study has examined the predictive power of non-
cognitive tests, either alone or combined with a range of other
selection tools, throughout the length of a medical course.
However, demonstrating predictive validity of medical student
selection is particularly difficult because most medical schools
record course outcomes only in terms of the results of
academic and clinical examinations. Such outcomes are
seldom in an appropriate form to reflect, or sufficiently
robust to evaluate, students’ non-cognitive and behavioural
attributes (Schuwirth & Cantillon 2005), and seldom include
final summative or barrier assessments of professional
behaviour.
The aim of our study was to examine what student
attributes and qualities, alone or in combination, best predicted
a range of outcomes of medical education. We therefore
undertook an in-depth longitudinal study of an entire entry
cohort of medical students through a five-year medical course
at one UK medical school, in whom most potential predictors
were measured either before or soon after the start of the
course. This longitudinal study explores the question of
predictive validity of the selection methods employed, and
of other potential selection tests, in relation not only to the final
examination outcomes but also to clinical and professional
behaviour throughout the course. The approach was not
hypothesis-driven but exploratory, thus allowing for emergent
relationships between variables.
The data included all the initial selection parameters, plus
results from cognitive and non-cognitive test results not used
in selection, as well as summative written and clinical
examination results from each sitting over the five years of
the course. These examination data allowed novel ways of
exploring clinical performance, because the clinical examin-
ations in the final two years (Objective Structured Long
Examination Reports, OSLERs, and Objective Structured
Clinical Examinations, OSCEs) provided not only measures
of success but also of deficiencies in various aspects of clinical
performance, expressed as ‘‘penalty points’’ (PPs). In addition,
data were collected from regular structured observations made
by students’ tutors (all of whom were experienced clinicians,
the problem-based learning tutors in the first two years as well
as the clinical placement tutors in Years 3–5), which included
assessments of professional behaviours. Uniquely, the tutor
data were collated with the records of the school’s fitness-to-
practise committee to provide a measure of lapses of profes-
sional behaviour, quantified for the purpose of this study on
a scale of ‘‘fitness to practise penalty points’’ (FTPPPs).
We considered that this wide range of assessments, and in
particular the design of the clinical examinations, would
provide a range of measures not only of students’ academic
prowess, but also of their likely personal and behavioural
qualities as clinicians. The exact assessment methods will be
described in detail.
This complex longitudinal data set was analysed to
determine associations between predictor variables (including
demography, prior academic qualifications, cognitive and non-
cognitive/personality and behavioural qualities), and to link
these to criterion variables measured during or at the end of
the course reflecting academic and clinical performance, and
to professional behaviour. The findings from Years 1 and 2 of
the five-year course have been reported earlier (Adam et al.
2012); this report deals with Years 3 to 5 inclusive.
Methods
Study sample
All students admitted to Hull York Medical School (HYMS) in
September 2007 participated in the study, which had ethics
approval from HYMS’ Medical Education Ethics Committee.
HYMS offers a five-year, problem-based, spiral curriculum
within which a large proportion of the clinical experience is
met in primary care. The cohort comprised 146 individuals, of
whom 140 agreed to complete the non-cognitive tests. There
were 62 males (43%) in the study sample. One hundred and
eleven (76%) of the sample were aged under 21 years at the
time of entry and 120 (82%) were British citizens. Ethnic origin
was not recorded. The mean age at entry of the study sample
was 19.9� S. D. 3.9 years (range: 18–42, median age 18).
Data collection
HYMS selection parameters
UCAS form. Selection into the HYMS medical school
programme for this cohort was based on a score derived
from information contained in the Universities and Colleges
Admissions Service (UCAS) form submitted centrally by UK
university applicants, and an interview score.
Practice points
The study uniquely documents� the progress of a complete entry cohort of students
throughout a five-year medical course.� The consequences of an extended range of predictor
variables (including non-cognitive qualities) measured
at the start of the course.� student progress on an extended range of outcome
data collected during the course, including.� frequently repeated behavioural observations
documented by Year 1 and Year 2 tutors.� standardised observation and reporting of all
lapses in professional behaviour.� summative course outcome measures that distin-
guish between performance in clinical assess-
ments and academic examinations.
Together, these features have allowed the creation of a
correlation matrix to determine the strength of linkage
between entry, progress and outcome variables.
J. Adam et al.
2
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
HYMS academic score. Each applicant’s UCAS form
listed their academic results (either predicted or already
obtained), usually as either advanced-level grades (or equiva-
lent) or a degree classification. The academic grades likely to
be required for entry were published before applications
opened. Administrative staff categorised the academic results
listed and determined a numerical HYMS academic score,
using guidelines that ensured comparability with other systems
of examination results such as the International Baccalaureate.
The HYMS academic scores, using the example of A level
grades, were: Unsatisfactory¼ 0, predicted 2 A level grades
below likely requirement; Just satisfactory¼ 2, predicted 1 A
level grade below likely requirement; Clearly satisfactory¼ 5,
predicted or obtained likely requirement; Outstanding¼ 8,
above likely requirement, either already obtained or predicted
(in which case supporting evidence was required from earlier
examination results). Only applications with a HYMS academic
score 40 were considered further. The mean academic score
in the study cohort was 6.1 (range 2–8, median 5 and mode 8).
Personal statement and referee’s report. The remain-
ing UCAS applications were assessed by two independent
readers using a structured information sheet to assess the
referees’ comments and record evidence of a realistic under-
standing of medicine, examples of self-motivation, acceptance
of responsibility, communication skills and team working, and
any other distinctive attributes (e.g. social disadvantage or
exceptional sporting achievement). The data were then scored
by administrative staff. The average score per reader was 25
(range 17–35).
The HYMS academic score and mean of the two reader
scores were added together to give the UCAS form score,
maximum 50 points.
Interview scores. The applicants with the top UCAS form
scores were invited to interview. Structured, scripted 20-
minute interviews were conducted by two interviewers,
yielding a maximum possible interview score of 50, compris-
ing 40 points based on the answers to eight questions and 10
points from an overall assessment of suitability scored on a 0 to
10-point visual analogue scale.
The UCAS form score and interview score were added to
contribute equally to the final selection score on which
applicants were ranked; places were then offered to the top
scoring candidates.
Potential selection instruments
Several standardised instruments were administered to this
cohort as part of this study, but were not used in selection.
Traits and skills measured by the cognitive UK Clinical
Aptitude Test (UKCAT) and non-cognitive tests (Personal
Qualities Assessment, PQA; Resilience Scales Questionnaire,
RSQ; Trait Emotional Intelligence Questionnaire, TEI) are
written in italics. Combined traits derived from these are
written IN ITALIC CAPITALS.
Cognitive tests
UKCAT. The UKCAT is a mandatory standardised test of
cognitive ability for those applying to study medicine at the
majority of UK medical schools. It was first taken in 2006, by
applicants to courses starting in 2007 (see www.ukcat.ac.uk).
UKCAT results were available for 131 students; 13 had applied
the year before the test was introduced, one student was
exempt and one result was unavailable. The UKCAT
comprised four cognitive skills subtest scores: verbal reason-
ing (VR), numerical reasoning (NR), abstract reasoning (AR)
and decision analysis (DA) (Childs 2012) and a total score. The
UKCAT scores were not revealed to the HYMS selectors at any
stage and did not inform selection decisions.
Non-cognitive tests
PQA. This comprised three tests:
(a) The Interpersonal Traits Questionnaire, which measures
the traits narcissism, aloofness, confidence (in dealing
with people) and empathy and produces a summary
score for the combined trait INVOLVEMENT (versus
DETACHMENT) in which confidence and empathy are
positive, narcissism and aloofness negative (Munro et al.
2005).
(b) The Interpersonal Values Questionnaire, which measures
the extent to which the respondent favours individual
freedoms (versus societal rules) as the basis of their moral
orientation (Bore et al. 2005).
(c) The Self-Appraisal Inventory, which measures the com-
bined traits EMOTIONAL RESILIENCE (comprising scales
measuring anxiety, moodiness, neuroticism and irra-
tional thinking) and SELF-CONTROL, in contrast to risk
taking tendency, (using the scales of restraint, conscien-
tiousness, permissiveness and anti-social tendencies). The
inventory also contains a Lie scale (Bore et al. 2009;
www.pqa.net.au).
RSQ. RSQ is a self-report questionnaire that identifies six
cognitive, behavioural and affective components, named
self-esteem, optimism, self-discipline, control, emotional non-
defensiveness and image management (impression manage-
ment) (Childs 2012).
TEI. The TEI Short Form is a 30-item questionnaire designed
to measure the global trait EMOTIONAL INTELLIGENCE
(Petrides 2009; Cooper & Petrides 2010).
The non-cognitive tests were all delivered in a paper-based
format under examination conditions at the University of Hull
and the University of York; PQA and TEI in October 2007 and
RSQ in October 2008.
Tutor ratings and grades
Problem-based learning tutors in Years 1 and 2 met with their
students twice weekly for 90 minutes. They assessed the
individual students’ interpersonal skills and professional
behaviours, which were recorded on standard scales in Year
1 (May 08) and Year 2 (January 09 and May 09), as described
in Adam et al. (2012). ‘‘Tutor ratings’’ represent the sum of the
scores for each of 14–17 defined skills or behaviours. At the
same time, these experienced problem-based learning tutors
made a global assessment, grading each student as either
Predictors of outcome at medical school
3
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
‘‘problematic’’, ‘‘average’’ or ‘‘particularly promising’’. This is
the ‘‘tutor grade’’.
Course summative assessments
Course assessments were undertaken in accordance with
HYMS’ Code of Practice on Assessment and Examination for
MB BS in Phases II & III. There was no summative assessment
in Year 3. The Year 4 and Year 5 summative assessments are
summarised below.
Year 4 written examination. Marks in the written examin-
ations at end of academic year 4 were allocated to one of two
HYMS themes: Theme A (Life sciences and Clinical sciences)
and Theme C (Evidence-based decision-making, Population
health and medicine and Managing resources for quality and
efficiency). The papers were mapped to course outcomes
including those from pharmacology and therapeutics. Theme
C tested not only application of clinical knowledge, but also
analytic and numerical evaluation skills across a range of
medical contexts. Theme B (Clinical Techniques and Skills;
Person-Centred Care) was examined in Year 4 only by the
clinical examinations.
Year 5 written examination. The written examination at
the end of academic year 5 was an integrated paper covering
all aspects of the course (Themes A, B and C). The questions
were based on clinical scenarios, each one orientated around a
common management problem including preventive strate-
gies. Therapeutic issues were a major focus, but other
management issues were also examined in this paper. The
pass mark was determined by the Hofstee method (McKinley
& Norcini 2014). Students sat the paper in March and those
who did not achieve the pass mark took another paper in May.
This was an independent paper but it was not considered a
resit. If the students passed at the second attempt, then they
passed the written examination.
Year 4 and Year 5 clinical examinations. The Year 4 and
Year 5 clinical examinations used a number of Objective
Structured Long Examination Records (OSLERs), each being a
45-minute observed clinical assessment in which the student
met and talked to a real patient, undertook appropriate
examinations, spent 15 minutes alone preparing a written
summary and plan, and then discussed this with the two
examiners and the patient. Student performance was assessed
in four categories of competence in Year 4: gathering
information, clinical examination, problem solving, and
relationship with patient. A fifth category, patient manage-
ment, was added in Year 5.
The Year 5 examinations also included Objective Structured
Clinical Examinations (OSCEs), which were 7-minute stations
assessed by direct observation by one examiner, the majority
designed to test students’ high level ‘‘communication skills’’
using simulated patients.
Practical clinical procedures were not routinely included
in the clinical examinations. Each student was required to
have reached a satisfactory standard in every specified
practical clinical procedure, assessed earlier in a controlled
clinical environment, before being allowed to take the
OSCEs.
The HYMS clinical examinations used a sequential design
and a non-compensatory marking system, fully described
elsewhere (Cookson et al. 2011). The key features are
summarised below. The sequential design required all students
to take the first part of the examination. The best performing
candidates (usually around 70%) were found to be clearly
satisfactory at this point. The remaining approximately 30%
were required to take the second part of the examination; the
additional results from the second part, together with the
results from the first part, were summed to give increased
reliability in determining which side of the pass/fail boundary
each student lay. Around 5–8% eventually failed the examin-
ation. The Year 4 clinical examinations consisted of two
OSLERs in the first part and three OSLERs in the second part.
The Year 5 examinations comprised four OSLERs and six
OSCEs in the first part, and the same number in the second
part. Four of the six OSCEs in the first part and five of the six
OSCEs in the second part addressed communications skills; the
remainder involved a written task or skill.
Grading and marking system
Each category of competence examined in the OSLERs, and
every OSCE, was graded by the examiners using the following
grade descriptors shown in Table 1.
The outcome of these examinations was pass/fail only. On
the basis that candidates should not be able to compensate for
Table 1. Grade descriptors, PPs and scores for OSLERs and OSCEs.
Grades DescriptorPenalty pointsYear 4
Penalty pointsYear 5
Researchscore
A Capable in all components to a high standard – – 6
B Capable in all components to a satisfactory standard and
a high standard in many
– – 5
Cþ Capable in all components to a satisfactory standard – – 4
C� Capable in a majority of components to a satisfactory
standard, inadequacies in some components
– 1 3
D Capable in a minority of components.
No serious defects
2 2 2
E Capable in a minority of components.
One or more serious defects
3 3 1
J. Adam et al.
4
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
a serious deficiency in one area by a high standard in another,
grades below satisfactory were converted to PPs, and only the
PPs were used to make the pass/fail decisions. Candidates
accruing more than a fixed number of PPs in the first part were
required to take the second part of the examination. HYMS
determined the examination outcome from the total number of
PPs accrued, across either the first part or both parts of the
examination if taken, to give a pass/fail result. The grades A to
E form an ordinal categorical scale (Cookson et al. 2011), so for
the purpose of this study they were converted to positive
scores, as shown in Table 1. For this study we have expressed
all examination data as an average per case, derived from the
scores from the first part of the examination, and from the
second part if taken, whether positive scores or PPs. These are
designated OSLER score, OSCE score, OSLER PPs and OSCE
PPs. A further summary score and penalty point measure was
calculated for the entire Year 5 clinical examination by adding
together the OSLER and OSCE scores (equally weighted) and
the OSLER and OSCE PPs (equally weighted), called OSLER þOSCE sum score and OSLER þ OSCE sum PPs, respectively.
The maximum number of possible PPs (designated Clinical
Examination penalty points, CEPPs) from all the Year 5
OSLERs and OSCEs adding the first part and the second part
together was 276; the observed range in this cohort was 0 to
56, mean 11.04, median 6, mode 4.
Other outcome measures
Other relevant outcome measures detailed here include
subscales derived from the summative examinations that
were considered particularly relevant to explore for this
study, such as communication skills and Honours awards,
and empathy scores provided as formative examination
feedback to the students.
OSLER communication scores. The OSLER competence
categories ‘‘gathering information’’ and ‘‘relationship with
patient’’ rely heavily on communication skills. The scores
and PPs accrued for these competences alone were added
together and a mean score per case calculated as described
above, to give the OSLER communication score and OSLER
communication PPs.
OSCE communication scores. Four of six OSCE stations in
the part 1 examination and 5 of 6 in the part 2 examination
assessed communication skills in challenging situations; the
mean score and PPs per case accrued from only these stations
yielded the OSCE communication score and OSCE communi-
cation PPs.
OSCE empathy scores. At each communication OSCE
station, the examiner and simulated patient undertook inde-
pendent assessments of the student’s perceived empathy,
graded A to E, ranging from ‘‘excellent’’ to ‘‘poor’’ empathy
skills (Wright et al. 2014). These grades were collected for
research, and provided to students as formative feedback. For
the present study the grades were converted to empathy
scores (A, B, C, D, E¼ 5, 4, 3, 2, 1 points) and PPs (D, E¼ 1, 2
PPs) and used to calculate a mean OSCE Empathy score and
OSCE Empathy PPs per case, using data from both parts of the
examination.
The clinical examination variables thus fall into two
categories, those that give positive scores for performance,
and those that give negative scores for deficient performance.
Better performance is denoted by higher totals in the OSLER
score, OSLER communication score, OSCE score, OSCE
communication score, OSCE Empathy score and OSLER þOSCE sum score. The indices of deficient clinical performance
are OSLER PPs, OSLER communication PPs, OSCE PPs, OSCE
communication PPs, OSCE Empathy PPs, OSLER þ OSCE sum
PPs and clinical examination PPs.
Criteria for graduation with honours
HYMS applied specific criteria for recommending students for
graduation with Honours, based on weighted overall perform-
ance in summative examinations and student projects through-
out the course. For this cohort approximately 6% were
awarded Honours.
Fitness to practise penalty points
The final important outcome measure was FTPPPs. The HYMS
Fitness-to-Practise committee received confidential reports (in
which students were identified only by number) not only
about serious lapses in professional behaviour, but also about
lower level concerns arising from structured formative end-of-
block reviews. These reviews were undertaken regularly
between student and their current clinical tutor at approxi-
mately two monthly intervals throughout the entire course,
and included grading of the student’s professional behaviour
(using a structured score sheet with clear grade descriptors),
either Excellent, Satisfactory, Borderline or Unsatisfactory,
under each of the following headings: ‘‘relationships with
patients’’; ‘‘awareness of ethical and moral aspects of sub-
ject’’; ‘‘ability to deal with uncertainty and awareness of
limitations’’; ‘‘evidence of self-education, enthusiasm and
motivation’’; ‘‘teamwork’; ‘‘dress, attendance and punctuality’’.
A ‘‘borderline’’ or ‘‘unsatisfactory’’ grade in any aspect of
professional behaviour was automatically notified to the
Fitness-to-Practise Committee. This allowed HYMS to identify
and react when students showed repeated patterns of
undesirable behaviour.
J. A. reviewed all the reports of the HYMS Fitness-to-
Practise committee covering the HYMS academic years 2007–
2013 inclusive, and summarised every instance concerning a
member of the study cohort. Each instance was given one or
more FTPPPs devised for this study; the total number of points
accrued by each student across the years of the course was
then recorded. For example, a professional behaviour grade of
‘‘borderline’’ was given 1 FTPPP and an ‘‘unsatisfactory’’ grade
was given 2 FTPPPs. Other misdemeanours were awarded one
or two points, after assessing their seriousness through careful
evaluation of all the relevant information including contem-
poraneous verbal accounts and written records. In total, 45
individuals were identified in Fitness-to-Practise records, of
whom 26 had only 1 FTPPP. There were long-term conse-
quences for a significant proportion of those accruing three or
Predictors of outcome at medical school
5
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
more FTPPPs, with referral to formal fitness-to-practise pro-
cedures that can impose serious penalties, the most serious
being either a formal warning reported to the UK General
Medical Council on registration as a doctor, or being required
to leave the course.
Data handling and statistical analysis
All data were anonymised and entered into an SPSS database.
All analyses utilised SPSS version 20 (Chicago, IL). Scores on all
multiple-component measures (OSCEs, OSLERs, communica-
tion and empathy measures, etc.) were computed and checked
for internal consistency. Except where noted (e.g. age) no
extreme skewness and no extreme outliers were detected.
Relationships between continuous variables were computed as
Pearson correlation coefficients (r); one-way analysis of
variance (ANOVA, F-statistic) was used to compare scores on
categorical variables (sex, citizenship, etc.) and on compari-
sons of extreme groups, such as top versus bottom 20% of
scores. Statistical significance was based on unadjusted
probabilities (p¼ 0.05, 0.01 or 0.001). Two-tailed tests of
significance were applied in most cases, except where a clear
directional relationship was predicted. In order to establish the
best overall predictors of outcomes, a series of linear regres-
sion analyses were undertaken and are described in the
Results section.
Results
The tables show all the statistically significant results. Table 2
shows the significant correlations found between the initial
predictor variables (available from the start of the course),
intermediate tutor evaluations (from Years 1 and 2), demo-
graphic factors and the outcomes from the Year 4 and Year 5
written and clinical examination results. We considered that
graduating with Honours or with notably high written and
clinical examination results were both desirable selection
outcomes, whereas failing to complete the course or having
documented instances of significant unprofessional behaviour
were undesirable selection outcomes. Table 3 shows which
predictors were significant when ANOVA was applied to three
pairs of comparisons based on positive exam marks: those
students who graduated with Honours (n¼ 11) versus those
who left the course (n¼ 9); those students with Year 5 written
exam scores in the top quintile versus the bottom quintile;
Table 2. Predictors of examination scores.
Initial predictors Intermediate predictors Demographic predictors
HYMS academic UKCAT PQA RSQ HYMS tutors
Years 1 & 2
examinations Age Sex UK/Non-UK
Written exams
Year 4 theme A r 0.174* Jan 09g r 0.261**
May 09g r 0.257**
08þ09g r 0.236**
f (F)**
Year 4 theme C r 0.200* Tot r 0.181* Jan 09g r 0.246**
May 09g r 0.218*
08þ09r r 0.194*
08þ09g r 0.239**
y (F)* UK (F)*
Year 5 final exam Tot r 0.175*
AR r 0.231**
f (F)* UK (F)*
Clinical exams
Year 4 OSLER score ASoc r �0.181* 08þ09r r 0.179*
Year 4 OSLER communication
score
o (F)* f (F)*
Year 5 OSLER score r 0.213* Lie r 0.198*
Emp r 0.257**
En-d r 0.219* Jan 09g r 0.176*
May 09r r 0.176*
May 09g r 0.197*
08þ09g r 0.197*
y (F)** f (F)**
Year 5 OSLER Communication
score
r 0.215* Conf r 0.192*
Emp r 0.260**
(TEI r 0.201)* f (F)*** UK (F)*
Year 5 OSCE score r 0.170* Tot r 0.204*
VR r 0.244**
AR r 0.250**
y (F)** UK (F)**
Year 5 OSCE Communication
score
Jan 09r r 0.185*
May 09r r 0.188*
Year 5 empathy score UK (F)*
Year 5 OSLERþOSCE
sum score
r 0.256** Emp r 0.194* May 09r r 0.191*
May 09g r 0.190*
Year 1A r 0.254**
Year 1B r 0.228**
Year 1C r 0.215*
Year 1T r 0.296***
Year 2A r 0.301***
Year 2B r 0.337***
Year 2C r 0.271**
Year 2T r 0.359***
y (F)* f (F)* UK (F)**
Analysis of variance (F statistic) for categorical variables and Pearson correlations (r) for continuous variables, *p50.05; ** p50.01; *** p50.001 (N¼131–146;
significance levels are not adjusted for repeated comparisons).
UKCAT Tot, total UKCAT score; VR, verbal reasoning; AR, abstract reasoning; PQA Antisoc, anti-social tendencies; Lie, lie scale score; Emp, empathy; Conf,
confidence; RSQ En-d, emotional non-defensiveness; TEI, test of emotional intelligence. HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2
(January 09, May 09). Years 1 & 2 exams: themes A, B and C, and totals, T, of AþBþC, y: younger; o: older at entry; f: female; UK: UK citizen.
J. Adam et al.
6
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
those students with Year 5 clinical examinations (OSLER þOSCE sum) scores in the top quintile versus the bottom
quintile. Table 4 shows the significant predictors of deficient
clinical performance or undesirable outcomes, from correl-
ations with clinical examination PPs, and from comparisons of
three groups of students achieving undesirable outcomes:
those who left the course versus the rest; those who left the
course grouped with those who gained 3 or more FTPPPs
versus the rest, and those who gained any FTPPPs versus those
who did not. Finally, Table 5 shows the outcome of the
multiple regression analyses to determine the best predictors
of Year 4 and Year 5 outcomes.
Predictors of achievement and progress
Demographic data
Age. Students under 21 at the time of entry performed
better than older students in the Theme C component of the
Year 4 written examination and in the Year 5 clinical
examinations (Year 5 OSLER, Year 5 OSCE). Older students
outperformed their younger peers only in the Year 4 OSLER
examination, specifically in the ‘‘communication’’ segments
(Table 2).
Sex. Females performed better than males in the Theme A
component of the Year 4 written examination and in the
Year 5 final written examination (an integrated examination
covering Themes A, B and C). They outperformed males also
in Year 4 and Year 5 OSLERs, and achieved a better overall
score in the combined Year 5 OSLERþOSCE sum score
(Table 2). They received fewer OSLER and OSCE PPs than
their male peers (Table 4). Females were better represented
than males in the top 20% of achievers in the final year, in both
written and clinical examinations (Table 3). Males did not
outperform females in any component of the Year 4 and 5
examinations.
Citizenship. Non-UK citizens performed at a lower level in
Year 4 and 5 written examinations and the Year 5 clinical
examinations than UK citizens (Table 2), and gained signifi-
cantly more PPs and penalty marks in all of the clinical
examinations (Table 4).
HYMS selection process
The HYMS academic score, an index of prior academic
performance, was the only item used in the selection
procedure that was a useful, significant predictor of progress,
correlating with a number of assessment outcomes. Written
exam performance (Year 4, Themes A and C) and clinical
exam performance were predicted (Year 5 OSLER and Year 5
OSCE) (Table 2). The HYMS academic score also correlated
(negatively) with the Year 5 OSLERþOSCE sum PPs and the
total clinical exam penalty points (CEPPs) (Table 4). No other
Table 3. Predictors of top and bottom performers.
Initial predictors Intermediate predictors Demographic predictors
HYMSacad UKCAT PQA RSQ HYMS tutors Years 1&2 exams Age Sex UK or non-UK
Graduation
with Honours
versus left course
May 08r (F)*
Jan 09g (F)**
May 09g (F)***
Year 5 written examination Tot (F)* Yr 1A (F)*** f (F)*
Top 20% versus bottom 20% VR (F)**
AR (F)**
QR (F)**
Yr 1C (F)***
Yr 1 T (F)***
Yr 2 OSCE (F)*
Yr 2A (F)***
Yr 2B (F)***
Yr 2C (F)***
Yr 2 T (F)***
Year 5 clinical examination
(OSLERþOSCE sum score)
Top 20% versus bottom 20%
(F)* Mood (F)*
Consc (F)*
Conf (F)**
(TEI (F)*)
En-d (F)*
May 09r (F)*
May 09g (F)*
08þ09r (F)*
08þ09g (F)*
Yr 1 OSCE com (F)**
Yr 1 OSCE prac (F)*
Yr 1 OSCE T (F)**
Yr 1A (F)**
Yr 1B (F)**
Yr 1C (F)*
Year 1 T (F)***
Yr 2 OSCE (F)***
Yr 2A (F)***
Yr 2B (F)***
Yr 2C (F)**
Year 2 T (F)***
y (F)* f (F)**
Analysis of variance (F statistic) for categorical variables, *p50.05; **p50.01; ***p50.001 (N¼131–146; significance levels are not adjusted for repeated
comparisons).
HYMS acad: computed academic entry score, see Methods section; UKAT Tot, total UKCAT score; VR, verbal reasoning; AR, abstract reasoning; QR, quantitative
reasoning; PQA Mood, moodiness; Consc, conscientiousness; Conf, confidence. RSQ En-d, emotional non-defensiveness. TEI, test of emotional intelligence.
HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2 (January 09, May 09). Year 1 & 2 exams: themes A, B and C, and total, T, of AþBþC, and
OSCE practical and communication skills stations and sum total, T, of ‘‘prac’’ and ‘‘com’’. y: younger age at entry; f: female.
Predictors of outcome at medical school
7
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
variables derived from the UCAS form personal statement &
referees’ report score, nor variables from the HYMS interview
question scores and visual analogue scale, were useful
predictors; indeed, ‘‘interview score’’ correlated significantly
only with Year 4 OSLER communication PPs, but in the
unexpected direction (Table 4).
Cognitive tests: UKCAT sub-scores and total score
UKCAT total score predicted some written exam scores (Year
4, Theme C and Year 5) and clinical performance assessed by
Year 5 OSCE. Some UKCAT subtest scores (Abstract
Reasoning and Verbal Reasoning) were also significantly
correlated with the written and some clinical exam scores in
Year 5, but not Year 4 (Table 2).
Non-cognitive tests: PQA
Many significant correlations were found between some trait
and combined trait scores and clinical exam scores, CEPPs and
FTPPPs. In contrast, none of the PQA test scores predicted
written exam performance in Years 4 and 5 (Table 2). The
PQA traits confidence, conscientiousness, anti-social tenden-
cies, empathy, moodiness, neuroticism and aloofness and the
combined trait INVOLVEMENT were shown to be predictors of
clinical examination scores (Tables 2 and 3) and of PPs in
these exams (Table 4).
Non-cognitive Tests: RSQ
Like PQA, some RSQ components (emotional non-defensive-
ness, self-esteem and optimism) predicted clinical performance
both good and bad, but not written exam performance
(Table 2).
Intermediate predictors: In-course Tutor ratings and
grades
Tutor evaluations (both ratings and grades) of students’
capabilities and deficiencies made during three terms in
Table 4. Predictors of undesirable outcomes.
Initial predictors Intermediate predictors Demographic predictors
HYMS
acad
HYMS
interview UKCAT PQA RSQ HYMS tutors
Years 1&2
exams Age Sex UK or non-UK
Year 4 OSLER PPs 08–09g r �0.192*
May 08g r �0.185*
Year 4 OSLER
Communication PPs
r 0.190* Neurot r �0.177* E n-d r 0.211*
Year 5 OSLER PPs r �0.187* Emp r �0.255**
Inv r �0.187*
m (F)* Non (F)*
Year 5 OSLER
Communication PPs
Emp r �0.241** m (F)***
Year 5 OSCE PPs VR r �0.218**
AR r �0.274**
QR r �0.261**
Tot r �0.304***
Conf r �0.198* Self-est r �0.220*
Optm r �0.231*
Non (F)*
Year 5 OSCE
Communication PPs
VR r �0.177*
AR r �0.196*
Conf r �0.195* May 08g r �0.175*
08–09g r �0.187*
Non (F)*
Year 5 OSCE
Empathy PPs
Non (F)*
Year 5 CEPPs r �0.235** Emp r �0.369***
Inv r �0.297**
08–09r r �0.177* m (F)* Non (F)*
Year 5
OSLERþOSCE
sum PPs
r �0.208* VR r �0.214* Emp r �0.179* Yr 1A r �0.251**
Yr 1B r �0.180*
Yr 1C r �0.230**
Yr 1T r �0.284***
Yr 2A r �0.288***
Yr 2B r �0.338***
Yr 2C r �0.275***
Yr 2T r �0.352***
m (F)** Non (F)**
Left course
(versus or not)
Antisoc (F)* Optm (F)* May 08 (F)*
Jan 09r (F)*
Jan 09g (F)**
May 09r (F)**
May 09g (F)**
08–09r (F)*
Yr 2C (F)***
Yr 2T (F)*
Yr 2 OSCE (F)**
o (F)**
Left course or43
FTPPPs versus
‘‘not’’ and ‘‘none’’
Antisoc (F)* Jan 09r (F)**
08–09r (F)*
Yr 2 OSCE (F)*
Yr 2C (F)**
Yr 2 T (F)*
o (F)*
FTPPPs (versus none) Aloof (F)* Image (F)* Yr 1B (F)*
Yr 1 OSCE (F)*
Yr 2B (F)*
Yr 2C (F)**
m (F)*
Analysis of variance (F statistic) for categorical variables and Pearson correlations (r) for continuous variables, *p50.05; **p50.01; ***p50.001 (N¼ 131–146;
significance levels are not adjusted for repeated comparisons).
UKCAT Tot, total UKCAT total score; VR, verbal reasoning; AR, abstract reasoning; QR, quantitative reasoning; PQA Antisoc, anti-social tendencies; Aloof, aloofness;
Emp, empathy; Conf, confidence; Neurot, neurotic; Inv, INVOLVED. RSQ En-d, emotional non-defensiveness; Optm, optimism; Image, managing own image.
Self-est, self-esteem. HYMS tutors: ratings, r, and grades, g, given in years 1 (May 08) and 2 (January 09, May 09). Year 1&2 exams: themes A, B and C, and totals,
T of AþBþC, and OSCE; o: older at entry; m: male; f: female; UK/Non: UK citizen or non-UK citizen. PP: penalty points; FTPPP: fitness to practise penalty points.
J. Adam et al.
8
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
Years 1 and 2 were not only significant predictors of many
components of written and clinical examination performance
in Years 4 and 5 (Table 2), but also significant predictors of the
groups of students who gained Honours or left the course
(Tables 3 and 4).
Intermediate predictors: Year 1 and Year 2 examina-
tion performance
Year 1 Themes A, B and C results and Year 2 Themes A, B and
C results each individually predicted Year 5 OSLERþOSCE
sum score (Table 2) and Year 5 OSLERþOSCE sum PPs (Table
4). Themes B and C were also found to significantly predict
who left the course and FTPPPs (Table 4). The top and bottom
20% of achievers in Year 5 written and clinical examinations
(OSLER & OSCE) were generally predicted by Year 1 and 2
results (Table 3). However, Year 1 and 2 examination results
did not predict graduation with Honours or leaving the course
(Table 3).
Desirable and undesirable outcomes
Graduation with honours
By 2013, nine individuals (6%) from the study sample had
graduated with honours, seven of whom were aged 521 at
entry, three were males and eight were UK citizens. Year 1
and 2 tutor ratings (r) and grades (g) were the only significant
independent predictors of ‘‘graduation with honours’’
(Table 3).
Leaving the course
Eleven individuals from the study sample (8%) left the course
without completing their medical degree; six (55%) were aged
521 at entry, two (18%) were males and eight (72%) were UK
citizens. Those who left scored more highly on RSQ optimism
and PQA anti-social tendencies. Year 1 and 2 tutor ratings and
grades (May 08, January 09 and May 09) were consistent
predictors of those who left the course (Table 4).
Comparison of those who gained Honours with those
who left the course
The only significant predictors of leaving the course versus
graduating with honours were Year 1 (May 08) tutor ratings
and Year 2 (January 09, May 09) tutor grades (Table 3).
Comparison of top 20% with the bottom 20% of
achievers in the Year 5 final examinations
Year 5 written examination. Being among the top 20% of
achievers was significantly associated with being female, or
having a higher UKCAT total score (and better UKCAT
quantitative reasoning, verbal reasoning and abstract rea-
soning scores). Students who had performed to a higher
standard in Year 1 and Year 2 examinations were significantly
more likely to be represented in the top scoring group in the
Year 5 written examination (Table 3).
Year 5 clinical examination. The following variables
significantly indicated the likelihood of being among the top
20% of achievers: being female, younger and scoring better in
some of the non-cognitive tests (PQA: lower moodiness, higher
conscientiousness and higher confidence; RSQ: higher emo-
tional non-defensiveness; TEI: higher emotional intelligence).
Better tutor ratings and grades in Year 1 and 2 predicted higher
achievement, as did better overall marks and individual
components by theme of Year 1 and Year 2 examinations
(Table 3).
Comparison of those with extreme high or low non-
cognitive PQA scores with central scores
Bore et al. (2009) proposed that extreme scores (both high and
low) on the PQA non-cognitive tests would be likely to predict
low scores on positive course outcomes and/or high scores on
negative course outcomes. In order to test this hypothesis,
outcome data from the students in the top and bottom 20% of
scores on each non-cognitive PQA trait were combined and
compared with the middle 60%, using analysis of variance. The
hypothesis was confirmed for the PQA combined trait
INVOLVEMENT in relation to the Year 5 OSCE score
(p¼ 0.036), Year 5 OSCE empathy score (p¼ 0.042) and also
Year 5 OSCE empathy PPs (p¼ 0.035).
Comparison of those reported to the fitness-to-practise
committee with the rest of the cohort
None of the HYMS selection criteria predicted incidents or
behaviours of concern to the Fitness-to-Practise committee.
Males were significantly more likely to gain FTPPPs, but
neither age at entry nor UK citizenship were significant
predictors. Those students who had underperformed in
components of the Year 1 and Year 2 examinations were
more likely to be those who gained FTPPPs. Neither UKCAT
sub-scores nor total scores predicted FTPPPs, but PQA
aloofness and RSQ managing image did predict FTPPPs
(Table 4).
Individual characteristics of the subgroup of students
reported to the fitness-to-practise committee
Overall, 45 students gained FTPPPs. ANOVA (F-test) showed
that several predictors characterised this subgroup: UKCAT
higher verbal reasoning scores (p¼ 0.016); PQA lower con-
scientiousness (p¼ 0.023); PQA higher impulsiveness
(p¼ 0.041); PQA higher confidence (p¼ 0.015); RSQ lower
self-discipline (p¼ 0.015); RSQ lower control (p¼ 0.011).
Lower Year 1 and 2 tutor ratings were also a significant
predictor (sum of May 08þ January 09þMay 09 ratings;
p¼ 0.018). A stepwise regression which included all of the
above predictors found that RSQ control was the predominant
predictor (beta¼�0.539; p¼ 0.004). The subgroup with
FTPPPs was also associated with poorer clinical performance:
in Year 4 a lower OSLER score, p¼ 0.004; lower OSLER
communication score, p¼ 0.004; more OSLER PPs and OSLER
communication PPs, p¼ 0.000 and p¼ 0.003, respectively, and
in Year 5 more OSCE PPs (p¼ 0.010) and clinical examination
PPs (p¼ 0.018).
A further comparison of interest from the selectors’
perspective is reported in Table 4. The worst selection
outcomes are students who either leave the course or manifest
Predictors of outcome at medical school
9
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
serious professional misbehaviour. We therefore grouped
together those students who had either left the course
prematurely or who had gained three or more FTPPPs
(n¼ 22), and compared them with the rest of the cohort, to
look for significant predictors of these undesirable outcomes.
This group were older at entry, had scored higher on the PQA
anti-social tendencies measure, and all had scored lower on
the summed Year 1 and 2 tutor ratings (May 08þJanuary
09þMay 09). These students had also performed less well in
components of the Year 2 examination (Table 4).
Best predictors of Year 4 and Year 5outcomes: Regression analyses
To clarify the complex relationships between predictors and
outcomes, a series of linear regression analyses were under-
taken using the SPSS routines, which revealed that the best
overall predictors of all outcomes were the demographic
variables (age, sex and citizenship), the prior ‘‘ability’’ indica-
tors (HYMS academic score and total UKCAT score) and the
non-cognitive test scores for empathy (PQA) and emotional
non-defensiveness (RSQ). The apparent selection power of the
demographic variables is likely to be mediated by other
confounding variables, so the two ability variables (the most
readily available for selection purposes) were entered into the
first stage of a multiple regression analysis, and the two non-
cognitive variables (which might be added to the selection
procedures) in the second stage. The results are shown in
Table 5, which also shows the percentage of variance in each
key outcome variable accounted for by the ability and non-
cognitive predictors. In a third stage, the three demographic
variables were then entered, to show how much additional
variance might be accounted for by the other factors they
represent.
The results show that the prior ability variables alone would
be useful predictors of Year 4 examinations, Year 5 OSCE and
OSLER scores and most of the associated penalty point scores,
but not the Year 4 OSLERs or Year 5 OSCE empathy score. The
HYMS academic score appears to be a better predictor than
the UKCAT total, which in the case of Year 5 OSLER scores
showed a negative relationship with the outcomes. The
addition of the non-cognitive variables would make a signifi-
cant contribution in the case of the Year 5 OSLER scores and
PPs, with PQA empathy and RSQ emotional non-defensiveness
contributing about equally. The difference between the value
of the adjusted R-squared (percentage of the variance) at stage
1 and 2 and after stage 3 shows that the additional
demographic variables, particularly sex and citizenship,
would attenuate the predictive usefulness of the prior ability
and non-cognitive variables.
Discussion
The aim of our study was to examine what student attributes
and qualities, or combination thereof, best predicted outcomes
of medical education. This was undertaken longitudinally in
the context of a specific medical school cohort with predictors
being measured some four to five years prior to measurement
of the outcome variables. The approach was not hypotheses-
driven but exploratory thus allowing the data to reveal
relationships between variables. The principal findings were:
Table 5. Best selectors for Year 4 & Year 5 examination outcomes.
1st stage 2nd stage 3rd stage
Outcome variableUCASacad
UKCATtotal
PQAempathy
RSQe n-d
Stages 1þ2adjusted
R-squared(% of variance) Sex Age UK/Non
Stages 1þ2þ3adjusted
R-squared(% of variance)
Years 4 & 5 – positive outcomes
Year 4 Theme A exam 0.22* 0.08 0.05 0.12 4.3 0.30** 0.00 0.14 10.9
Year 4 Theme C exam 0.31** 0.04 0.06 0.14 8.8 0.14 0.14 0.15 10.5
Year 4 OSLER total �0.02 �0.07 0.13 �0.12 0.0 0.22* 0.19 0.15 2.9
Year 4 OSLER communication �0.01 �0.11 0.01 �0.16 0.0 0.29** 0.22* 0.11 8.0
Year 5 OSCE total 0.14 0.18 0.05 0.02 2.3 �0.04 �0.22* 0.24* 11.6
Year 5 OSCE communication 0.21* 0.05 0.01 0.18 4.2 0.01 0.08 0.17 4.1
Year 5 OSLER total 0.30** �0.22* 0.21* 0.25** 20.1 0.19* 0.12 0.16 23.3
Year 5 OSLER communication 0.24* �0.29** 0.21* 0.20* 17.7 0.26** 0.13 0.16 23.7
Year 5 OSLERþOSCE sum score 0.29** �0.01 0.17 0.17 11.2 0.09 �0.08 0.28** 17.9
Year 5 OSCE empathy score 0.11 0.01 �0.05 0.11 0.0 0.07 0.08 0.21* 1.4
Years 4 & 5 – penalty points
Year 4 OSLER PPs 0.00 0.17 �0.01 0.11 0.0 �0.12 �0.20 �0.06 0.0
Year 4 OSLER communication PPs 0.09 0.14 �0.04 0.28** 6.8 �0.13 �0.06 0.08 6.0
Year 5 OSCE PPs �0.07 �0.31** �0.04 �0.06 7.6 �0.10 �0.17 �0.18 9.6
Year 5 OSCE communication PPs �0.06 �0.13 �0.14 �0.03 0.0 �0.05 �0.13 �0.21* 0.2
Year 5 OSLER PPs �0.24* 0.07 �0.18 �0.29** 15.6 �0.08 �0.07 �0.23* 18.2
Year 5 OSLER communication PPs �0.18 0.10 �0.26* �0.18 11.2 �0.24* �0.03 �0.15 15.0
Year 5 Clinical exam PPs �0.21* �0.01 �0.39** �0.08 16.3 �0.06 �0.16 �0.23* 19.7
Year 5 OSCE Empathy PPs �0.10 �0.01 0.01 �0.10 0.0 0.06 0.03 0.13 0.0
Year5 OSLERþOSCE sum PPs �0.20* �0.15 �0.21* �0.16 12.2 �0.11 �0.15 �0.27** 17.6
Cells in columns 2–5, 7–9 contain standardised coefficients; cells in columns 6 and 10 give percentage of variance accounted for.
Statistical significance: *50.05 **50.01 (t-test, two-tailed), N¼131–146. Abbreviations as elsewhere.
J. Adam et al.
10
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
Prior academic achievement was related to written Year 4
(but not Year 5) exams, and also to Year 5 (but not Year 4)
clinical exams and penalty marks.
Being younger, or female, or having UK citizenship were
each associated with better performance on many outcomes.
Cognitive reasoning ability, as assessed by the UKCAT
tests, was related to the Year 5 written exam and OSCE
performance.
Non-cognitive variables were not significantly related to
written exams, but were shown to be related to performance in
clinical examinations and other aspects of the course in a
number of ways, both positive and negative, as summarised
below:� Measures of empathy, emotional non-defensiveness,
confidence and emotional intelligence were related to
Year 5 OSLER performance.� Students with high Year 5 OSLER and OSCE PPs tended to
have low empathy and low confidence, with low self-
esteem and low optimism also associated with low Year 5
OSCE scores.� Conscientiousness, confidence, moodiness, emotional
intelligence and emotional non-defensiveness all differ-
entiated the top 20% of Year 5 clinical examination
achievers from the bottom 20%.� Anti-social tendencies and optimism were related to
leaving the course, while aloofness and (poor) image
management related to FTPPPs.
Two general observations can be made. First, the correl-
ations between the different assessment outcomes within the
HYMS course were generally strong, indicating a high degree
of assessment coherence. Second, our findings suggest that
tutors have considerable insight into their students’ behaviour
which eventually correlates with their academic and profes-
sional performance. Tutor ratings in Years 1 and 2 were
positively related to Year 4 performance in written exams,
Year 5 OSLERs, and graduating with Honours, and negatively
related to leaving the course. This consistency may depend on
HYMS’ use of experienced clinicians as PBL tutors, a role tutors
had chosen to undertake as part of a ‘‘portfolio career’’. The
combination of early examination results and such tutor
assessments could thus be useful for identifying those students
who might benefit from targeted professionalism mentoring as
well as academic interventions later in the course.
The breadth and depth of this study is, to date, unique. The
entire cohort of 146 students was tracked through the course,
94% of whom had completed all the initial non-cognitive tests.
The small size of the cohort is compensated to an extent by the
lack of range restriction in the cognitive and non-cognitive
parameters and the completeness of follow-up. The outcome
data covered not only examination results and tutor assess-
ments, but also evidence of problematic professional behav-
iour ranging from minor to serious, collected under HYMS’
system for monitoring fitness-to-practise. Although the large
number of relationships studied means that about 1 in 20 will
appear significant by chance at the 5% level, their consistency
with each other and with the underlying meanings of the
constructs leaves us confident in the validity of our findings.
This approach has yielded evidence that measures of past
academic performance, cognitive skills, and also personality
traits that reflect emotional engagement with people, plus a
positive disposition, all make weak but significant contribu-
tions to the prediction of a range of both desired and
undesirable outcomes from the medical degree course.
A large percentage of complaints about medical practi-
tioners arise from breakdowns in the doctor’s communication
with patients, relatives or colleagues rather than from failings
in procedural skills (see, e.g. the recent annual reports of the
General Medical Council of Great Britain). Communication is a
key skill for a doctor, but is difficult to quantify in students.
This prompted us, a priori, to devise various ways of
measuring different aspects of students’ communication
abilities using the clinical examination data, by breaking
down the marks into components directly related to commu-
nication skills. The different indices derived from the OSLER
and OSCE examinations were thus designed as important
investigative tools to delve into this difficult area, rather than
an attempt to proliferate statistical indices.
We expected to find strong relationships between prior
academic achievement and outcomes, given that McManus
et al. (2013) found that measures of past academic perform-
ance were the main predictors of future academic perform-
ance. This was indeed the case: prior academic performance
(the HYMS academic score) had significant predictive value for
some outcomes. However, none of the other student selection
parameters used by HYMS (UCAS personal statement, referees’
reports and interview scores) predicted performance in any of
the outcome measures in this study, although of course their
use as selection parameters inevitably introduces significant
range restriction.
In contrast, there was no restriction of range for the
cognitive tests and non-cognitive tests as they were not used to
select applicants into the cohort. The non-cognitive variables
that were found to be significant describe aspects of being able
to relate well to other people: empathy, confidence with
others, not being aloof, being able to manage one’s emotions,
being optimistic and having high self-esteem. This is consistent
with the findings of a number of recent studies (e.g., Lambe &
Bristow 2010, Haight et al. 2012, Koenig et al. 2013, Simpson
et al. 2014), which together suggest that such variables may
provide suitable proxy measures for the desired elements of
the ‘‘character’’ advocated by Smyth (1946).
Negative course outcomes such as leaving the courses or
having three or more FTPPPs were related to high scores on
the trait anti-social tendencies. The questions that contribute
to the anti-social tendencies score reflect a disregard for the
laws and norms of one’s society; perhaps, this predicted a
disregard for, or intolerance of, the rules and norms of the
medical educative experience leading to leaving the course or
earning PPs.
The variables of moral decision making (individual freedom
versus societal rules), and the traits of narcissism, anxiety,
irrational thinking, restraint, and permissiveness were found to
be unrelated to outcome as were the scores of self-discipline
and control. Given the numerous literature reports in both
medical education (e.g., Ferguson et al. 2014) and organisa-
tional psychology research that describe the importance of
conscientiousness and other related traits such as self-control
Predictors of outcome at medical school
11
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
and self-discipline, we had expected to find these traits better
predictors.
The relationships described here differ from those reported
when the cohort had completed Year 2 (Adam et al. 2012). For
example, in the first two years of medical education the
variables narcissism, aloofness and irrational thinking pre-
dicted poorer performance, while conscientiousness, confi-
dence and INVOLVEMENT predicted better performance in
Year 1 and Year 2 OSCEs. UKCAT overall scores and the
decision analysis subtest score also predicted Year 1 and Year
2 overall exam scores. Over the complete medical course (five
years) reported here, only the three UKCAT reasoning tests
and the traits of INVOLVEMENT (confidence plus empathy)
have remained as significant predictors.
The demographic factors co-varied with some of the
predictor variables, for example females producing higher
empathy scores compared to males (as is commonly found in
review, e.g. Munro et al. 2005; Wright et al. 2014) even though
there was no evidence of gender difference in prior academic
achievement. The likelihood that demographic factors (age,
gender, citizenship) were confounding variables is supported
by their accounting for additional variance in examination
outcomes over and above that due to ability and personality
variables (Table 5).
The present study demonstrates that it is possible to identify
attributes and qualities, or combination of attributes and
qualities, which might predict outcomes in the academic and
clinical domains from a medical education. As the ultimate
purpose of our research is to assist medical school selectors,
we have also addressed the question of whether the pre-
existing selection method (scrutiny of application form and
interview), other standardised selection tests of cognitive
abilities and non-cognitive traits, intermediate indicators from
the early medical school years (e.g. tutor evaluations) or
demographic variables can reliably predict those more likely to
achieve desirable versus undesirable outcomes.
In summary, our findings show that the qualities that
contribute to good outcomes from a medical course are
academic ability, reasoning ability and a stable and positive
personality. This study is the first to show that measurement of
non-cognitive traits, in combination with measures of aca-
demic and cognitive ability, can predict desirable (and
undesirable) outcomes from medical education. Refinement
of the specific traits and their measurement, together with
medical schools developing assessments in the non-cognitive
domains of medical education, are now the challenges for
future researchers in medical education. There is, after all, an
ethical imperative to ensure we choose the best candidates to
educate as future doctors.
Notes on contributors
JANE ADAM, MA, PhD, MB, BChir, MPH, FFPHM, was an Associate Dean
for Admissions and a PBL tutor at Hull York Medical School from 2003 to
2011, with a research interest in methods for selecting medical students.
MILES BORE gained his PhD in Psychology in 2002, is a registered
psychologist and a senior lecturer at the University of Newcastle, Australia.
His teaching and research interests are in the areas of psychometrics,
personality, moral orientation and the selection of applicants to health
professional education.
ROY CHILDS, BSc, PGCFE, AFBPsS, is the Managing Director of Team
Focus and a Chartered Occupational Psychologist. He works as a coach,
facilitator, trainer and researcher and his main focus is building sustainable
relationships that enhance well-being and performance. His challenge to
orthodox thinking is embodied in an innovative range of psychometric
instruments.
JASON DUNN studied for a PhD in Human Sciences at the Hull York
Medical School, graduating in 2013. He remains an honorary research
fellow of HYMS.
JEAN McKENDREE, PhD, is a senior lecturer in medical education at Hull
York Medical School. She is a cognitive psychologist whose research
involves applying cognitive science principles to improving educational
theory and practice.
DON MUNRO, PhD, is a former staff member and now conjoint associate
professor in the School of Psychology, University of Newcastle. His
interests are in personality and motivation, psychometrics and selection
testing.
DAVID POWIS, PhD, has been a university teacher of, and researcher in,
physiology and medical education since 1972. He is currently conjoint
professor in the School of Psychology at the University of Newcastle. Over
the past three decades, he has worked particularly in the area of medical
student selection with the aim of establishing fair principles and appropri-
ate strategy for selecting students for health professional courses.
Acknowledgements
We are grateful to Professor Barry Wright for allowing us to use
the simulated patients’ ratings of empathy. We express grateful
thanks also to Emeritus Professor John Cookson and to
Professor Jonathan Bennett both of whom read and com-
mented on earlier drafts of the article.
Declaration of interest: Drs. Bore, Munro and Powis are
joint authors of the Personal Qualities Assessment battery of
tests and receive royalty payments when the PQA is used
commercially. Mr. Childs is the owner/manager of Team Focus
who are the commercial publishers of the RSQ. Dr. Adam was
an unpaid member of the UKCAT executive board from 2005
to 2010. Dr. McKendree and Dr. Dunn report no declarations
of interest.
References
Adam J, Bore M, McKendree J, Munro D, Powis D. 2012. Can personal
qualities of medical students predict in-course examination success and
professional behaviour? An exploratory prospective cohort study. BMC
Med Ed 12:69ff.
Barr DA. 2010. Science as superstition: Selecting medical students. Lancet
376:678–679.
Best J. 1989. The politics of the sand pit. Med J Australia 150:158–161.
Bore M, Munro D, Kerridge I, Powis D. 2005. Selection of medical students
according to their moral orientation. Med Ed 39:266–275.
Bore M, Munro D, Powis D. 2009. A comprehensive model for the selection
of medical students. Med Teach 31:1066–1072.
Campbell EF, Rank BK, Sinclair AJM. 1974. Selection of medical students –
A burning question. Med J Australia 1:785–788.
Childs R. 2012. [Accessed 28 January 2015]. Available from: http://
www.teamfocus.co.uk/tests-and-questionnaires/understanding-motiv-
ation/resilience-scales.php.
Cookson J, Crossley J, Fagan G, McKendree J, Mohsen A. 2011. A final
clinical examination using a sequential design to improve cost-
effectiveness. Med Educ 45:741–747.
J. Adam et al.
12
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
Cooper A, Petrides KV. 2010. A psychometric analysis of the Trait
Emotional Intelligence Questionnaire-Short Form (TEIQue-SF) using
Item Response Theory. J Person Assess 92:449–457.
Ferguson E, James D, Madeley L. 2002. Factors associated with success
in medical school: Systematic review of the literature. BMJ 324:
952–957.
Ferguson E, James D, O’Hehir F, Sanders A. 2003. Pilot study of the roles of
personality, references, and personal statements in relation to perform-
ance over the five years of a medical degree. BMJ 326:429–432.
Ferguson E, Semper H, Yates J, Fitzgerald JE, Skatova A, James D. 2014. The
‘dark side’ and ‘bright side’ of personality: When too much conscien-
tiousness and too little anxiety are detrimental with respect to the
acquisition of medical knowledge and skill. PLOS One 9(2):e88606,
1–11.
Goodenough Committee. 1944. Report of the Interdepartmental Committee
on Medical Schools, HMSO, London, p 106. Cited in Smyth DH 1946.
Gray SA, Deem LP, Straja SR. 2002. Are traditional cognitive tests useful in
predicting clinical success? J Dental Educ 66(11):1241–1245.
Groves M, Gordon J, Ryan G. 2007. Entry tests for graduate medical
programs: Is it time to rethink? Med J Australia 186:120–123.
Haight SJ, Chibnall JT, Schindler DL, Slavin SJ. 2012. Associations of
medical student personality and health/wellness characteristics with
their medical school performance across the curriculum. Acad Med 87:
476–485.
Kelly ME, Regan D, Dunne F, Henn P, Newell J, O’Flynn S. 2013. To
what extent does the Health Professions Admission Test-Ireland
predict performance in early undergraduate tests of communication
and clinical skills? – An observational cohort study. BMC Med Ed 13:
68ff.
Koenig TW, Parrish SK, Terregino CA, Williams JP, Dunleavy DM, Volsch
JM. 2013. Core personal competencies important to entering students’
success in medical school: What are they and how could they be
assessed early in the admission process? Acad Med 88:603–613.
Lambe P, Bristow D. 2010. What are the most important non-academic
attributes of good doctors? A Delphi survey of clinicians. Med Teach 32:
e347–e354.
Lancet [editorial]. 1984. Medical student selection in the UK. Lancet 2:1190.
Lockhart LB. 1981. Why aren’t they choosing the right candidates for
medicine? Lancet 7:546–548.
Lynch B, MacKenzie R, Dowell J, Cleland J, Prescott G. 2009. Does
UKCAT predict year 1 performance in medical school? Med Ed 43:
1203–1209.
McKinley DW, Norcini JJ. 2014. How to set standards on performance-based
examinations: AMEE Guide No. 85. Med Teach 36(2):97–110.
McManus IC, Woolf K, Dacre J, Paice E, Dewberry C. 2013. The academic
backbone: Longitudinal continuities in educational achievement from
secondary school and medical school to MRCP (UK) and the specialist
register in UK medical students and doctors. BMC Med 11:242ff.
Munro D, Bore M, Powis D. 2005. Personality factors in professional ethical
behaviour: Studies of empathy and narcissism. Australian J Psychol 57:
49–60.
Petrides KV. 2009. Technical manual for the Trait Emotional Intelligence
Questionnaires (TEIQue). London: London Psychometric Laboratory.
Poole P, Shulruf B, Rudland J, Wilkinson T. 2012. Comparison of UMAT
scores and GPA in prediction of performance in medical school:
A national study. Med Ed 46:163–171.
Schuwirth L, Cantillon P. 2005. The need for outcome measures in medical
education. BMJ 331:977–978.
Simpson PL, Scicluna HA, Jones PD, Cole AMD, O’Sullivan AJ, Harris PG,
Velan G, McNeill HP. 2014. Predictive validity of a new integrated
selection process for medical school admission. BMC Med Educ 14:86.
Smyth DH. 1946. Some principles in the selection of medical students. BMJ
Sep 14:357–367.
Wright B, McKendree J, Morgan L, Allgar VL, Brown A. 2014. Simulated
patients’ ratings of empathy in medical student final year clinical
examination. Are they useful? BMC Med Educ 14:199.
Predictors of outcome at medical school
13
Med
Tea
ch D
ownl
oade
d fr
om in
form
ahea
lthca
re.c
om b
y U
nive
rsity
of
New
cast
le o
n 02
/10/
15Fo
r pe
rson
al u
se o
nly.
Top Related