We
had the opportunity to speak to Professor John Norcini, President and CEO of
FAIMER (Foundation for Advancement of International Medical Education and
Research) about his six-year relationship with the National Healthcare Group
(NHG) Education. Prof Norcini who introduced work-place based assessments to
NHG, shared how his views on NHG’s shift from summative to formative
assessments, and how evaluation and assessments are changing in this era of big
data.
How
did you get to know us, NHG?
I was invited by Associate Professor Tham Kum Ying to start (the
professional development workshops). She invited me over to Singapore to give a
workshop on evaluation and assessment, something which I’ve worked with much of
my career. It started with that,
focusing on workplace- based
assessment (WBA). So that’s where my relationship with NHG began.
Looking back over the past six
years, has the focus of the NHG Education office changed?
NHG has changed dramatically over time. I think when I started,
much of the assessment was focused on doing summative assessments; and now there is much more focus on doing
assessments to support formative learning. The ability of the folks to do
things like provide feedback, changed dramatically; there has been incredible development.
The other thing that is
dramatic was when we started with the physicians, and now we’ve gone beyond the
method that I came here for, branched out into several other methods. And we’ve
also branched across many of the health professions such as pharmacists,
podiatrist, etc.; which has been incredibly exciting for me.
It is really fascinating for me to have the opportunity to talk
to people from other professions; to learn what is important in their
profession, to begin to understand the commonality across the various
professions and things that everyone values.
Since we’ve expanded from
evaluations for the medical professions to the allied health, nurses, what
needs to be tweaked when you expand the programme?
I think that some of the fundamentals are the same, regardless
of the professions; the ability to communicate, get along with colleagues, all
of those things are common across the professions. But the actual nature of the
work is different and that helped me to determine the requirements; not for the
methods, but they all need to be tailored to the particular professions. It’s
not rocket science, basically you are watching somebody, you evaluate them, and provide feedback. It’s more about what aspect of the person [in
that profession] that you are observing, and the kinds of judgement that is
made which differs from profession to profession.
How
would you do it differently for
nurses and doctors? How would a nurse’s evaluation be done in comparison to the doctors?
I am not sure if it would be dramatically different; nurses’
in-patient “encounters” are the same as doctors; what differs perhaps, are the
skills you would be looking for. There are also commonalities, when you watch
that encounter both are looking for communication skills. At the core there are
similarities, but they express themselves differently.
If
we were to just broaden beyond people who deal with patients, and
bring it to the other people in healthcare i.e. the administrative and support
positions; is there a way to draw the same principles that you have
of evaluation and assessment?
Absolutely! No question. One of the instruments or methods that
everybody is using is 360 degree evaluations, I mean that’s started in the
business world and then came to healthcare professions education; so it’s a
natural for these positions, but beyond that, if you look at jobs in general,
you have people engaged with tasks, and observing people as they engage with the
tasks that are part of the job, evaluating it and then providing some feedback around it; is a framework
that cuts across everything, that everybody does everywhere.
The process is identical and the questions that you ask in the
surveys might be a little different.
In a 360 assessment, what you care about mostly are
communication skills; working with other people, in a professional
collaboration, leadership and management; all those things cut across just
about everything that everybody does. There are great areas of commonality.
Looking
at NHG, and comparing to the other countries that are also trying to institute
this American- style residency medical education, how do we compare?
There are a couple of things,
first of all, many places are not trying to adopt the American-style of medical
education; we [FAIMER] actually think what’s happening is an
“international-style”, not American-style, Singapore has adopted the accrediting
models and training models of the US in terms
of the ACGME and all that. But in terms of the underlying things you are
doing principles of assessment and teaching, they really are universal and not American. And I think Singapore is well in-advance of many
places in terms of caring about things like feedback, informative assessment,
and carrying it out; not American-style or international- style, but a style
that suits Singapore.
I see Singapore among the leaders,
there’s a special cultural style here that it doesn’t necessarily exist
everywhere; much of the sense that
if the [Singapore] institution decide to move in a particular direction, people
cooperate in that direction. In other places, there isn’t that collective will,
and the sort of individualistic characteristic getting in the way of
getting things done and making changes.
I think Singapore is a special place, you have both the
resources and willingness to make this change.
Do
we always need separate type of assessments? E.g. summative and formative, can
you design an instrument that can be used for both?
The instruments are probably similar or the same, what’s
different is the way people react to them, different effects.For instance, if you were to do a Mini-CEX (Mini Clinical
Evaluation Exercise) with a trainee, and if the trainee knows that the
information is going into his or her portfolio, where a judgment is going to be
made at the end of the day based at least in part on that information, they
would approach the whole thing differently; in contrast to, if they think that
this is for their education, and data or scores not taken as judgment about
them.
The sort of free flow, learning and feedback becomes different. If
I know that you [as an assessor] are giving me feedback, using that to evaluate
me, I’m going to take a very different view of that feedback, whereas if you
are sitting here and teaching me, I would be willing to explore, to be more
ignorant, share my lack of understanding; in ways that would be more reluctant
to exhibit if you are going to make a judgement about me.
Is the
culture of psychological safety important in these assessments?
Yes, it really goes to purpose, it’s really important that the
purpose of the assessment be clear. And I also think it’s important that there
only be one purpose –not having the ulterior motive to assess someone else. It
is important that if you do formative assessment, you do formative assessment,
it must be clear. It’s not that you
don’t give feedback based on some of
these assessments; it’s not that you can’t be useful; it does change the
psychological space, I think it makes a difference.
What
sort of the assessments do you think are in the horizon now? How is evaluation
and assessment going to change in the future?
I actually do not think that it’s going to change dramatically.
There are 2 or 3 things that would be happening over the next few years that
might influence it; the first is the increasing use of simulation and
sophistication of simulation; that is going to make a significant difference in
the way assessment is done, it has already made a difference and will continue
to make a difference. The other thing that I think is going to make a big
difference is the increasing capture of patient data and activities in large
online data sources, because I think that information ultimately play a much
bigger role in the evaluation of physicians and other healthcare providers. So
I see big data, the increasing access and use to that data is important, as
well as the growing sophistication of simulations and the ability to do things
that we do now in other ways.
In
recent years, the inclusion of cultural competency in assessments is quite
prevalent, especially in the US, due to the increase in greater patient
diversity that is being seen, how do
you test for cultural competency?
I think that’s a real challenge, even in Singapore… We are more polarised in the US, that makes
it more challenging and in some ways more
important to assess that kind of competency. I’ve always marvelled at
your peaceful co-existence of the ethnic
groups in Singapore.
One of the things that inadvertently happen is the folks who are
not of the main cultural group end up feeling scrutinised, as though they are
different, so it’s a real challenge. One of the keys to doing that properly is
to ensure that the trainees e.g. medical students, all the professions, etc., are diverse
to begin with. Because they
really learn from each other, that is the key to getting it right. It’s not any course
you gonna give; it’s not really any assessment that you have to do; it’s getting the right people into
the room to ensure that the students represent the population and the faculty
as well.
How do
you find a balance between imposing a western view of what medical education
should comprise, and being sensitive to people’s beliefs about what their
health involves [which are culture bound], how do you institute a universal
programme for medical education?
I don’t think that having a
universal medical education programme is a good idea, having common core values
across different culture or places is a good idea; but I don’t think it can be
that way exclusively. There is a general movement towards social accountability
that is extremely crucial. I think you start with the patients, the population
and the area it needs and you work back to education from those patients, so
you are accountable to those.
There’s a general sense that
there is a mismatch in many countries between the kind of doctors and nurses
being produced and the needs of society. So in some countries like mine, there
are not enough primary care providers but there
are a large number of specialists. Those kind of balances need to be worked out
over time, so starting with the patients and
then working backwards
I think is an accountable way
to manage this, now in doing that I think that there will be similar commonality,
and it focusing on that as the global piece I think is a positive. But I think
that if we are only global, we will serve the needs of no particular patient
population as well as we could.