We had the opportunity to speak to Professor John Norcini, President and CEO of FAIMER (Foundation for Advancement of International Medical Education and Research) about his six-year relationship with the National Healthcare Group (NHG) Education. Prof Norcini who introduced work-place based assessments to NHG, shared how his views on NHG's shift from summative to formative assessments, and how evaluation and assessments are changing in this era of big data.
How did you get to know us, NHG?
I was invited by Associate Professor Tham Kum Ying to start (the professional development workshops). She invited me over to Singapore to give a workshop on evaluation and assessment, something which I've worked with much of my career. It started with that, focusing on workplace- based assessment (WBA). So that's where my relationship with NHG began.
Looking back over the past six years, has the focus of the NHG Education office changed?
NHG has changed dramatically over time. I think when I started, much of the assessment was focused on doing summative assessments; and now there is much more focus on doing assessments to support formative learning. The ability of the folks to do things like provide feedback, changed dramatically; there has been incredible development.
The other thing that is dramatic was when we started with the physicians, and now we've gone beyond the method that I came here for, branched out into several other methods. And we've also branched across many of the health professions such as pharmacists, podiatrist, etc.; which has been incredibly exciting for me.
It is really fascinating for me to have the opportunity to talk to people from other professions; to learn what is important in their profession, to begin to understand the commonality across the various professions and things that everyone values.
Since we've expanded from evaluations for the medical professions to the allied health, nurses, what needs to be tweaked when you expand the programme?
I think that some of the fundamentals are the same, regardless of the professions; the ability to communicate, get along with colleagues, all of those things are common across the professions. But the actual nature of the work is different and that helped me to determine the requirements; not for the methods, but they all need to be tailored to the particular professions. It's not rocket science, basically you are watching somebody, you evaluate them, and provide feedback. It's more about what aspect of the person [in that profession] that you are observing, and the kinds of judgement that is made which differs from profession to profession.
How would you do it differently for nurses and doctors? How would a nurse's evaluation be done in comparison to the doctors?
I am not sure if it would be dramatically different; nurses' in-patient "encounters" are the same as doctors; what differs perhaps, are the skills you would be looking for. There are also commonalities, when you watch that encounter both are looking for communication skills. At the core there are similarities, but they express themselves differently.
If we were to just broaden beyond people who deal with patients, and bring it to the other people in healthcare i.e. the administrative and support positions; is there a way to draw the same principles that you have of evaluation and assessment?
Absolutely! No question. One of the instruments or methods that everybody is using is 360 degree evaluations, I mean that's started in the business world and then came to healthcare professions education; so it's a natural for these positions, but beyond that, if you look at jobs in general, you have people engaged with tasks, and observing people as they engage with the tasks that are part of the job, evaluating it and then providing some feedback around it; is a framework that cuts across everything, that everybody does everywhere.
The process is identical and the questions that you ask in the surveys might be a little different. In a 360 assessment, what you care about mostly are communication skills; working with other people, in a professional collaboration, leadership and management; all those things cut across just about everything that everybody does. There are great areas of commonality.
Looking at NHG, and comparing to the other countries that are also trying to institute this American- style residency medical education, how do we compare?
There are a couple of things, first of all, many places are not trying to adopt the American-style of medical education; we [FAIMER] actually think what's happening is an "international-style", not American-style, Singapore has adopted the accrediting models and training models of the US in terms of the ACGME and all that. But in terms of the underlying things you are doing principles of assessment and teaching, they really are universal and not American. And I think Singapore is well in-advance of many places in terms of caring about things like feedback, informative assessment, and carrying it out; not American-style or international- style, but a style that suits Singapore.
I see Singapore among the leaders, there's a special cultural style here that it doesn't necessarily exist everywhere; much of the sense that if the [Singapore] institution decide to move in a particular direction, people cooperate in that direction. In other places, there isn't that collective will, and the sort of individualistic characteristic getting in the way of getting things done and making changes. I think Singapore is a special place, you have both the resources and willingness to make this change.
Do we always need separate type of assessments? E.g. summative and formative, can you design an instrument that can be used for both?
The instruments are probably similar or the same, what's different is the way people react to them, different effects.For instance, if you were to do a Mini-CEX (Mini Clinical Evaluation Exercise) with a trainee, and if the trainee knows that the information is going into his or her portfolio, where a judgment is going to be made at the end of the day based at least in part on that information, they would approach the whole thing differently; in contrast to, if they think that this is for their education, and data or scores not taken as judgment about them.
The sort of free flow, learning and feedback becomes different. If I know that you [as an assessor] are giving me feedback, using that to evaluate me, I'm going to take a very different view of that feedback, whereas if you are sitting here and teaching me, I would be willing to explore, to be more ignorant, share my lack of understanding; in ways that would be more reluctant to exhibit if you are going to make a judgement about me.
Is the culture of psychological safety important in these assessments?
Yes, it really goes to purpose, it's really important that the purpose of the assessment be clear. And I also think it's important that there only be one purpose –not having the ulterior motive to assess someone else. It is important that if you do formative assessment, you do formative assessment, it must be clear. It's not that you don't give feedback based on some of these assessments; it's not that you can't be useful; it does change the psychological space, I think it makes a difference.
What sort of the assessments do you think are in the horizon now? How is evaluation and assessment going to change in the future?
I actually do not think that it's going to change dramatically. There are 2 or 3 things that would be happening over the next few years that might influence it; the first is the increasing use of simulation and sophistication of simulation; that is going to make a significant difference in the way assessment is done, it has already made a difference and will continue to make a difference. The other thing that I think is going to make a big difference is the increasing capture of patient data and activities in large online data sources, because I think that information ultimately play a much bigger role in the evaluation of physicians and other healthcare providers. So I see big data, the increasing access and use to that data is important, as well as the growing sophistication of simulations and the ability to do things that we do now in other ways.
In recent years, the inclusion of cultural competency in assessments is quite prevalent, especially in the US, due to the increase in greater patient diversity that is being seen, how do you test for cultural competency?
I think that's a real challenge, even in Singapore… We are more polarised in the US, that makes it more challenging and in some ways more important to assess that kind of competency. I've always marvelled at your peaceful co-existence of the ethnic groups in Singapore.
One of the things that inadvertently happen is the folks who are not of the main cultural group end up feeling scrutinised, as though they are different, so it's a real challenge. One of the keys to doing that properly is to ensure that the trainees e.g. medical students, all the professions, etc., are diverse to begin with. Because they really learn from each other, that is the key to getting it right. It's not any course you gonna give; it's not really any assessment that you have to do; it's getting the right people into the room to ensure that the students represent the population and the faculty as well.
How do you find a balance between imposing a western view of what medical education should comprise, and being sensitive to people's beliefs about what their health involves [which are culture bound], how do you institute a universal programme for medical education?
I don't think that having a universal medical education programme is a good idea, having common core values across different culture or places is a good idea; but I don't think it can be that way exclusively. There is a general movement towards social accountability that is extremely crucial. I think you start with the patients, the population and the area it needs and you work back to education from those patients, so you are accountable to those.
There's a general sense that there is a mismatch in many countries between the kind of doctors and nurses being produced and the needs of society. So in some countries like mine, there are not enough primary care providers but there are a large number of specialists. Those kind of balances need to be worked out over time, so starting with the patients and then working backwards I think is an accountable way to manage this, now in doing that I think that there will be similar commonality, and it focusing on that as the global piece I think is a positive. But I think that if we are only global, we will serve the needs of no particular patient population as well as we could.
