Feedback Informed Treatment: Interview with Scott D. Miller, Ph.D.

[Episode 131] In today's episode, I speak with Scott Miller, Ph.D. about how we can use feedback to improve client outcomes. Scott is one of the developers of Feedback Informed Treatment, or FIT - an "empirically supported, pantheoretical approach for evaluating and improving the quality and effectiveness of behavioral health services. It involves routinely and formally soliciting feedback from clients regarding the therapeutic alliance," client outcomes, and using that information to improve services (https://centerforclinicalexcellence.com/wp-content/uploads/2021/02/FIT-what-is-it-2020.pdf).  

Scott

Download MP3 [49:02]

Transcript

Introduction

Hey there Podcast listeners, Jonathan here.  In today's episode, I speak with Scott Miller, Ph.D. about how we can use feedback to improve client outcomes. Scott is one of the developers of Feedback Informed Treatment, or FIT - an "empirically supported, pantheoretical approach for evaluating and improving the quality and effectiveness of behavioral health services. It involves routinely and formally soliciting feedback from clients regarding the therapeutic alliance," client outcomes, and using that information to improve services (https://centerforclinicalexcellence.com/wp-content/uploads/2021/02/FIT-what-is-it-2020.pdf).

One of the most terrifying phrases I remember my supervisor saying to me When I first started working as a social worker was, Hey, Jonathan, can I give you some feedback? No joke, even as I say it now I feel my heart starting to race. Pavlov is smiling somewhere. There was something about the idea of feedback that terrified me. I think the obvious reason is because I assumed it was bad. I mean, if somebody wanted to give me a compliment, they wouldn't say, “hey, can I give you feedback? I thought you did a really good job today.” No, they would just be like, “I thought you did a really good job today.” So when someone says, “can I give you some feedback” it suggests that I’ve done something wrong. Now, I know the cognitive therapists in the crowd are automatically thinking (see what I did there) that I’m making some assumptions about what the feedback might be. And that’s true. But, I don’t think I’m alone in this.

There have been times when I’ve been more open to feedback. Many of those times were when I was a student. I was in the learner role. I was expected to make mistakes, get feedback, and improve. These days I spend a lot of time teaching. I know I make mistakes, which is why at the end of every class I ask students to anonymously give me feedback about how the class went. I incorporate that feedback into next week’s class. But, I’m getting ahead of myself.

In 2004, I read an article by Scott D. Miller where he talked about losing faith in psychotherapy (https://www.psychotherapy.net/article/scott-miller-losing-faith-psychotherapy). It is an amazing read, I’ve put a link to the article on the Social Work Podcast website. He wrote about leaving the annual Psychotherapy Networker conference, not with a rush of enthusiasm for the work, but with doubts. Doubts about clinical techniques, about the latest and greatest trends in therapy, and the benefit of therapy overall. He ends the article wondering if the answer to his doubts lies with client feedback. A few years later, in 2007, Scott wrote two articles for the Psychotherapy Networker (the magazine, not the conference). One was called “Supershrinks.” The other was called, “How Being Bad Can Make You Better: Developing a Culture of Feedback in Your Practice.” Super shrinks? I want to be a super shrink? Being bad? Been there, done that. This was the first time I read about his Outcome Rating Scales and Session Rating Scales. The feedback he had alluded to in his 2004 article finally had some names. 

Over the years, Scott Miller and his frequent co-authors Barry Duncan and Michael Lambert opened my eyes to the common factors - things that were common across psychotherapies, like therapeutic alliance, shared goals, client belief in the therapist, that appeared to contribute far more to client change than any specific ingredient in any given therapy. And while the evidence is pretty clear that people who get psychotherapy do better than those who don’t, the evidence is also pretty clear that all effective therapies are about equally effective. So, imagine my surprise in 2017 when I read another of Scott’s articles in Psychotherapy Networker, this one entitled, How Psychotherapy Lost Its Magick” (https://www.psychotherapynetworker.org/magazine/article/1077/how-psychotherapy-lost-its-magick). The title made me think I was going to read another reflection about Scott losing faith in psychotherapy. But no. It was about what therapists can learn from psychics and mediums. Wait, the same Scott Miller who has debunked so many of our most cherished myths about flavor-of-the-month therapies was selling psychics? Well, not exactly. He was reminding us that the way psychics and mediums practice their magick holds important lessons for therapists. I’ve put the link to all of these articles on the Social Work Podcast website.

So, it turns out that feedback might be scary. But, if you’re a professional, it is really important to get feedback about how you’re doing. Because it isn’t about you. It is about your clients. How do you find out how well things are going? How do you find out if your clients improving? Well, Scott Miller and his colleagues have developed an approach called feedback informed therapy, or FIT. And who better to talk to about this than Scott Miller himself.

In today’s episode I speak with Scott D. Miller, Ph.D., founder of the International Center for Clinical Excellence an international consortium of clinicians, researchers, and educators dedicated to promoting excellence in behavioral health services. Scott and I talk about how to set up a culture of feedback in the therapy room, what the Outcome Rating Scale and the Session Rating Scale are and when to use them, how to integrate feedback into a therapy session, and ultimately how to use feedback to improve client outcomes. Scott debunks a couple of myths along the way, including the myth that training and supervision are useful in improving client outcomes, and that therapists actually want to get feedback from their clients. He provides examples of how to talk with clients about feedback and what to do when clients say that everything is going great. We end our conversation with a call for therapists to think deliberately about practice. He suggests that getting feedback from clients, identifying patterns about what we’re doing that works or doesn’t work, and then doing the work to improve those deficits is true professional development. Learning that we miss the mark with a specific problem or type of client over and over again and doing the work to fix it leads to greater professional humility. He makes a compelling argument that we should become confident that we can address our deficits as clinicians rather than being confident that we’re great therapists. I found the conversation inspiring and I hope you do to.

A note about this interview. We recorded our conversation at Scott's house, in his kitchen, in May 2018. As I was packing up, Scott asked if he could give me some feedback about the interview. I smiled at how meta his question was, and also felt a little anxious. But, as it turns out, he really enjoyed our conversation and thanked me for the interview. And just like that I realized that some people really do preface a compliment by asking if they can give feedback. When I got home, I made a low-quality MP3 version of the recording so that I could listen to it on my way to work, and promptly lost the master tapes for 3 ½ years. But, luckily for you, I found them and edited them into the high quality interview you’re hearing today. 

I'd like to thank Rebecca Robertshaw for transcribing this episode. Not only did Rebecca put words to paper so that people can read the conversation while listening, or instead of listening, she looked up all of the articles that Scott referenced during the interview. So, Rebecca, thank you for your generous donation of time and attention to detail. If you appreciate the transcripts and would like to volunteer to transcribe an episode in exchange for a shout out on an upcoming episode of the podcast, please contact me at jonathan dot b dot singer at gmail dot com. 

Now, I’ve mentioned Psychotherapy Networker magazine a couple of times in this introduction. I think the Networker is a great resource for therapists. They have articles like Scott’s, a clinician’s digest where they summarize practice insights from research, interviews and articles with top therapists, and loads of videos and web content. I’m telling you all of this because Psychotherapy Networker is offering listeners of the Social Work Podcast, that’s you, with a special price for a one-year subscription. This isn't a paid promotion, I'm not getting a kickback and I still pay full price for my annual subscription. But, Scott Miller has a ton of amazing articles on the Networker. With your subscription, you get access to the current issue and the entire archive of back articles articles, videos, and clinician updates. If you’ve been wanting to give yourself the gift of the Networker, but never wanted to shell out the $36, please go to https://www.psychotherapynetworker.org/socialworkpodcast and get a one-year subscription for only $6. I know. It is amazing. 

And now, without further ado, on to Episode 131 of the Social Work Podcast. Feedback Informed Treatment: Interview with Scott D. Miller, Ph.D.
 

Interview 

Jonathan Singer: So, Scott, thank you so much for being here on the Social Work Podcast and talking with us today. I’m really curious: what do you mean by ‘feedback’?

Scott Miller: We have a very specific meaning. It is using formal measures or scales of progress in the working relationship to monitor the client’s experience of care, and to feed back their experience to the therapist.

Jonathan Singer: And those measures are the Outcome Rating Scale and the Session Rating Scale, right?

Scott Miller: Well, what’s known as Feedback Informed Treatment could use any particular measure, or combination of measures; but our own two tools, the ORS, the Outcome Rating Scale, and the Session Rating Scale – or ORS and SRS – are ones that you can use and download and access from my website, if you’re an individual practitioner, for free.

Jonathan Singer: So that’s interesting. So the idea of Feedback Informed isn’t specific to those measures, but it’s this broader idea of feedback. What’s important about getting feedback? Why is that important for clinicians to understand?

Scott Miller: There are so many lines of evidence that indicate why it’s essential for us to get feedback. The first real reason is that the outcomes in our field haven’t improved in 40 years. Virtually every therapist that goes to a workshop thinks that they’re getting the latest information about how to be effective; when we look at the overall outcomes of psychological treatments, they’re exactly where they were in 1976. The second piece of data that I think is really important to know is that individual clinicians’ outcomes don’t improve either; and in fact, if there’s any evidence, it’s that clinicians’ outcomes deteriorate with time and experience. And a real toxic addition to that is that therapists become more and more confident about their work over time.

[02:11]

Jonathan Singer:

Okay, this is really interesting: so you’re saying that outcome is both how clients’ outcomes might change, but also you’re talking about how therapists are doing; and you’re saying that they are not doing as well, as they get more years in the profession?

Scott Miller:

Typically, and on average, no. Therapists think they’re getting better, when they’re not. We also have a fair bit of data that shows that therapists overestimate just how effective they really are by about 65%. So it’s not a small amount. We have an illusory superiority bias that makes us think that we’re much better than we are. And that actually serves a benefit to therapists: believing that we’re effective is critical to staying in the field and not burning out. At the same time, it’s toxic for actually getting better at our craft, which would ultimately reduce burnout if we could stick at what it takes to become better, long enough.

[03:10]

Jonathan Singer:

So it seems like those are some pretty compelling reasons why it’s important to get feedback: both if you want your clients to improve, but also if you want to do better. How did you come up with the domains, or the areas, that are evaluated on the ORS and the SRS?

Scott Miller:

The ORS is a very specific measure: it measures a particular domain, and that is clients’ wellbeing. For most of the history of the field, we have focused as researchers and clinicians on symptom reduction. The problem is that clients can experience a reduction of symptoms, but not feel any better in their lives, and not function very well. So we’ve chosen instead to look at how they’re doing in their individual, relational, and social lives. It turns out when people feel pretty good about those areas of their lives, they are much less likely to say they need mental health care, or physical health care; to be relatively happy; to experience fewer sick days and a whole host of other, very positive, gains or outcomes in their lives.

[04:13]

Jonathan Singer:

So these are areas that you knew about, that there was some research that this was important to measure, and that you created a scale? Is that how that came about?

Scott Miller:

Well, we stand on the shoulders of a lot of researchers – giants, actually. I was research assistant for a while for Michael Lambert, who developed a tool called the Outcome Questionnaire 45. Michael’s tool assessed four domains using, as the name implies, 45 different items. We were looking for a shorter version to assess those same domains, and used a visual analogue scale (which is just having clients mark on a line to tell us how they were doing in those areas), and it turns out that using that scale is about as good using the much longer scale.

[05:05]

Jonathan Singer:

Which is so nice to have, [laughter] a shorter scale that works as well as the longer scale. So in terms of this Outcome Rating Scale, can you walk us through how you use it?

Scott Miller:

Sure. I think It’s important to know, first off, that if you ask a therapist, are they asking for and responsive to feedback, virtually all of them say they are. The problem is that they tend to ask for feedback at times when it may not be necessary; and they don’t really have a way to assess whether or not the particular feedback they are giving is at odds with behavior that would normally engender positive changes with their clients. So we think it’s just critical that they have a standardized measure that they use, against which they can compare all of their work. It’s sort of like learning to play scales on the piano. Scales aren’t music; the measures aren’t therapy. But what it does give you is a way to look at your work over time, comparing it to a common standard.

And when you start this process, I think it can be kind of a steep learning curve. This isn’t about simply beginning to administer scales; you have to first create a culture of feedback with your clients – you have to let them know that what you really want is for them to tell you whether or not you’re being helpful. That means starting off by saying something like, “I work a little differently than you may be accustomed to. I’m not always successful; many times, I am, but if I’m not, I want to know. If we are successful working together, we can work as long as you like. If not, then I’ll do some changing and adjusting to see if I can’t make a better fit, to do something that is helpful; if not, then we can get some consultation. You’re more than welcome to follow with me as I get that consultation, to see if there’s not something else that I could do to be more helpful. And ultimately, and in some cases, we might decide together that I’m not the right person. And that doesn’t mean that I’m just going to kick you out; it means that I’ll work with you to find somebody that can be, perhaps, more helpful than I’ve been.”

[07:11]

Jonathan Singer:

That’s a pretty radically different way of working with folks than I think most providers would present themselves as. I don’t think most providers would present themselves quite like that. But it’s a really interesting framework, to say, “I want to hear from you what’s working and what’s not working, and I will respond to that.”

Scott Miller:

Right. Again, I think most therapists believe they’re asking for feedback. The whole idea of using measures like this, and then changing the service based on the client feedback, is what most people expect when they see a healthcare professional. If you see your physician and it’s not helping, the physician doesn’t take it personally when you say, “hey, that intervention, that surgery, that method, didn’t really help me.” They should be, theoretically speaking, grateful that you would give the feedback, because that might open up the doors to different types of interventions, or even different providers – maybe specialists. We’re just making this explicit in the language of therapy.

Clients have no idea, really; and if they’ve watched television, they’ve gotten a distorted idea about what ‘doing therapy’ actually is. We’re saying that this is a purposeful contract between the two of us; that I’m being hired to achieve a specific outcome; if it doesn’t work, we should know that along the way, [and] be able to adjust; if it still doesn’t work, we’ll find somebody else that might be more helpful. The good news is that when we make a referral, after having a failed therapy experience with a client – in other words, what I do doesn’t work – the probability of the client being successful the next time they see a therapist is equal to what it was when they saw me, at the beginning. In other words, there doesn’t seem to be a history effect, so clients don’t stand less of a chance because they have prior history of treatment failures, at least when psychotherapy is the intervention we’re using.

[08:59]

Jonathan Singer:

That’s really interesting. I want to kind of punctuate something that you said, which is that you have to start with creating this culture. And as you were talking, I was imagining somebody in an agency setting, or a school setting, or a residential setting, where the protocols and procedures for getting people off your caseload or referring people out might be a little different than if you were in an inpatient setting. In those situations, it seems like it would need to be really an agency, or organizational, culture shift, to say: if it doesn’t work with this one person, then we will try it with somebody else.

Scott Miller:

On January 1st, Joint Commission enacted a new standard that mandated that agencies certified by Joint Commission had to have an outcome measure in place. It’s a great move. The problem with this is that cultures don’t change overnight, and you’ve highlighted the most important aspect. If we’re starting to refer people amongst ourselves, and to different agencies, there’s going to have to be a massive cultural shift – in some respects because I’m sending dollars out the door to somebody else, and this can be a big challenge in publicly funded agencies with very meager and limited resources. At the same time, keeping clients that we’re not helping, for long periods of time, isn’t ethical. We’ve found that implementation is the key challenge, and most people who begin adopting FIT, despite this new mandate from Joint Commission – and SAMHSA, the Substance Abuse and Mental Health Services Administration, has followed suit as well – despite that, most agencies take between three and seven years to achieve that cultural shift. What’s required is a dedication to creating it in the first place: use of the measures, and setting policies and procedures in place so that the therapist and client can respond to the feedback that’s being given.

[11:12]

Jonathan Singer:

So the Joint Commission recognizes that this is important to have: this outcome measure is important to have. So let’s imagine: fantasyland, that it’s seven years down the line; the agencies, organizations, have adopted this. What is it that a provider is expected to do, or how are they expected to use, say, the Outcome Rating Scale?

Scott Miller:

The Outcome Rating Scale should be administered on an outpatient basis, at every session, at the beginning of the session. We have to know: did what we do last time actually help the person, so that we can adjust what we do in the present visit. You need to know what your blood values are before you titrate the dose of insulin you give yourself; it’s the same thing in psychotherapy. It’s not about delivering the protocol that’s standardized to be given across everybody with that particular issue. And at the end of each session, we want to give the SRS, which is the Session Rating Scale, which is the client’s assessment of how good, how well, are we working together. That measure also contains four items, and before I administer it, I also need to work to create a culture of feedback. And I think with the SRS, it’s even more of a challenge, because there’s a power imbalance naturally in the relationship and I really need to encourage the client to share openly how they feel about the work: in terms of feeling understood, whether we worked on what they wanted to work on and talk about, whether the approach made sense to them, and whether it felt complete or something’s missing. So I say, “we’re at the end of our visit; as I told you in the beginning, I work a little bit differently. I really want to make sure that how we’re working best sets you up for success in the time that follows before our next visit. I’m not interested in perfect scores. Ten out of ten doesn’t mean very much to me. Life’s not perfect, neither am I. Sometimes you may think, well, the session was pretty good, so I don’t have to mention X, Y, or Z. You should mention X, Y, and Z, and we can see together whether or not it would be useful to change some of that to better facilitate the outcomes that you’re looking for.”

[13:17]

Jonathan Singer:

So at the beginning of the session, they fill out the ORS. When you get those scores, do you look at them with the client? Do you have a conversation? Does it become part of the therapeutic content?

Scott Miller:

Yes. As would the therapist listening to your heartbeat through the stethoscope; it’s going to become a part of the service that’s offered. I have to react to what I hear and see. So, depending on which format you’re using – if you’re using paper and pencil, you can score them relatively easily, you plot them on a graph; if you’re using one of the electronic systems, all of them provide an electronic prediction of the client’s progress over time. You can compare, then, the client’s actual score against that expected treatment response to see whether the client is progressing like people who end as successfully, or who do not. They either deteriorate or they drop out from services. And I can have a conversation with the client right then. If there’s progress, for example, between this and the last visit, I can ask them, what happened? The scores have gone up – how did you do that? What did you do during that time? If there isn’t progress, I want to explore that as well. What happened? Why didn’t what we talked about last time make a difference for you? How do we need to adjust the service so that whatever we do here today makes a difference in your life?

[14:41]

Jonathan Singer:

So these scores that you’re talking about, that you’re matching your client against: are those the scores of your other clients, or is this sort of a standardized score from millions of clients that have used this form?

Scott Miller:

Interestingly enough, it’s the latter, and the particular scores that we have are based on millions of clients worldwide. And the advantage is that these are actually predictive trajectories. So imagine that you go to the physician, and they deliver some bad news. They say that you have an illness; let’s say it’s cancer. One of your first questions is going to be, how long do I have? And then the physician’s going to say something, or physicians use to say something, that sounds really informative but actually doesn’t tell you much of anything at all. They’ll say, you have a 68% chance of survival at five years. That sounds pretty good; it sounds better than 67%, but not quite as good as 69%. The problem is it doesn’t tell you what your chances are. It doesn’t tell you if you’re part of the 68% or not, because you’re only 100% dead at five years or 100% alive. What you need to know is, as treatment progresses, am I responding like the people who are part of the 68%? That’s what these trajectories do, and they’re based on the same ideas, and logic, and mathematics, as trajectories that are now being used to predict cancer outcomes. Are you progressing, from session to session, like people who end the treatment with this particular provider successfully? And critically, if you’re not, that’s good news for us, because then we can adjust the service. We can – to give an example, we can change the chemotherapy; we can augment it with something else. That’s the whole idea behind plotting this course and talking about it with the clients.

[16:28]

Jonathan Singer:

So you have these scores and you have a context for understanding what an up, or a down, score is. And these are the scores from the beginning of the session, the Outcome Rating Scale? Is that right?

Scott Miller:

That’s right. On the SRS, which is the score at the end of the session, the client gets the same 40-point span, but really what matters is low scores in the beginning. Paradoxically, our data have shown us that therapists who manage to get clients to complain about the service early on, which is then improved (as shown by their scores moving up over time) actually have better outcomes, by about 50%. So let me say it again: clients whose scores improve on the SRS from start to finish have better outcomes – almost 50% better. We’ve also found that top performing therapists, those with the best outcomes worldwide, tend to create an atmosphere in the room that clients feel safe making these small complaints. Generally, these complaints – this is the problem – generally, these complaints aren’t big. And we are a tolerant species, much of the time, especially when we’re dependent on another person. So clients are inclined not to complain. These top performers manage to create and environment where people can make small complaints, and feel like that was an important part of the process. Doing so gives the therapist crucial information about where the therapy pinches, or hitches; they can smooth that out – and therefore, they have better outcomes by the end of the service.

[18:02]

Jonathan Singer:

So if I had ten therapists whose clients didn’t like them in the beginning, but then liked them by the end, are you saying that that correlates to better client outcomes? Did I understand that right?

Scott Miller:

Yes. And what we do know is that when we’re looking at therapists, not all are created equal; some therapists have far better outcomes than others, and so, since 19 – I’m sorry, since 2007, we’ve been looking closely at those practitioners to see if there was something we can learn. One of the things we found out first was – again, paradoxically – better therapists, that is, more effective therapists, seem to score lower on standardized alliance scales. It took us a long time to piece this puzzle together. You could have a therapist who got great alliance scores, and they had sort of modest, or average, outcomes. Top performing therapists have lower SRS scores in the beginning, and they have better outcomes? The answer to this dilemma was that these therapists create an atmosphere, again, where the therapist could really share what – or the clients, rather – could really share what they were feeling. So, all in all, you want a therapist you feel like you can tell what needs to be changed about the work; and you feel that that information is received gladly, and you see how they alter it – the services, that is – going forward.

[19:34]

Jonathan Singer:

You know, I can see some real parallels with clinical supervision, in the sense that when I think about going to therapy – me, Jonathan – sometimes it’s hard, even though I am a therapist, and I teach therapy, it’s hard for me to share things that aren’t going well with the therapist – especially if it’s not egregious. I mean, if it’s egregious, that’s an obvious thing, but if it’s a little thing, it is hard to share. And so it’s interesting – what you’re saying actually fits, for me, pretty well, about the importance of being able to talk about the little things. I can also see how, as a supervisee (especially if you’re paying out of pocket, for your clinical license, whatever you’re moving up towards) that there could be some real benefit in knowing that this was an environment where you could say to your supervisor, “you know what, that doesn’t really work for me”, or “the way that you said that didn’t fit; I didn’t really feel like you understood me”, or whatever it is. Have you found the same thing with feedback for an employee-supervisor relationship as has been found in a client-therapist relationship?

Scott Miller:

You mean a supervisor-supervisee, in a therapeutic context?

[21:07]

Jonathan Singer:

Yeah.

Scott Miller:

The answer’s no. [Jonathan: Oh, okay.] But part of the reason for that is that we virtually have no evidence that supervision makes any difference, in terms of client outcomes. There is evidence, albeit correlational, that clinical supervision – that is mandated by virtually all licensing boards – leads to the acquisition of skills associated with particular protocols; greater self-confidence as a therapist; but no evidence – despite how valuable supervisees say supervision is – that it leads to better outcomes for clients. In fact, the latest data, a study by Rousmaniere and group in 2015 (DOI: 10.1080/10503307.2014.963730) found that supervisors accounted for less than 1% of the variability in supervisee outcomes, so it’s next to nothing, really.

[21:57]

Jonathan Singer:

So are these supervisees who are, say, still in school? Like, in social work you have your supervisor, and you have your BSW or MSW students if they’re still in school; and then you have your postgraduate out in the field, doing clinical supervision towards a clinical license or something like that. Which supervisees were in this study?

Scott Miller:

We don’t have any evidence of either, really, that it improves the outcomes of students or of practitioners that are already in the field. Rønnestad and Orlinsky conducted a very long study (https://www.apa.org/pubs/books/4317078), looking at therapist professional development, and when they asked therapists, “do you value supervision?” I think it’s number two on the list of three or four things that they say are absolutely essential for professional development; but we just don’t have any data. The study that I’m referring to looked at both student trainees and staff that were already licensed, and still, very little evidence that the supervisor affects client outcomes. So the question should be why, since so many people value it – and if it’s going to last, there’s going to have to be evidence that it actually benefits the people we’re serving – who really cares if the practitioners have a nice life, or learn new skills, if they don’t translate into any difference in outcome for the recipients of care?

One of the reasons we think that supervision hasn’t contributed directly to client outcomes is that it’s the supervisee who picks the case to discuss. And very often, supervisees are picking the case they’re going to discuss based not on the outcome, but rather the process, of care. Frankly, doing therapy isn’t easy. It can be painful; it can be challenging and difficult interpersonally. Those are the cases that get brought to supervision. But if you’ve been in a long-term committed relationship – I’m not talking about a therapy relationship, but just a personal relationship – you know it’s not all moonbeams and roses; it’s difficult. And in fact, having those difficulties in some ways predicts better relationships – how you handle those difficulties. So that leads to a paradox: again, that you can have cases that are improving, but are interpersonally difficult, that get brought up for supervision, and the process actually disturbs what’s working. Because sometimes, interpersonally challenging interactions lead to growth, and change. If, instead of focussing on process, supervision started with the outcomes – that we actually looked at cases that were not progressing, or were at risk of deterioration (in other words, in our measures, they’d be in the red zone, on track for a negative or null outcome) then we can start to show evidence that the supervision process actually improves outcomes. And, in fact, in a study in 2016 lead by Simon Goldberg (http://dx.doi.org/10.1037/pst0000060 ??), that’s exactly what we found. Therapists asked to bring cases that were documented by outcome measures not to be improving that were then given supervision had better outcomes in the end.

[25:14]

Jonathan Singer:

That’s so interesting [laughs]. So these are clinicians who brought in cases where the scores suggested that things weren’t going well. Were those the ORS, or the SRS scores?

Scott Miller:

All ORS. Because you can have a low SRS score, but if the outcomes are good, then I’m going to be less concerned about that. Because, as I said, therapy can be interpersonally challenging. However, if the outcomes are bad, or inconsistent, or the client is at risk for a negative or null outcome, then to me it doesn’t matter if the relationship is good. I have to figure out how to improve that outcome for that particular client. And of course if the outcomes are bad, and the SRS is bad, well we have a clue as to where to look to improve the chances that the client is likely going to experience benefit from the care provided.

[26:10]

Jonathan Singer:

I think this is such an important point, because if I were a clinician using the ORS and the SRS, I think my gut sense is that the SRS would be more important to look at, because it is about our relationship; but you’re making the point that it’s about the outcome. Outcomes might be improving because things are tense, or difficult, or rough in the relationship, and that’s what’s being measured in the SRS.

Scott Miller:

I’m not one to draw causal inferences lightly, so I would never say it’s because; but what I can say is despite the difficulties, the outcomes are still good, then I’m less concerned about the relationship. So I’m going to be asking the client, of course, if the outcomes are good and the relationship is trouble, I’m going to still ask the client a bit about the relationship. But very often, these are exactly the type of people who say to me, “we did some tough work, and I noticed afterwards that that helped me.”

[27:18]

Jonathan Singer:

So the other day I was talking with a student and she said, “I have a client who’s a student in a school”, she’s a school social worker, and she said, “I know that things are not going well in this client’s life. But when I look at the ORS, I would imagine that she would say everything was great. And I think that that’s one of the problems in her life, is that despite all the problems, she’s not addressing it, or willing to talk about it – with me or with other folks.” Our whole conversation so far has been about making sure folks are – that you know what these measures mean. What do you do when you have somebody who’s marking these outcome measures, and they seem to be not reflective of what’s happening in their life?

Scott Miller:

Well, the operative word in the question from the student was ‘I imagine’. Stop imagining and start asking, what the client says. Stop trying to think for them, and instead solicit their perspective. And remember, if you’re not getting feedback, it’s not because of the client. It’s because of you. The feedback varies depending on who’s asking. So I’m going to think about how well did you create the culture of feedback? Did you say, “I’m really interested in how things are for you. Based on what you say, we’ll either keep going with more of the same, or we’re going to stop and adjust – change what we’re doing, change where we’re doing it, and in many ways, or ultimately, if it’s not helping, change who’s doing it.” Let’s say you’ve done all that; and I will tell you that 70% of the time the problems with using routine outcome measures is failure to adequately create a culture of feedback. Seventy per cent of the time. Everybody wants to attribute it to the client’s pathology; I’m attributing it mostly to the therapist’s inadequate preparation of the client for giving that feedback.

But if somebody gives me a high score, I make no interpretation of that score. I don’t assume, that because I see things that they don’t see, that their score’s a lie. Instead, I act in accord with that score. So, if I get a perfect score – which I rarely get; I may get a few points off perfect – then I’ll, in my setting, I’m going to be asking, “why did you come to talk with me? Because these scores actually look fairly good.” When I ask that question, I get one of three responses. The first response is, and most common: “someone made me come”. Parent, partner, employer, the courts. It’s at this point that I say, “perfect, so you’re not here because you think you have a problem.” Now, I’m joining, here. They say, “yeah, no.” But we’re very compliant: “they told you to do it, you decided to come and follow through.” “Yes.” “So let’s fill out the measure a second time. And would you please fill it out as if you were the person who sent you here?” That way, the scores usually drop. They don’t have a problem, the person says; but they know full well, because they’ve just told me they’re here become somebody else sent them.

The second most common reason – and this was, the one I just gave you, is probably 90% of the time – but the second most common reason people have high scores is because they in fact have a life that’s pretty good; but they have a very circumscribed problem. And a classic example of this is phobias. Your life can be actually pretty good, and you’d still be morbidly afraid of spiders. What do you do? You just make sure there are not any spiders. Or you’re going to a place with spiders. And then it’s pretty good. But if you’re suddenly going to go, say, on a camping trip: now, this becomes real. When you give the ORS, their wellbeing’s going to be high, because it is. What you’re going to have to do in that particular case is focus on that very specific problem. And then you might want to use an alternate measure to track the client’s response to the care provided around that specific problem. You can also administer the ORS on a continuous basis to make sure you don’t make them worse.

The third possible reason that people come up – and this is very rare in my experience, but it does come up on occasion – when you say, “well, why are you here? Your scores are high”, and the person says something like, “well, I just moved to your town,” and you say, “What? You know, why would that make you come to me?“ “Well, I’ve been in therapy my whole life, and you’re just the next one in line.” So in fact, this person’s wellbeing is actually really pretty good; but they’re using therapy for something other than change. Ongoing support. Checking things out. In that particular case, again, I’m going to personally work on a strengths-oriented basis and see if I can’t facilitate moving out of therapy as quickly as possible.

[32:17]

Jonathan Singer:

That’s so helpful. And I really like how you reframed the student’s question to say, what is it that you have control over, as the provider? You can imagine, but even if you aren’t imagining – even if the responses are as you thought they were going to be – again, what is it that you can control? What is it you can do? What sort of environment, and how do you respond? Which, just to me, sounds like good, responsible, clinical care.

Scott Miller:

Right. Somebody comes to you, and their scores are very high. It’s not my job to do psychoanalysis, and to figure out why, or even break through the denial. What I really want to do is engage them in further conversation. And the obvious question for anybody who rates their wellbeing very high while you’re seeing them is, what are we doing here? Why is this happening? “Well, the principal, or my teacher, said I need to come talk to you.” “Ah. Would the teacher score the measure different?” ”Yes, he or she would.” “Perfect.” And then, I can have a discussion from that. “The teacher rates you a ten, but you said you were a thirty. What is it you know about yourself that the teacher doesn’t know or see, that makes you think things are much better than your teacher thinks?” And we’re off to the races at that point.

[32:45]

Jonathan Singer:

That’s so good. So I Imagine that some of the folks listening are going to be thinking, this is great, I’m going to start doing this. I’m going to use the ORS, the SRS. I’ve been looking this kind of way to enhance or improve my services. Is that enough?

Scott Miller:

Well, in a word, no. First off, there’s a bit of a learning curve involved in picking up the measures. Even getting negative feedback takes time; we find, for example, that when clinicians start using the SRS initially they get almost no negative feedback, and sometimes they’ll come to me and they’ll say, “Well, I asked this question, but… it doesn’t work; the SRS doesn’t work.” I said, “It’s working perfectly fine; your clients have figured out it’s not safe for them to tell you the truth.” [Jonathan laughs] And then they’re frustrated, they say, “What are you talking about?!” And I say, “You’re going to have to reflect on how you administer the scale, and how you give the client the words to critique what you’re doing. So here’s a couple of clues: don’t ask evaluative questions. Don’t ask, how did I do? Instead, ask, what did I do that lead you to rate this this way? Don’t even use personal pronouns. When you say, how did I do, or what did I do, any response the client gives is now a direct criticism of you. So, work at creating questions, and clarifying questions, that leave out personal pronouns. You’re much more likely to get negative feedback in the process. Once you’ve started doing this, you’re going to find that the feedback does improve your outcomes, mostly because it’s pointing to the clients that you’re not helping.

We’ve discovered, though, through a series of very clever experiments, that therapists learn from the feedback to respond in the moment. It’s like seeing the red light, telling you to stop, and your foot just kind of automatically pushes on the brake. The problem is, it doesn’t appear that that type of feedback generalises. We learn to stop at that red light, but we don’t learn to stop at all red lights. Or we don’t learn to stop when we should, but there doesn’t happen to be a stop light there, if you get my drift. We don’t anticipate it. And this requires a whole new activity that our research attention has just been turning to, and it’s some of the most exciting research going on in the field right now on a subject known as ‘deliberate practice’. When you get negative feedback, you have to take time to reflect on the broader meaning of that feedback, relative to your style. What does it mean about me? Is this the type of feedback that recurs, over and over? Am I getting it from one particular type of person, in one particular context, applied to one particular problem? That means I’m likely going to have to do some changing in how I think. And I may find deficits in my performance that are not random – that are routine, that are baked into my cake, so to speak. That means I’ll have to do some reading, and some studying, maybe even get some coaching, specific to those deficits in my performance.

It all sounds painful; but to me, it’s some of the most exciting work we’re doing. It really is opening up the door to true professional development. Not that I feel more confident about my work, but rather that I’m actually better for my clients, over time. And curiously, something puzzling happens as you engage in deliberate practice: you experience more humility in your work, as opposed to confidence. Now, I’m a very confident clinician, but I’m not confident about my abilities; I’m confident that I will take the time to reflect, identify my non-random errors, and eventually I’ll figure it out if I put in the time.

[37:56]

Jonathan Singer:

It sounds like you’re confident that you know how to address deficits in your practice, which is very different than saying, “I’m confident that I’m a great therapist.”

Scott Miller:

No. I’ve never had that [laughter], and this has only made it worse. Because getting the feedback really gives me an ongoing source of information about where my growth edge is.

[38:24]

Jonathan Singer:

Well, Scott, thank you so much for taking the time and talking with us today about Feedback Informed Treatment. I really appreciate it.

Scott Miller:

My pleasure.

Transcription generously donated by Rebecca Robertshaw, Think Ahead/University of York Masters in Social Work programme (UK) 

References & Resources

International Center for Clinical Excellence: https://centerforclinicalexcellence.com/ 

Joint Commission Standard CTS.03.01.09: https://www.jointcommission.org/accreditation-and-certification/health-care-settings/behavioral-health-care/outcome-measures-standard/

Outcome Rating Scale (ORS) and Session Rating Scale (SRS): https://scott-d-miller-ph-d.myshopify.com/collections/performance-metrics/products/performance-metrics-licenses-for-the-ors-and-srs

Miller, S.D., Hubble, M.A., & Chow, D. (2020). Better Results: Using Deliberate Practice to Improve Therapeutic Effectiveness. Washington, D.C.: American Psychological Association. 

Feedback-Informed Treatment in Clinical Practice: Reaching for Excellence (2017). https://amzn.to/3GL0kLV 

The Heart & Soul of Change (2nd Edition) https://amzn.to/3E1QrYc

Goldberg, S. B., Babins-Wagner, R., Rousmaniere, T., Berzins, S., Hoyt, W. T., Whipple, J. L., Miller, S. D., & Wampold, B. E. (2016). Creating a climate for therapist improvement: A case study of an agency focused on outcomes and deliberate practice. Psychotherapy, 53, 367–375. https://doi.org/10.1037/pst0000060

Orlinsky, D. E., Rønnestad, M. H., & Ambühl Hansruedi. (2005). How psychotherapists develop: A study of therapeutic work and professional growth. American Psychological Association.

Rousmaniere, T. G., Swift, J. K., Babins-Wagner, R., Whipple, J. L., & Berzins, S. (2016). Supervisor variance in psychotherapy outcome in routine practice. Psychotherapy research : Journal of the Society for Psychotherapy Research, 26, 196–205. https://doi.org/10.1080/10503307.2014.963730

---

APA (7th ed) citation for this podcast:

Singer, J. B. (Host/Producer). (2021, February 16). #131 - Feedback Informed Treatment: Interview with Scott Miller, Ph.D. [Audio Podcast]. Social Work Podcast. http://www.socialworkpodcast.com/2021/12/FIT.html