[Episode 44] Today’s podcast is the second in a two part series on measurement for clinical practice and research. In today's podcast I speak with Dr. Mary Rauktis about the difference between measurement in the field and measurement in research settings. We talk about some of the ways that social workers can think about measurement as a tool to improve clinical practice, and some ways that social workers in the field can develop measures that will really benefit their clients. We talk about some of the challenges social workers have using measurement tools because of how rarely measures are integrated into social work courses. We talk about some ideas for how to better integrate measurement into social work education, particularly beyond the required research classes. We end Part II with a discussion of some resources for social workers interested in learning more about measurement.In Part I, Mary and I spoke about about how she became interested in measurement; some key concepts needed to understand measurement including reliability, validity and error; and how to understand measures used in research articles.
Download MP3 [29:21]Dr. Rauktis mentioned that measurement in clinical practice is really about:
- Decision support
- Outcome monitoring
- Quality monitoring
Mary Elizabeth Rauktis Ph.D. is an Assistant Professor of Research in the Child Welfare Research and Training Program. She is a 1993 graduate of the University of Pittsburgh, School of Social Work.
Prior to her appointment at the University of Pittsburgh, she was the Director of Research and Evaluation at Pressley Ridge, an international nonprofit provider of services to children and families. She was an adjunct at the University Of Pittsburgh School Of Social Work and Robert Morris University Business School, nonprofit management and a visiting professor at the University of Minho, Institute of Child and Family Studies in Portugal.
Contact InformationThe University of Pittsburgh
Room 2326 Cathedral of LearningPittsburgh PA 15260
Office 412 648 1225
Cell 412. 716.9061
fax: 412.624.1159
Mar104@pitt.edu
www.socialwork.pitt.edu
Transcript
Introduction
[Episode 44] Today’s podcast is the second in a two-part series on measurement for clinical practice and research. In today's podcast I speak with Dr. Mary Rauktis about the difference between measurement in the field and measurement in research settings. We talk about some of the ways that social workers can think about measurement as a tool to improve clinical practice, and some ways that social workers in the field can develop measures that will really benefit their clients. We talk about some of the challenges social workers have using measurement tools because of how rarely measures are integrated into social work courses. We talk about some ideas for how to better integrate measurement into social work education, particularly beyond the required research classes. We end Part II with a discussion of some resources for social workers interested in learning more about measurement.
In Part I, Mary and I spoke about how she became interested in measurement; some key concepts needed to understand measurement including reliability, validity and error; and how to understand measures used in research articles.
Interview
[01:22]
Jonathan Singer: Mary in the first part of our conversation you were talking about what was the purpose of measurement, what were some of the key concepts, and understanding measurement. We talking a little bit about measurement in research settings versus the field, but I was wondering are there differences between measurement for research and measurement that clinicians would use in the field?
Mary Rauktis:: There are differences but there are also similarities. Regardless of the purpose, if it is for clinical practice or if it’s for research, you want the measures you use to have good psychometric properties, and internal consistency, for example. It is important, the reliability piece I talked about earlier. If they’ve been used in other studies, what have people found. For clinical reasons, I would always look for something that was brief with clear language; that is comprehensive but not redundant. For example, when we began to reconceptualize restriction and the Restrictiveness of Living Environment Scales, or ROLES, one of the things people said to us very clearly when we talked to clinicians and practitioners about it was, absolutely this needs to be revised, we need something new but don’t change it. Which sounds like it is a contradictory message. But what they were saying to us is, this is short, we can do it very quickly, it is in language we understand, it’s not redundant. So, don’t change that piece of it and it makes sense to us, it has face validity for us. So, for practice measures I would always look for something that wasn’t overly redundant, short, precise and often times that’s why in clinical practice you see measures often times like the CGAS for children, the clinical general assessment of functioning scale, which is actually I believe going zero to a hundred and coming up with a number. And it covers many things. Actually, its functioning is rather general. You’ll often times see the sort of general measures being used in clinical practice because it gives people sort of a place to put a stake so to speak. This is what I think, I’m seeing overall in this child or this adult’s functioning. In research often times you have measures that might be more redundant because you’re interested in generating new knowledge and less about using information to make decisions you might see longer measures, multi-item, where you might have 106 items about something. Doing 106 items in the field would be very time consuming and most clinicians won’t do it. But in a research setting, where you have perhaps more time and it’s not being used to make decisions about clinical care or effectiveness of care, then people might be more willing to do this. And it’s really about generating knowledge as the purpose. When I think about using measurement for clinical practice, it’s really about three things. One is decision support. Using the information to make appropriate decisions at key points. Okay, so I am using this REMY for example to determine whether or not a child can move up a level because they’ve been here for two months. I am using it for outcomes monitoring. When they leave, I want to see whether or not they’re leaving to an environment that is less restrictive perhaps than the one that they came from. The third purpose for me would be I would like to use it for quality monitoring, how well, sensitive is this to programmatic change. Does it tell me if my program overall was doing well or if its not? So, the utility piece of it for me, for measurement in clinical practice is really about decision support, outcomes monitoring, and quality monitoring. If you are interested in this area there is actually a very good book that addresses this in more detail by John Lyons and Dana Weiner. It’s a chapter in an edited book by Mark Maruish called Measurements as Communication and Outcome Management. And in this case they are talking about the utility of measurement for clinical practice.
[06:02]
Jonathan Singer: We can put a link to that book on the social work podcast website at socialworkpodcast.com. So it sounds like those are a couple important differences when I have worked with measures. This issue of face validity has come up and I’ll sit there and I’ll say, you know what, these questions have nothing to do with the kids I am working with. Or, but this is great but what do I do with it? That sounds like that’s really important because if you have a measure that was created for the creation of knowledge and somebody has told you to use it in your agency as a clinical measure, there can be a big disconnect.
Mary Rauktis: Yes.
Jonathan Singer: Not that that ever happens of course.
Mary Rauktis: No, never. Never.
(M and J laugh)
Mary Rauktis: Actually I have a wonderful quote from a parent about one measure that we were using a she said, “This is great. This just doesn’t tell me what’s wrong, but it tells me what needs to happen.” And that I believe sort of encapsulates this whole idea about measures and the utility of it. It’s just often times measures used for research purposes aren’t necessarily telling you what to do with it. They are operationalizing depression or opposition. But measures designed with the end in use, decision support, effectiveness of treatment, and quality monitoring have that end in mind. And they may look different and are focused on that idea. It tells me what to do. It gives me some guidance about what to do. I believe that when parents, families, clinicians have the opportunity to use this kinds of measures, they are much more likely to use them because it gives them sort of the map. On the other hand, I will tell you that it’s not necessarily something that happens, it doesn’t happen automatically. People need a little bit of training to do that. People need some help. Unfortunately, what happens is people perhaps will use a measure with someone and then they’ll put it in the record and to go on to write whatever they were going to plan to write anyway for their treatment plan. And so, people actually need to be coached about how to take that, again, if it’s a tool that was really designed for clinical decision making and for outcomes monitoring, they need some help in taking that and translating it into goes into a plan.
[08:38]
Jonathan Singer: I’m thinking back to when I worked in Austin and we had to do the child behavior checklist talking about CBCL and it was 113 items and it was always fun. And we had our administrative assistants who could score it and there was usually a one-week turnaround time. And I don’t think anybody used the CBCL as part of treatment planning. It was regulation. We had to get it at intake and then again at 90 days. Although I’m not sure if the CBCL was normed for 90 days, I think it was actually six months. Which is another issue; Using a measure differently that what it has been tested for. The second issue that it brings up is there is this agency issue, how fast can measures be turned around, how fast can you get the information, how to use it. The other thing is my experience is that schools of social work don’t train social workers on administering assessments or the interpretation of assessments. That’s really seen as a psychology thing. So, there is a disconnect, again, in what you’re saying about if you’re out in the field, it’s just not like it’s one on one or your traditional therapy but you’re working residential treatment facilities, you’re working in inpatient, you’re doing case management. Sounds like there are measures that can be used but these are not things that schools of social work actually train folks to do.
Mary Rauktis: I agree with you. I think that Education and Psychology, Schools of Education and Schools of Psychology, are much better at training students in that area. For example, I have a Master’s Degree in Education and we had two classes on testing, and just not traditional testing in terms of IQ testing. But also a full range of testing from career testing to testing for depression, anxiety etc. So, that’s a part of the curriculum. I would agree with you Jonathan. My experience is that there is much less emphasis put on this in social work historically. And what happens is people come out to practice in the field and they learn that on the job. But they don’t always learn it very well on the job because there often is not the time to learn. And so what typically happens, having worked in a provider setting for many years as a director of research, what happens is that this gets rolled into training perhaps on treatment plans. Which are often very much dictate by funders. And so it gets much less emphasis. People learn it in the field, but it really doesn’t receive the time that it deserves. I’ve done some, for example, some interviews with child welfare workers about how they make decisions and I find it really interesting that very few of them mentioned how they use a risk assessment matrix in making decisions.
[11:56]
Jonathan Singer: Is this risk assessment matrix something they’re required to do? Or they’re trained in using?
Mary Rauktis: Oh, very much. They are required to use and have extensive training. And so it makes me wonder if decision making processes, this is a fascinating area and beyond the scope of this podcast, but how we can help people change the scheme of how they make decisions to include measurement as a source of information. One piece but not the only piece, obviously a lot of things come into play. I’m finding out what they know from the past, how they were raised, what they’ve learned in school, but also this piece of information that they’ve collected in a systematic way. So that becomes more natural in the decision-making process. Because I don’t think it happens naturally all the time and then if people don’t learn it in their master’s programs and they learn it sort of haphazardly in the field, I just don’t feel like it get used very well. As you mentioned what happens is you do it to meet regulations and then it goes into the record and that’s it.
[13:06]
Jonathan Singer: As we’re having this conversation, I am wondering If it would be useful for students to receive information on measurements not just in a research class but in their practice class or their community organizing or whatever it is they do to have something built in to say, okay so you’re talking depression we’re talking about family therapy, or we’re talking about working with people in the community. How would you measure this idea? How would you answer this question? How would you help people to answer this for themselves?
Mary Rauktis: The way that we teach measurement is that we teach it in, we pull it out of context and that isn’t how people learn. My dream would be that research, in particular measurements, would be integrated into all classes in social work. So that people on the clinical tracks are learning about measurements as part of evidence-based practices. That it becomes part of their language. That in community organizing, that the measurement piece is part of the classes, the curriculum and just not a separate research class. That in administration, that people are talking about and using assessments of burnout or secondary trauma, etc. That it becomes not pulled out of context but integrated into these classes. That would be my dream.
[14:34]
Jonathan Singer: I think that’s a very reasonable dream to have. I know that in the practice theories course that I taught at the University of Pittsburg, we incorporated some of Scott Miller’s scales, outcome rating scale and the session rating scale, and students were really excited about that and really found it reassuring that they can administer a 4 item scale at the beginning of a session, at the end of a session, every couple sessions, gather data, be able to explain it to their clients, have their clients provide feedback, and sort of engage in that scientist practitioner model. And it fit. It wasn’t something that was like “oh no. Okay? Wait didn’t I take my research class already?” It really fit and I think it enhanced the practice theory course.
Mary Rauktis: I think it could be a natural enhancement of classes about supervision, administration, just not in your clinical classes, is your community organizing classes of social theory. Sometimes social workers come into the foundations of research classes that I teach, and they say, you know I’m really not a researcher, this has me very anxious. I hope at the end of class, people leave with a real sense of excitement about the fact they do research all the time, that they can. It’s not separate, something that’s done to them or they bring someone else in to do. But they do it. In the process of working with people, as you just described through Scott Miller’s work or if their supervisors, as social workers often are, and their working with new therapists or new clinicians that they can do the same kind of process with their supervisees. In terms of assessing how the supervision process is going. What they’re learning, what they’re not, where they need help. That this doesn’t perhaps need to stand up to the same level of scientific scrutiny. For example, if they were doing it for another purpose, but for the purpose they’re doing it works and makes sense. And most importantly they come out with the ability to do that from a school of social work.
[17:05]
Jonathan Singer: I think one of the challenges then to using measurement in the field where we are having this conversation is because students don’t leave having studied ten or fifteen different scales or they don’t have in their back pocket a couple of things they can pull out that they’re really comfortable with. The question is out of the millions of scales that are out there, how do you know where to get them? How do you know which ones to use? How do you even know what’s available? And you talked briefly about some resources but do you have other ideas for resources and places that social workers can go to not just learn more about measurement but to find some measures that they can use in their own practice or in the community?
Mary Rauktis: Well there is always a great resource book called Measures for Clinical Practice by Corcoran and Fisher and I believe they’re on their third edition. It might even be fourth at this point.
Jonathan Singer: And that’s through Oxford University Press. It used to be through Free Press but they changed their publisher.
Mary Rauktis: Yes. And I’d like to think that every non-profit would have a copy of this but I know that isn’t always true. But that’s a great resource to go to. I mentioned the book earlier, The Use of Psychological Testing for Treatment Planning and Outcomes edited by Mark Maruish, we’ll put the link to that. It’s relatively recent from 2004. Also I think a great resource for people is actually the internet. Now again, there’s a lot of stuff out there that you want to stay away from, it’s really a mixed bag. But, if you use something say like Google Scholar, and you begin to search that way. Say you’re interested in trying to find a good measure of I don’t know, social support. Well you can type that in, and then you’ll start to get some citations. Now many people in practice settings don’t have access to electronic databases the way we do if you’re sitting at a university, so I know that myself, having worked in a practice setting. So, what do you do? You can’t necessarily get the PDF of that article that you’re interested in. So often times in the abstract they’ll have the emails of the authors. I encourage people to email the authors and say “I’m very interested about what you’ve written about in this article, I can’t easily access a copy of it but I wonder if you can send it to me or tell me more about your measure of social support.” I find that people are remarkably generous with their time in doing this. I get many requests for things, like the ROLES or the REMY, and I’m really glad to respond to people to tell them where to go, to send them articles, copies, so there’s a whole community. The great thing about the internet, I believe for measurement purposes, is it has opened the community up a little bit, so you don’t need to necessarily go to a reference book. Although, those are great resources. But that you can, if you’re sitting in an office, in Clarion, Pennsylvania, and you’re not, well you’re actually close to the university there. Alright, another city. Someplace in the middle of Pennsylvania without access to university library. That you can use the internet to connect you to a community of people who know about this and who can advise you. When I was working in a practice setting, as I mentioned as a director of research, I depended on colleagues a great deal to help me out. So, we would use email and often times there are LISTERVs that you can go on and you can belong and it doesn’t cost anything. And people send emails out all the time. “Hey can you give me or does anyone know a good measure with come psychometric properties and validity or reliability about social support?” and then people will say to check out such and such. So it takes time, but I believe the internet has really created a community for people that we didn’t have before. So I would always check the classic sources, such as something like Corcoran and Fisher, but then I would also search using something like Google Scholar or PubMed which is free.
Jonathan Singer: Yup. It’s the government.
Mary Rauktis: Right. And to start to type in some general search terms and narrow it from there and see if you can find the names of people or access to free articles. They’re out there.
[21:46]
Jonathan Singer: I think that’s great. And we could put a link to Google Scholar and PubMED and the books that you’ve mentioned. One thing I did before I went back to get my Ph.D. was, I was still in contact with my professors. So, I’d send them an email, or I’d call them and I’d say, “Hey what about this” and of course it’s easy for them. Especially now, although it was less easy back then. It’s real easy now because almost everything is a PDF file, and they can say okay great, well here is the PDF file.
Mary Rauktis: Absolutely. I get emails all the time from students. Particularly when I’ve taught overseas in Portugal. I just got an email last week saying, “We really are interested in a measure of developmental changes for children and we know it won’t be in Portuguese, but can you send us some articles in English that might point us in the right direction?” So, I respond to these emails all the time. So yes. Keep the emails of your professors and send them emails about once you’re in practice. These are really good connections to keep up and social work professors are always happy to, I believe, to continue to respond to people for requests because my goal for people to leave a foundations of research class is for them to incorporate this into their practice. So, I will support that in any way when they leave. So, I’ve been known do to that a lot; to send articles, to find the PDF files for people and email them to them so that they can do this. So perhaps the lack of access to electronic databases doesn’t hold them back from doing it.
[23:35]
Jonathan Singer: Well that sounds very generous. I didn’t have any other questions about measurement. Are there any other hot topics that you think would be important for social workers to know about measurement that we haven’t touched on yet? And if not, that’s okay.
Mary Rauktis: My last comment Jonathan would be that measurement is important for social work. Once again, we often think it’s the domain of psychologists, but the fact is that measurement can be used against people or it can be a force for social justice. Which is important for social work. I think it’s important for social workers to be concerned about measurement and how its used and to realize that it isn’t only the domain of certain specialists but rather it belongs to the field as well, the field of social work. For us to be concerned about how certain things are measured and sometimes how they’re used. For example, IQ. That’s a whole other podcast. But for us not to believe that measurement is simply the domain of research and not areas that are traditionally associated with social work, justice, community, etc.
[24:53]
Jonathan Singer: As you said, it made me think that people in the field typically do not develop the measures. Even though they might be consulted by those who do. But there are a number of measures that are developed ostensibly for the field and so it sounds like one of the way that social workers can think about it is, if I really want to make an impact on this measure, I need to actually use it and give feedback.
Mary Rauktis: Yes.
Jonathan Singer: I can’t just say well I didn’t develop it; it was developed in an ivory tower so therefore it has nothing to do with me. But rather to say, you know what I have to use this so I’m going to let them know what I think.
Mary Rauktis: Exactly.
Jonathan Singer: This doesn’t work. This is 113 items.
Mary Rauktis: Right. Email, call, send letters. Absolutely, because measures should be revised. If they’re not working, then we need to know about it. So, there needs to be feedback, it just doesn’t go out in a journal and that’s it, it’s the gold standard forever. So yes, people should feel free to contact the developers and talk about it. The other thing is even though I encourage people not to create their own measures, I would tell you that you can do it and you can do it in a collaborative environment. So if you’re out working in the field and you really see a need for something, I would encourage you to try to find other collaborators including some people in universities, perhaps faculty that you’ve worked with in the past in your program, to work together. I mentioned in the earlier podcast the REMY. That was not developed out of a university. That was developed by five practitioner researchers who work in provider settings, Casey Family Programs, Pressley Ridge, Girls and Boys Town. So, it was developed working in conjunction with some people from George Washington University and with the University of Washington. So you can see, I’d like to see, more practitioners saying, hey we really need a new measure of such and such and working together. I might add actually without a budget, it was all in kind. So it wasn’t like you need a lot of money; you just need time, expertise, and help. So, I’m not suggesting that every practitioner go out and try to measure something, I will tell you it can be done and not to be closed to that possibility.
[27:26]
Jonathan Singer: And you’re actually talking about a measure, an instrument that could be outside of a specific setting in which it was developed. You’re actually talking about developing something with good psychometric properties and things like that. Which is different than creating the kinds of scales Insoo Kim Berg talks about in Solution Focused Treatment, where you say, okay so mom, tell me about what’s going on with you. Okay so on a scale of 1 to 10, this is your own personal scale that’s just for you. So those are two different types of scales.
Mary Rauktis: Right. Exactly. They both are useful, they have great utility. They are both for slightly different purposes, the purpose is different. But yes, you can do both in a practice setting. Social workers should also consider the fact that while they can do something that is unique perhaps to one client in one situation. Working in collaboration, they can also create something that has a different purpose and can go on to create the body of knowledge that validates that tool. We can do it all, Jonathan.
[28:39]
Jonathan Singer: We can do everything. We are social workers. Well Mary, thanks so much for talking with us today about measurement. It was a real pleasure.
Mary Rauktis: Well thank you for inviting me, it was wonderful. Thanks.
End Transcript
Transcription generously donated by Katherine Person, 2019 MSW graduate of the California State University, San Bernardino
- Google Scholar: http://scholar.google.com/
- Fisher, J, & Corcoran, K. (2006). Measures for clinical practice and research: Volume 1: couples, families and children (4th ed.). New York: Oxford University Press.
- Lyons, J., Howard, K., O”Mahoney, M., & Lish, J. (1997). The measurement and management of clinical outcomes in mental health. NJ: Wiley Publishing Company.
- Maruish, M. E. (2004). The use of psychological testing for treatment planning and outcomes assessment: Instruments for children and adolescents (3rd ed., volume 2). New Jersey: Lawrence Erlbaum Associates.
APA (6th ed) citation for this podcast:
Singer, J. B. (Producer). (2008, October 20). Measurement in clinical practice and research (Part II): Interview with Dr. Mary Rauktis [Episode 44]. Social Work Podcast [Audio podcast]. Retrieved from http://socialworkpodcast.com/2008/10/measurement-in-clinical-practice-and_19.html
No comments:
Post a Comment