Based on your discussion of the pros and cons of various measures in the discussion thread this week, make a final decision on which measure(s) should be used in your program eva

2

Week 4 discussion

Student’s Name

Institution Affiliation

Date

Participants to Be Evaluated

Participants to be measured will include those who have, in one way or another, been actively engaged with the MHA programs, whether through direct participation in stigma reduction or increased access to mental health services. Subjects will be selected using stratified random sampling to ensure a balance in demographic representation according to age, socioeconomic status, geographic location, and past experiences with mental health services. This helps ensure that the assessment captures the diverse experiences of targeted groups of the program. However, stratification can be resource-intensive, requiring very careful planning to identify and recruit participants from all relevant subgroups.

Staff Administering the Evaluations

The assessment will be carried out through MHA's trained staff or the contracted assessors. These shall carry out various surveys and interviews and gather information using observational approaches. Evaluators should be trained to ensure consistency and minimize evaluator biases when collecting data. Experienced personnel may increase validity and grant more weight to the assessment, although, in this case, requiring more training may elevate the cost and lengthen project timelines.

Sample Size and Assessment Schedule

The sample will be targeted to ensure a minimum of 300 participants per program for statistical significance. This number creates a reliable comparison across subgroups and program sites. The participants will be assessed at three key points: at baseline (before intervention), immediately following the implementation of the program, and six months post-intervention. This longitudinal approach provides insights into both the immediate and sustained impacts of MHA programs. While this design realizes valuable data longitudinally, the impact of participant dropout, especially in long-term follow-up measurements, may jeopardize the reliability of such findings.

Validity and Reliability of Instruments

The tools to be used in the assessment include but are not limited to the Mental Health Knowledge Schedule (MAKS) and the Patient Health Questionnaire (PHQ-9) which have been pretested. MAKS captures(stigma-related knowledge and attitude) while the PHQ-9 focuses on mental health outcomes corresponding to the severity of depression. The validity of the two tools used here has lots of support. For example, the works of Singleton et al. (1998) show that self-report questionnaires such as MAKS have been determined to possess construct validity since they ranked high on the criterion measure of validity In a similar vein, works carried out by Kroenke et al. (2001) reveal that the PHQ-9, a self-administered inventory that is popularly used in mental health research has strong scores in Construct validity. Besides, these tools have a powerful reliability; for example, PHQ-9 is said to be 0.89; which reveals that this tool has recorded high levels of Cronbach’s alpha. That is why their application at MHA may require certain modifications to meet the organization’s goals and objectives more accurately.

Cost and Accessibility of Instruments

Cost is a very sensitive parameter when it comes to decisions regarding which tools would be most suitable for evaluation. PHQ-9 is also in the public domain and therefore relatively cheaper for this evaluation. On the other hand, adaptations by MAKS may need permission or charge for this sort. If cost becomes an issue then other freely available tools such as the Generalized Anxiety Disorder (GAD-7) scale can be looked at as options. However, using public-domain instruments usually involves a low cost, but it is unclear whether they can be customized according to MHA’s specific program requirements, and whether they are suitable for this kind of task at all (Torjesen, 2022).

Conclusion

Meanwhile, utilizing a balanced implemented, stratified random sampling valid, and reliable equipment and a long-term research timetable will allow MHA to avail themselves of the measurement of programs for effective outcomes. This will provide important information on matters bearing stigmatization and improvement of the health system in a sampled and cost-effective manner. This will further improve MHA’s capacity to adapt programs in practice and be better placed to meet the needs of society’s needy groups( Fink, 2015).

References

Fink, A. (2015). Evaluation fundamentals: Insights into program effectiveness, quality, and value

(3rd ed.). Thousand Oaks, CA: Sage.

Torjesen, I. (2022). Access to community mental health services continues to deteriorate, survey

finds. BMJ: British Medical Journal (Online), 379, o2585.

https://doi.org/10.1136/bmj.o2585

,

Okay. Welcome to Week four in Program Evaluation. I'm actually restarting a recording, which you won't know because there wasn't anyone in attendance tonight, but I thought I'd mention it because everybody has scaffos, and I didn't like the way the recording was going. I even had my first interruption by my dog this semester. I decided to start fresh and go through the material again. Again, that isn't something that is all that relevant, but I thought that I'd mention this is maybe my second time through some of this material. Again, we're in Week four. This is going to be a very busy week with assignments, as I've promised next week, Week five, will be very heavy lifting work. We'll be using SPSS. Some of you will be using other programs you've talked to me about. We'll be entering our made up data and getting into assessment of it. But for this week, we're going to be continuing on with our methods section after doing a discussion. So we got started with the method section last week with a couple of the subsections within that, and now we're going to go through a couple of more, a few more actually. But as always, let's start with our discussion this week. Essentially, our work is our assignments. O assignments are our work. So we just go through that each week. For the discussion this week, you're going to be involved in selecting your measure for this program evaluation. Again, we're only selecting one measure. Some of you have gone as far as two measures, but in this eight week class, we're really completing the whole report between weeks two and seven. There's really no way to go too much further than that. The selection of measures or measure is an incredibly important aspect of program evaluation. So states our discussion. In your discussion, you should discuss possible measures that could be used in your program evaluation at your agency, including the pros and cons of the possible options as they relate to the following issues. Um The first thing we really want to get at here is how we want to assess the variable we're most interested in in our program evaluation. By going through the questions that are listed here and really evaluating the pros and cons of each measure you consider, I think you'll be able to narrow it down pretty quickly to the one or possibly two measures you want to use. Again, they're going to be pros and cons here. There's no perfect measure out there. That's why we always run multiple studies with multiple measures when we're doing academic research. In the real world, we're sometimes constrained with program evaluation. In this case, as I've already mentioned, we're really constrained in our quasi program evaluation or made up evaluation this semester. Some of the things to think about when you're thinking about the pros and cons of a measure, who are the participants that will be evaluated? Are you looking, for example, at children, at adolescents, at adults? Are you looking at people who have mental health symptoms, at people who have drug abuse or use symptoms, et cetera. You really need to pick a measure that's reliable and validated for the population, the sample you're looking at? In particular, again, you want to make sure that your instrument is reliable and valid for children if you're looking at children in your program, adolescents, if you're looking at adolescents in your program, and adults if you're looking at adults. The other thing to bear in mind when selecting a measure is who are the staff who are going to be administering the evaluations. We ourselves are not the researchers here. We don't go in and do this. We advise and then we deal with the data. But the folks who are going to administer and collect the evaluations are the people who work at the agency. So upon picking a measure, we have to be sure that it's appropriate to be used by the folks who will be administering it. So for example, make sure if there's a licensing requirement for the measure you select, that the people at your agency would be licensed to do that. Sometimes it can take three to four days or longer in training and then getting some supervision before you can use it. And a lot of people can't allow that time for agency employees to go get the training. So ideally you would pick a measure that doesn't have any licensing requirements or anything particular in terms of training for the agency folks to administer the evaluation. Another thing to keep in mind here is that self report measures are probably going to be your friend. You won't need a clinician for a self report measure, and a lot of our agencies only hire a few or maybe no clinicians at all. Self report is going to be really the direction you head with most of these measures. Yeah. Another thing that I made a note of was, if you use more than one agency representative to evaluate or administer rather these evaluations, you would then have to get an inter reader reliability. Again, we don't want to go very far with that in this class. You do want to have a sense of how many participants will be evaluated because price of the instrument of the measure is always very critical, and so bear that in mind as well. If you're going to be giving the evaluation, the instrument at both intake and discharge, now you have double the cost if there's a big cost involved. Bear in mind that knowing how many folks are going to be involved is very important, how many participants will be evaluated because that will affect the price of your instrument if there is one attached to your instrument. You also want to think about the assessment schedule. In other words, how often and when the participants will be evaluated. That's already what I was hinting at in the previous statement. That will impact how many of the instruments you actually need, and that again, might come back to pricing. Numbers five and six are very, very important and where you're probably mostly going to bring in your sources. First, when you mention your measures, in the discussion, you'll need to give a source from where you learned about these potential measures. But now when you describe their reliability and validity, you very much need to provide sourcing citations. In the discussion and later in the report. Sections we'll talk about for your assignment this week. You really need to show the evidence, show the citations for the instruments validity, and for the instruments reliability. A couple of times now I've mentioned price already. Mainly when thinking about who the participants would be, how many there would be, who would be administering it and how often. But do think about the cost of the instruments. Can your agency afford to pay for the instruments? Are there alternatives in the public domain that would work? Google is really going to be important this week. You need to take a look out there for the various measures, particularly that are in the public domain, for the variables that you want to assess. Evaluate alternatives if you're looking at something that's quite costly. Now we'll move into the assignment for the week, and essentially, you're going to be continuing the method section that you started last week, the method section that you just started last week. Just as a reminder, before we get into this assignment, here's where we've been with this report and where we're going this week. You won't be turning in these prior sections this week, I'm grading them week by week, and then in week seven, you'll glue them together in a final report. But two weeks ago and the one that you've already gotten feedback then on this week. You were discussing the introductory section of the report, and that would have included the name of the section, I'm sorry, the name of the agency, a brief description of the agency, the problem, the program, and the evaluation. And then last week, which you would have just turned in by yesterday, so I haven't graded them yet, but I'll get to them later in the week. You would have turned in the subsections of a new section called method, that would have included the program and the design. Most of you were to skip the definition section, and last week was a very brief assignment. Just a couple sentences long, especially because you probably did skip the definition section. Now where we're going this week, Is continuing, me, jumped ahead there, the method section with three new subsections, the setting, the participants and the outcomes and measures. All of this you will have thought through in the discussion already. Now you'll write the formal report for the setting, the participants, and the outcomes and measures. To look back at this in terms of how it's presented in the assignment, It says, based on the discussion of the pros and cons of various measures in the discussion thread this week, make a final decision on which measures should be used in your program evaluation. This week, you will be continuing the method section of your program evaluation report. Please again, refer to page 222 and complete the following subsections of the method section. Let's start with setting. First of all, you're going to flush left, as your other subsections have been under the method section, which was centered in bolted. Flush left, put the word setting in bold, and then continue on. Essentially, you're going to ask yourself, where at the agency is this assessment going to take place? You've already described the agency in the introduction. Again, here, you're just going to have one sentence describing where in the agency, you'll be conducting the evaluation. The sentence will read This evaluation will be conducted at fill in the blank for your program for your agency. For example, this evaluation will be conducted at the Adult outpatient building. Period. Again, you might have to make this up because you aren't actually interviewing in every case someone from the agency, you're gathering this information from the website. Maybe you would say, this evaluation will be conducted at the agency's main offices or at the agency's library or at the agency's conference center. There could be a variety of words you fill in at that end of the sentence, but give a sense of where this is. Again, you may have to make it up, but I want you to have a completed report, so you have a template at the end of this class for any future program evaluations you might engage in. Next will come participants. Again, flush left, bold the word participants, and also include the assessment schedule in this section. The participants may be everyone who gets admitted to the program between certain dates, and you should specify those dates. Again, the assessment schedule in this section. One or two sentences might suffice here. For example, all clients who are admitted to the program from June 1, 2025, to June 30, 2025, will be evaluated, right? I meant to say January 1, 2025 to June 30, 2025. If I didn't say that, that's what I meant to say. Let me repeat. All clients who are admitted to the program from January 1, 2025 to June th, 2025, will be assessed or will be evaluated. They will be assessed at intake and discharge or whatever other assessment schedule you decide is relevant. Okay. And now we get to this very important section, which will be longer than the first two. Again, your settings, just one sentence, your participants, just one or two sentences. Now we get to the outcomes and measures portion of your report. Be sure to include research evidence of the reliability and validity of the measure or measures you selected. This will already be in your discussion. Now you want to get it formally into your report for this assignment portion of what you should get done this week. Essentially, you want to make sure that you cite the validity and reliability sources you have for this measure, that you use APA style for those citations, and if you want your reference section that you include this week, could be a running document or page that includes last week's citations as well as this week's citations. So I know I don't want you to repeatedly turn in sections. But if you want your reference section to just be a running document, so you don't have to in week seven suddenly be alphabetizing things and so on. That'll be fine with me. I might make the occasional comment. I don't see this citation referenced in your paper, but I think I'll remember that it's because you're keeping a running reference page. So that will be fine. Otherwise, don't turn in anything you've previously turned in. Not this week. In Week seven, you'll turn in everything as one Amalgam report. Now, when it comes to outcomes and measurements, measures, you need to include the outcome and measure that you finally focused on that you decided after the pros and cons, did the best job of focusing on your symptoms that you're interested in. It might be mental health symptoms. It might be substance. It might be eating disorders. The symptoms associated with reading disorders. So here you put in the instruments and the reliability and validity to justify your choice of the instrument. You might sort of go over what you've thought about in the past in terms of what the problem is and so forth. But this week, you really want to focus on that outcomes and measurements. This could be a few paragraphs, which we haven't done yet in this report outside of the introduction to the agency. So this really could take time because you need to make sure you provide the evidence for the validity and reliability in the same population you're going to be using that you're going to be sampling, even though our sample, as we've already discussed in the report, probably be the entirety of the population. But that's very important. You want to make sure, for example, that you don't pull a measure that's only been validated in China, for example, when you're using a North American or US data population here, excuse me. Make sure that it's validated on folks from America, people of the age group that you mean to, both genders, and so on. This is normally where I would leave room for questions about specific projects. I'm going to throw out some ideas that I was thinking about and I have come up with former students before things that you could Google and look into. The main thing here is Google Google Google, public domain and making sure you match your population and making sure you have a reliable and valid source. Couple of things to keep in mind. As far as measures go, you may want to think about using the depression anxiety stress scale. This is the DA SS, the Academy of Pediatrics. Uses this depression anxiety, stress scale, DA, SS. If you're looking at children or adolescents, it's been u validated and is reliable for seven to 18 year olds. I know sometimes people come to me looking for measures that deal with that age range. Seven to 18 is the DAS, depression anxiety stress scale because it may be different than using the B depression or the B anxiety and so forth. For substance abuse, you want to turn to NDA as the experts, the National Institute of Drug Abuse We're always going to do with drug abuse, a urine or drug test first before we give the self report. The reason for this and you should document this in your outcomes and measure section is that the research shows that people are more honest on self report if they've just taken a urine drug test. Even though it's not going to be our major measure, it's not our primary outcome, we're still going to do it because it will increase honesty on the self report measure you then might select. Just mention that you're going to do A Cleo waved CLIA waived ten panel urine test. Again, I'm going to give you the exact wording for this. It'll be a leo waved ten panel urine test for drug use. Okay? And you can Google that again and you'll be able to put in the information and so on. But very important that you pay attention to that. And as you're looking for tests, look at the price because that'll be important to your agency. You'll want to report that. So when I Googled, for example, there were 25 tests for $118. But it's also possible that your agency already pays for these urine tests, and they would already have included that in their budget, and they won't mind. But I thought that $25 for 118 I'm sorry, 25 tests for $118 was a good A good benchmark. Now, for your primary outcome when you're studying substance abuse, you may want to use the ASI, the addiction severity index. Again, you'll do this at intake. This would be a case where you'd have to have some of the folks at the agency who are certified to give the ASI. So look into those details there. Addiction, severity test. If you don't use the whole scale, but you just use section or items, excuse me, D one through D. Then you wouldn't need a certified individual to evaluate the addiction severity. There's all these different sections on the ASI. If you just focused on items D one through D 13, that would help you with a good outcome there. Um If you go with that, the ASI and just that section d1d 13, still say that there's reliability and validity, but say that you're only using items D one through D 13 and the ten panel urine test. What else did I look at? I looked at some anxiety and depression scales. You might be interested in using your measurement. We've already mentioned the DAS. No, this one is the DA SS 21. This is a good one for anxiety and depression. It's a world health organization instrument. The UN has approved it and it's been re It's been shown to be reliable and valid through many countries. You could also use the symptom checklist, which is known as the SCL, the SCL 90. It'll have nine subscales and 90 items. Twice, you'd have to enter 90 items because you'd be doing it at pretest and post test or intake and discharge. Think about that because that's a lot of data entry. Maybe you're interested in self esteem. There's a Rosenberg ten items self esteem test in the public domain. That's that's for people ages 12 and up. Again, here you could Google self esteem Assessment for children. You can also use it for adults, Rosenberg, ten item Scale. As you're looking at your measurements this week, this is to look forward to next week a bit. Notice if there's any reverse scored items, you're going to have to think about that as you do data entry next week. As you're reading through the items, do you notice that most go in the direction of showing severity of, say, drug abuse, but some items make it seem like low severity? That would mean that you're going to have to reverse score those items. We'll talk more about that next week. But the more you've looked through your measures by next week, the clearer you'll be about that assignment, Gosh, this is where I would say, are there any other questions? Do people want to bring up their agencies as examples and so on? We're such a tiny class that, of course, just reach out to me if you have questions or if you want to run your measure by me, you do not have to. That's what the discussion for, and I can give you feedback there. Then you can write up your report. This section of the report, the setting, the participants, and the outcome and measures. But this is literally where I would say, are there any questions? So We won't have any. Given that you're not here this evening. But again, it's a tiny class. And so we're just going to work around that the best we can. I hope I gave some examples that hit home for some folks, but it'll really depend on what agency you're evaluating. Okay. Good bit of work this week, next week, a lot of work. Again, bear that in mind with the holiday coming up. Hopefully, you got my email about that as well, and I'm enjoying reading all the plans you have so far. Thank you for the hard work you've been doing. Take care. Okay.