Deb Maher: So I'm Deb Maher. I'm the Director of the Student Mental Health Branch in the Department of Education. I'd like to introduce my colleague Phoebe Monagle, who is the manager of the Schools Mental Health Fund and Menu unit. And we're also joined today by Dr Georgia Dawson and Courtney O'Brien from the University of Melbourne's Faculty of Education.
Thanks for joining us today for the Schools Mental Health Menu evidence briefing, and we really hope that this session will be beneficial for your organisations. Please keep your microphones muted during the presentation and there'll be an opportunity for questions at the end of the presentation. I'd like to kick off by acknowledging the custodians of the diverse lands on which we're meeting today and acknowledge Aboriginal and Torres Strait Islander people and their ongoing connection to land, sea and community.
I'm speaking today from the lands of the Wurundjeri people of the Kulin Nation and wish to pay my respects to their elders past and present and extend that respect to any First Nations people joining us today. We acknowledge the protective role of culture, identity and connection to country for Aboriginal social and emotional wellbeing and recognise the leadership and ingenuity of many Aboriginal elders, communities and community controlled organisations in supporting better social and emotional wellbeing outcomes for Aboriginal people.
Welcome to the briefing today. It's really great to talk to you about the role of evidence in the development of the Schools Mental Health Menu and how your organisation can build a high quality evidence base for your programs. We're really pleased that over 200 people have registered to attend this session because it really shows that providers of mental health and wellbeing programs for young people share our understanding about the importance of evidence. We'll kick off with just some housekeeping. So as I mentioned earlier, if you've got any questions, please use the chat function as we go along, and we'll make sure we spend some time at the end of the presentation to respond to the questions in the chat. I'll also take this opportunity to inform you that the briefing's being recorded so we can publish it and share it with other interested people. And as a reminder, as was outlined in the invitation, this briefing won't provide you with any information about further ITS processes. But now I'm going to hand it over to Phoebe who will take you through the Fund and the Menu. Thanks Phoebe.
Phoebe: Thanks Deb and hello everyone. So I'll start with a very brief overview of the Fund and the reform landscape that it operates in. In 2021, the Royal Commission into Victoria's Mental Health System recognised the important role that schools play in providing universal supports for mental health and wellbeing of young Victorians. And it made a recommendation that we provide schools with funding to purchase evidence based programs that support the mental health needs of their students. The Victorian government invested $200 million over four years and 86.9 million ongoing to create the Schools Mental Health Fund.
So the Fund is a new, additional and permanent part of the Victorian government school funding system. Schools must use their funding to implement programs, supports and resources from the Menu. They must spend and acquit their Fund allocation during the calendar year in which it was received, and they can add funds to expand their use of programs and resources on the Menu. They can also use their allocation to fund relief teachers to enable staff to attend professional learning or other activities related to implementation of the Menu. And they can also work together and pool funding to share programs and resources, which is something that we know smaller schools and rural schools are doing.
So we'll have a quick look at the Menu and how it's structured. So the Royal Commission acknowledged that mental health and wellbeing is an important issue for all students, and consequently, the Fund and Menu includes a focus on positive mental health promotion and whole school approaches.
You can see on the diagram on the slide, those preventative approaches are captured in the green base of the triangle in Tier 1 and schools are encouraged to focus on Tier 1 programs and priority, sorry, are encouraged to focus on Tier 1 as a priority.
So we'll now have a look at the evidence requirements for the Menu and why programs included on the menu must have a strong evidence base.
So the Royal Commission's recommendation and the Menu are part of a broader shift towards developing an evidence based approach to supporting the mental health and wellbeing of students. Outside of the Menu Invitation to Supply process, there are other significant benefits to collecting evidence for your program, and it should form part of your regular evaluation and improvement activities. Evidence collection highlights areas of your program that could be developed further, supporting continual improvement, which will ultimately benefit the groups your program is targeted towards. It also allows you to identify the cohorts that benefit the most from your program and, possibly more importantly, who your program does not work for.
There is increasing evidence highlighting the risk of universal programs to a small cohort of students. Being aware of this risk allows you to develop supports and strategies to reduce any harm that may be caused to some students. And, even if you decide not to apply for any future ITS processes, we strongly recommend collecting evidence for your program as it is ultimately for the benefit of your audience.
As was outlined earlier, the Royal Commission recommended that schools be provided with funding to access evidence based initiatives. To ensure the evidence requirement for the recommendation was met, the department built a rigorous evidence evaluation into the ITS process for the Menu. The specific requirements for the ITS undertaken in 2024 are on the slide. And as you can see, the ITS asked program providers to submit clear evidence showing that they had achieved positive outcomes related to student mental health and wellbeing. Programs also needed to demonstrate that they had an evaluation process that supported program improvement and that they were suitable for intended cohorts.
Please note that these requirements are from a previous ITS process, and while they'll be useful in guiding your evidence collection, the specific requirements may change in future ITS processes. So to ensure the ITS evidence assessment process was rigorous and fair, the department engaged the University of Melbourne to conduct the evidence evaluation. We will go into detail in the next section, but here are a couple of high level points to note about how evidence is assessed.
So firstly, the evidence needs to show that your program, when implemented in educational context, has positive mental health and wellbeing outcomes for students. And I'd really like to emphasise that this is both mental health and wellbeing, so programs have to demonstrate outcomes in both areas. Secondly, the evidence must be related to your program. Evidence that supports the general approach of your program can be included, but program specific evidence is preferred.
You can see on the slide the evidence ratings for the current Menu, and it's really important to note that NO programs were appointed to the Menu with an evidence rating of less than two. An evidence rating for each program are published for schools to access and we understand that schools really value being able to quickly identify the level of evidence for the programs that they are interested in purchasing.
So I'm now going to hand over to Georgia from the University of Melbourne, who's going to go into some more detail around the evidence assessment process that the university undertakes as part of the ITS process.
Georgia Dawson: Thanks so much Phoebe. So this slide provides a very high level summary of the Invitation to Supply evidence assessment criteria for the 2024 process. And first of all, just to reiterate that when evaluating program evidence, only studies conducted in an educational context and that demonstrated positive mental health and wellbeing outcomes for students were considered.
In that top box as well, high quality evidence was represented by studies with a control or a comparison group. So this is where participants are allocated to either an experimental group where they receive the intervention or a control group where they don't, and differences in outcomes are compared after the intervention has been implemented.
This tells us if the program is effective over and above a group of students who might engage in regular activities. Preferably participants are randomly allocated, as is done in a randomised controlled trial. Random allocation decreases the risk of bias in group selection and systematic error in the comparison of group outcomes.
For some of the more established programs, high quality evidence was also represented through systematic reviews where researchers identify and synthesise the empirical evidence regarding a program's effectiveness, and it's the aggregation of individual studies for the same intervention to understand an overall effect of the intervention.
In that middle section, programs were also considered to be evidence based if they were tested using observational studies pre and post the intervention on mental health outcomes. This was considered foundational and emerging evidence, However, direct exports of routine post program satisfaction surveys were not sufficient to be considered a study demonstrating positive outcomes in student mental health and wellbeing. Programs that could show positive outcomes from studies examining the underlying practice on which the program logic was based were also considered to have foundational evidence. And finally, programs were considered to require further research around level one when the evidence for the program or practice was inadequate or not yet established and as mentioned, might have only relied on post program satisfaction surveys.
I'm now going to hand over to Courtney to provide further detail about the types of evidence evaluated and general findings from the assessment process.
Courtney O'Brien: Thanks Georgia. I'm going to walk us through some findings from programs in the 2024 process. So we're first going to look at some success factors from programs that submitted quality evidence and that we evaluated highly. So the first of those is measuring student outcomes both before and after they completed the program.
So we of course mentioned this before, and that routine post program satisfaction surveys are not sufficient to be considered a study that demonstrates positive outcomes of the program. And this is because they don't give a baseline, so we don't know if students perhaps had high levels of mental health and wellbeing before the program. Understanding then where students started and ended up allows us to measure the change in outcomes and gives more confidence in saying that these outcomes improve through completing the program. The next success factor was comparing students who completed the program with a control group who did not. So comparing the pre and post program data of students who completed the program with those who did not complete the program takes the quality of evidence a step further. It shows us whether change happened because of the program or if it happened because of other outside influences like students maturing, seasonal changes in mood or other school supports. So a control group is a marker of high quality research, and it does make us feel more confident in findings of positive student outcomes.
The next is programs that created opportunities for independence in their research. So research can of course be time consuming and expensive, but some providers partially avoided this by collecting their own data through routine program delivery and then commissioning an independent researcher or organisation to analyse and write it up. This brought expert knowledge to the evidence as well as supporting objectivity and reducing bias and again strengthened the credibility of findings.
Next is a clear underlying practice outlined in the program logic with evidence for this practice supplied. So programs that clearly defined their underlying practice in their program logic and supplied evidence of this practice achieving positive student outcomes were evaluated at least at level two, even if they did not have or weren't able to show that program specific evidence.
And lastly, for success factors, we've got using standardised measures to assess outcomes. So standardised measures are tools, often rating scale questionnaires that go through a process of validation to ensure that they accurately measure what they're supposed to. E.g., anxiety or wellbeing, as well as standardisation to make sure that they are used in a consistent way.
Some also have norms where they are tested on large populations to know what's considered typical or average for different groups, so that scores from individuals or groups can then be compared to the broader population.
Next in the orange box, we're going to go through some common pitfalls from programs who perhaps were evaluated lower than they expected.
So firstly, we've got not presenting a sound logic model based in evidence. So without a clear logic model, programs lack a clear evidence based explanation of how their activities and their underlying practice are expected to lead to positive student outcomes, which is at the heart of all of this.
Next is ignoring evidence of practice that has informed the program in favor of limited evidence for the program. Some providers focused on inadequate program specific evidence, like those routine post program surveys we mentioned or student testimonies, and they didn't spend enough time or words in their application. Specifying details of research that supports beneficial outcomes from the underlying practice that the logic model is based in.
So as a result, some providers that had inadequate evidence for level three, which is foundational and emerging evidence, then fell at a level one requiring further research, and they missed out on achieving a level two evidence for the underlying practice, which is the minimum level for inclusion on the Menu.
Next we have providing evidence that supports the need for the program. So some providers focused on providing evidence that their program is necessary rather than that it works. E.g., a program looking to decrease bully supplied statistics around bullying and research on its trajectories, rather than demonstrating that the practice of education or that their specific educational program decreases bullying or associated outcomes.
Next we have sending unprocessed data. So multiple providers sent us excel files with individual student survey responses or graphed aggregated survey results by question. So for us evidence needs to show that defined outcomes have meaningfully changed, and so that data needs to be analysed to be considered and evaluated. Next we have outdated evidence. So we are typically looking for evidence to be within a five year time frame, although this is case by case, and we will accept randomised control trials or systematic reviews outside of this due to their scale and the time it takes to conduct them.
Some providers submitted evidence for their programs underlying practice from as far back as the nineties, and the context of schools in young people's mental health and wellbeing has changed a lot since this time. So we hope to see up to date evidence conducted in this climate to be able to be confident that the program's mechanisms can achieve positive mental health and wellbeing outcomes for students now.
Next, we've got using artificial intelligence to write the ITS response. So AI can actually generate fake references that may include plausible elements such as real author or journal names or a convincing title or digital object identifier link but that don't actually exist. This is because AI language models are trained to patent match and they don't typically actually search databases. So we would encourage program providers to engage directly with the research themselves to build that deeper, more informed understanding of their program's evidence base. And last but not least, we've got being the jack of all trades master of none. So some providers tried to address a really wide range of outcomes, think anxiety, depression, self esteemed, social skills, bullying, resilience, academic motivation, all within a single program. So these programs therefore lacked depth and precision in addressing any one issue, which meant that the evidence for each individual outcome was inconsistent or weak.
So if we talk a little bit more about linking evidence to student outcomes, of course we mentioned that part of the criteria for consideration for inclusion on the Menu, is that evidence demonstrates both positive student mental health and wellbeing outcomes in an educational context. So if we look inside this circle, we have some examples of relevant student mental health and wellbeing outcomes that we've seen in program applications.
Thinking about positive outcomes, we might see improvements in self esteem, school engagement, mental health literacy, help seeking, emotion regulation, resilience, upstanding behavior, social connectiveness or wellbeing. And when it comes to negative outcomes, we might be looking to see decreases in anxious or depressive symptoms, challenging behaviors, mental health stigma, negative attitudes, substance use, bullying or stress. But as mentioned before, we wouldn't expect or want programs to be trying to impact all of these outcomes. Just the ones that are relevant to your program's focus and logic model. Outside the circle, we have some related outcomes that are not relevant to students or the educational context that we have seen focused on in applications, but they're not sufficient on their own. So one of these is adult outcomes. While these are important and often change in teachers or parents can support change in school aged children, it's not the current scope of the Menu. So we're looking to see flow on effects to student outcomes from programs targeted elsewhere. E.g., training for teachers on implementing a trauma informed approach might improve students' self esteem and decrease their trauma symptoms. Outcomes achieved outside their educational contexts are another one. So some providers submitted evidence of school aged children, participating in programs in other environments such as in elite sports clubs or inpatient environments. So programs do need to show that they are applicable to the school environment because differences in implementation can affect the extent to which outcomes are achieved. So in the broader literature, there are some interventions that have had good evidence for some groups or in some settings that haven't shown the same efficacy for outcomes when applied to students in the school context. E.g., mindfulness-based stress reduction has strong evidence supporting its effectiveness in reducing stress, anxiety, and depression in adults in clinical context and corporate environments.
However, high quality studies such as the myriad trial in the UK have shown limited to NO significant mental health benefits for when mindfulness is applied in the classroom. This is why it's important to demonstrate positive outcomes in the educational settings specifically. As factors such as developmental stage, implementation, engagement, and a context match have then been considered and investigated.
So I'm going to provide some illustrative examples of evidence.
So the evidence levels are based on the quality of evidence or the degree to which it is shown that student outcomes have been significantly impacted by the program or the underlying practice. So we'll look over some examples of evidence for each level. First I'll speak generally, and then I'm goiong to use a fictional acceptance and commitment therapy or ACT based program that we have called Activate your Mind to illustrate this.
Firstly, a level two, programs evaluated at this level usually publish or submit published research papers such as studies, systematic reviews or meta analyses that show evidence for the underlying practice informing their project logic. Program logic, sorry.
These could be cognitive behavior therapy, animal assisted programs or trauma informed teacher training, for example. They either do not have evidence for the effectiveness of this specific program or they may submit program specific evidence that is not conducted in the school context or is raw survey data that has not been sufficiently analysed.
So for our example, a program based in acceptance and commitment therapy without their own program specific data might submit a systematic review and meta analysis of ACT school based intervention randomised controlled trial studies. Programs at a level three usually submit a report or study with pre and post program survey data conducted either internally or independently. This data can include anything from program created survey statements to standardised measures looking at specific outcomes, but they must be analysed in some way to show either improvements in positive outcomes or decreases in negative outcomes. E.g., here we have a pre post delivery impact evaluation of the ACT program who obtained quantitative data from 50 students and analysed it to show significant increases in resilience and reductions in stress, which was also supported by qualitative data.
Lastly, programs at level four usually submit a report or a study that includes a controlled group of students who do not receive the intervention, which is used for comparison against students who do receive the intervention. This evidence may use program created measures, but often use those standardised measures that I mentioned earlier.
Due to the scale of these studies, they're often conducted by external organisations such as consultancies or universities. And, for example, the ACT based program might provide a randomised controlled trial with a large sample that also showed significant increases in resilience and reductions in stress for those who completed the program compared to the control group of students who did not.
So while we acknowledge that collecting evidence can be an expensive and time consuming process, there were a few smaller scale programs that were able to provide really good evidence. E.g., one program that was evaluated at level three, selected standardised outcome measures, collected their own data, and did their own analyses.
There were also a number of programs that applied back in 2022 and then reapplied in 2024 that improved their evidence rating level in this time period by collecting new evidence or refining their application. Most programs move from providing evidence for their underlying practice to providing evidence that they collected about the outcomes of their specific program.
There were a few notable improvements, so one was a peer support based program that moved from a level one, requiring further research, to a level three, foundational and emerging research, and was ultimately selected for the Menu. In 2022, this program did not have any of their own evidence, and they struggled to define their underlying practices and associated evidence.
But by 2024, this program had refined their program logic, collected their own data from students before and after the program and had this data independently evaluated by a lecturer at a university. And this evaluation found that compared to before the program, students experienced increases in positive mental health indicators.
And my second notable example is a social and emotional learning program that moved from level two, so evidence for the underlying practice, to a level four, established evidence, and also was selected for the Menu. Since their last application, the providers had commissioned a comparison study with a control group which showed significant improvements in students' overall emotional expression and understanding of their feelings when compared to students who did not complete the program. And prior to this independent comparison group, the organisation had also commissioned an impact assessment strategy and implementation plan. So on that note, I'm going to pass back to Georgia to talk about emerging topics in evidence for the future.
Georgia Dawson: Thanks Courtney. So, as the evidence landscape in mental health programs for education continues to grow and expand, which is so wonderful to see, a number of emerging areas are becoming increasingly important to be aware of as program developers and delivery partners. And attending to these emerging areas will build robustness and quality in the evidence for your program. I think together as a group of stakeholders with a shared interest in child mental health and supporting schools, it's important not to shy away from considering these areas, as addressing them will have positive benefits for students in schools and for the quality of your program.
So the first area that's already been briefly mentioned that's gaining a lot of research attention is around impacts on students that were not intended or resulting worsening of symptoms or other harms. And we refer to these as unintended consequences. They can have a positive or negative effect other than what was intended from the intervention or program and can include a harm, which is a negative effect on a student or a group of students that could plausibly be linked to the intervention or program. Within health and medicine, adverse outcomes must be reported as a compulsory measure, but this is not the case in other fields including education. So it's important for us to take the lead and be aware of this. The increased focus and attention on these adverse outcomes and even null effects in school based intervention has primarily emerged from the the big UK based myriad study that looked at a mental health intervention across 26000 adolescents in schools in the UK.
Detailed analysis from the study showed that some groups did not benefit from the intervention and a small group were in fact worse off after the intervention on measures of well being and mental health. The study also revealed that many students didn't engage in the intervention activities at home, so there were implications for dosage and the exposure effect, and considerations for the level of interest in the program from students. This is all really critical information to know and understand as program developers. So the key things to consider based on this research and evidence is around the status of mental health in some students before an intervention and that there can be an impact on how they respond to school based interventions resulting in a worsening of symptoms for some students. We really are still trying to understand the underlying mechanisms of this, but one possibility might be that increased introspection on your own mental health can reinforce any existing distress you maybe feeling. The second key finding to consider is adherence to a program might reflect the developmental stage of some students and therefore there may be differential beneficial effects. So for example, older students maybe more likely to adhere to a program of social emotional learning skills practice than younger students, it might feel more relevant to them.
Our recommendations in this area based on the work of Lucy Foulkes and others in the UK is to accept that unintended consequences are possible with your program, ensure that you identify them and measure them and report them.
To consider any dropout profiles or students that might have high absenteeism from your program and understand if there might be patterns within that. And to consider the opportunity cost of participating in the program, is it better for some students not to participate and to do something else? The key message here is don't always look for universality. Some programs are better targeted at specific groups of students. So along a similar line, the second area that we wanted to mention today is understanding needs and outcomes for specific groups, and we're considering this really in two broad areas culturally and linguistically diverse students and neurodiverse students. And we wanted to mention that some programs do this component very well, noting how they have adapted their approach to suit different student needs. However, not all universal programs are designed with adaptation in mind. So some of our key findings are, there are a limited number of programs specifically designed for students from a culturally and linguistically diverse backgrounds and that content in current universal programs suggests a large emphasis on western conceptions of mental health and wellbeing.
More specifically from a First Nations perspective, programs should be specific to this group and built with the community, not for the community. Several examples of this exist but more are are needed. For neurodiverse students, consider approaches used within the program that might not be conducive to neurodiversity. That might be things like the type or length of activities, the need for introspection and also consider that in the neurodiverse population, some students have higher rates of poor mental health.
So our recommendation here, first relates to design where we suggest you use the existing research and evidence base to understand whether your approach and topic area suits specific groups, whether it might need some adaptation or co-design or it might not be suitable for these groups at all.
And to consider identifying and measuring the impact of your program on specific groups of students so you understand the effect that it's having on those specific groups.
The third emerging topic in evidence for education is around student voice and incorporating student voice across program design, development, and implementation. And this parallels policy and grant funding guidelines around lived experience and community involvement. Recent work by our team has investigated the scope and extent to which student voice is captured in school-based universal mental health interventions, and we looked at interventions in the last ten years. Our systematic search identified really only 27 studies that mentioned student voice across that ten year period. In analyzing our data, we explored voice as either process, so what is found or voice as influence, what is done with student voice, based on the work by Gilett Swan and Baruitsas.
The majority of studies, 23 out of 27, captured student voice as process. So this occurred post implementation through the assessment of retrospective experience, including things like intervention appropriateness, feasibility, acceptance, and to a lesser extent adverse outcomes.
Very few of the studies we found used student voice as influence in the initial design of the program or by incorporating feedback into adaptations or iterations of the program. So we see this as an area of development for the field.
Our recommendations are to capture student perspectives and experience as a key stakeholder group, to do this in a developmentally appropriate way depending on the age groups you're working with and do this early in your design phase and use it to shape your program development.
We also recommend you ensure there's a process for continuing to use student feedback to iterate and improve the program and extend this to other stakeholders, including school staff, incorporating them into the evidence building process as a key voice, taking them with you in your evidence building journey.
And speaking of journeys, we wanted to mention that we continue to see great evidence in education for programs that support the mental health and wellbeing of students. And we also acknowledge, as Courtney did earlier, and understand that evidence takes time to establish and requires resources and funding. We recommend thinking about your evidence for your program in terms of a maturity or lifespan model, building the scope of evidence as you go and not trying to do everything all at once. So if you're in early stages where you're conceptualizing or designing your program, the key things to develop are a strong logic model, and I think that message has come through from the feedback we've given around the assessment process. And that logic model should be underpinned by the research and evidence in your program's topic area. What does your program say or do and how will you show that it's done this? A logic model is a map of impact, but also of your process. It includes the inputs to your program, the activities that occur, and the outcomes you want to see change in. It's a great opportunity at this point to embed student voice and other stakeholder engagement as you test your ideas. So key questions in this phase are, is there a need? What does the evidence tell us about this issue or our approach? And methods and activities will include co-design, listening to need, and reviewing the research evidence. You may then move to a small scale pilot that is testing the feasibility of your program.
Questions to ask here are, does it initially work? Can it work? Is it acceptable? What did participants think of it? Was it practical for schools to implement? It's generally smaller scale and pre post analysis and measurement of outcomes. Data collected might be through interviews and focus groups with participants to understand their experience of your program. You'll also test the measures you have for your outcome variables to see if they work well and are adequately measuring what you hope to see change in.
Based on the outcomes of your feasibility and small scale pilot testing, you may then work towards a large scale testing of impact and implementation and include a comparison group. Questions to ask here are, does it work? Does it work compared to the normal timetable or other activities? Who does it work for? Who doesn't it work for? Activities will include pre post assessment of impact between the intervention and comparison group, generally larger sample sizes than your feasibility. Evaluation of your process, did things happen as they were supposed to?
Subgroup analysis to understand the specific groups where it does or might not work as well and establish any unintended consequences and adverse outcomes. You may also incorporate independent analysis of data at this point.
Finally, seeking opportunities for grant funding to provide resources to evaluate and build evidence, and to work with universities and other research organisations to find opportunities to do this. Early on, they maybe able to give access to student researchers that can support the smaller scale, independent analysis of your program before you move on to anything larger. And on that note I'm going to hand back to Deb, thank you.
Deb Maher: Thank you much Georgia. So we'll just quickly run through some next steps.
So, while the timing and the date for the next Invitation to Supply process is yet to be determined, we strongly recommend you consider what's best for your organisation when deciding whether or not to make an application. As many of you will be aware, the application process is rigorous and labor intensive, and while we don't want to discourage any providers from applying, it may be worthwhile to consider collecting and continuing to collect evidence in developing your program in preparation for the following ITS process rather than submitting an application for one that's open at the time.
In addition, we also encourage suppliers to carefully read information about scope to ensure that your program meets the requirements as out of scope programs cannot be included on the Menu. Noting that scope and requirements may change in future ITS processes, we thought it might be useful just to call out the previous programs that were out of scope for the 2024 ITS process. So these included one off presentations or incursions that did not include supporting teaching resources, digital mental health or wellbeing apps and check in tools, and programs that primarily support teacher and or parent/carer mental health and wellbeing. So please ensure that you you review the scope and requirements when the next ITS is launched. So that concludes the formal part of our briefing.
[End of transcript]
Updated