This Fire Services Outcomes Framework Progress Report provides the Country Fire Authority (CFA), and Fire Rescue Victoria (FRV) fire services outcomes framework performance indicator results and commentary for Quarter 1 (Q1) of the 2023-24 financial year (FY). This report also presents historical data on CFA and FRV performance compiled by the Fire Services Implementation Monitor (FSIM) and FSIM commentary by exception.
This report is provided in accordance with FSIM’s requirements under section 141 of the Fire Rescue Victoria Act 1958 (FRV Act). FSIM publishes these quarterly reports at Fire Services Implementation Monitor publications.
FSIM Observations
Due to ongoing impacts of the cyber-attack incident in December 2022, FRV was again unable to report fulsomely on their outcome indicators. FRV reported on 12 (out of 19) quarterly indicators, and 19 measures out of 29 quarterly measures. CFA reported against all its quarterly indicators and three annual indicators from its outcomes framework.
I have now published three years of CFA and FRV quarterly outcomes data demonstrating:
- trends of agencies not meeting targets (e.g. FRV structure fire response times),
- agencies reporting on indicators which are heavily influenced by factors outside of their control (e.g. CFA number of house fires and number of grass and scrub fires and FRV number of preventable non-structure fire incidents), and
- a significant lack of data (and therefore fulsome reporting) by FRV.
Agencies consider the published outcomes important to their strategic goals and many outcomes and indicators relate to the safety of staff, volunteers and the broader Victorian public. Reflecting on published data and commentary received over the first three years of outcomes reporting, the following questions continue to arise:
- What actions are agencies taking when a rolling baseline, benchmark, target or broader intention (e.g. decrease/increase of an indicator) is not met?
- What are the impacts on Victorian public safety of not meeting these benchmarks, baselines, targets or broader intentions?
- When will FRV be able to fulsomely and retrospectively report on all indicators?
- Should outcomes, indicators or measures be amended or adjusted to more effectively capture the agencies’ role in delivering an outcome?
I have acknowledged in previous outcomes framework progress reports that reporting on outcomes is a long-term proposition and agencies have matured their data collection, analysis, and reporting approaches and frameworks. While agencies are not yet able to fulsomely answer these questions with respect to many of their indicators, it is important for agencies to, as far as practicable, explain what actions they are taking that are impacting results, particularly with respect to key public safety indicators and those that demonstrate progress towards longer-term strategies.
I recognise that this level of detail is not feasible nor appropriate on a quarterly basis due to the significant data collection and complex analysis required. However, there is benefit in yearly commentary on whether agencies consider their progress against outcomes as aligned to their longer-term vision and whether agencies consider that they are on the right pathway to deliver their intended outcomes.
I look forward to agencies being able to more fulsomely address these questions in future reports this financial year and encourage agencies to take action to address any agency-identified deficiencies in the outcomes, indicators and measures in their current frameworks and provide this information to the Victorian public.
Reporting on outcomes
Section 140(1) of the FRV Act requires CFA and FRV to each prepare an outcomes framework that sets out their respective outcomes-based fire service performance measures. In developing outcomes frameworks, CFA and FRV each determined their strategic organisational priorities and the associated indicators and measures of performance to be included in those frameworks. FSIM has published CFA and FRV progress reports for
Q1 2023-24 as appendices to this report.
Tracking the progress of agencies towards their outcomes is a long-term proposition and trends or patterns may only be observable over an extended period. FSIM therefore provides commentary by exception on indicators and measures that:
- demonstrate a notable achievement.
- show a continuing off-track trend over repeated quarter.
- vary more than 5 per cent from the agency’s baseline or average over time.
- are a significant change from the previous quarter (positive or negative).
FSIM notes that some performance measures are influenced by factors outside the control of CFA or FRV and in some instances, agencies may not be able to provide substantiated reasons, supported by evidence, for changes to results.
Maturing outcomes frameworks
Outcomes frameworks, when used to guide decision-making, can inform progress towards strategic goals and provide insights to inform how agencies respond and adapt to emerging trends. Ongoing review and refinement of outcomes frameworks is important to assist agencies to assess progress towards delivering against their objectives and support informed, evidence-based decision making.
CFA and FRV outcomes frameworks are based on each agencies’ strategic priorities. CFA and FRV set their own relevant indicators and measures, and agencies have been reporting on their outcomes frameworks since July 2020. Reflecting on three years of quarterly results, there is an opportunity for agencies to more comprehensively demonstrate how the outcomes frameworks and results are informing strategy, planning, and decision-making. When metrics are trending negatively, or where a metric is significantly influenced by factors outside of the agency’s control, agencies have the power to be accountable and identify an appropriate response to reverse the trend or re-evaluate the intended purpose of metrics to drive organisational change and continuous improvement.
For example, FSIM has observed a concerning trend that FRV has not met its target for measure ‘2.1.3a Percentage of response to structure fire incidents within 7.7 minutes (target 90 per cent)’ since Q2 2020-21 (12 consecutive quarters). FRV has indicated that factors outside of its control have consistently contributed to this trend. As part of FRV’s proposed outcomes framework review, there is an opportunity for FRV to:
- undertake further analysis, in districts where this target has been met, to identify the extent to which factors external and internal to FRV have contributed to meeting this target.
- fulsomely report on the extent to which external and internal factors contribute to not meeting FRV’s target.
- develop a new measure(s) to monitor and report on factors within FRV’s control to meet this target. This will allow for more meaningful reporting; it will also allow FRV to monitor the impact of its interventions to influence this measure.
For these frameworks to be useful in the context of the wider Fire Services Reform, agencies have provided information to demonstrate how they are using their data to guide decision-making, adapt responses or amend metrics and measures to be more useful and appropriate.
Updates to the agencies’ outcome frameworks for Q1 are outlined below.
CFA’s outcomes framework 2023/24
There have been no changes to CFA’s outcomes framework this quarter.
CFA is currently finalising the annual review of its outcomes framework with the Year 4 outcomes framework expected to be applied in Q3 outcomes reporting. Minimal changes are expected given the substantive update in 2022.
CFA advises the changes being considered will support greater clarity and transparency to reporting. Often when new indicators are introduced, historical (baseline) data is not available. The opportunity to enhance (or introduce) baselines and targets will be considered when the Year 4 outcomes framework is applied to Q3 reporting. The review process presents an opportunity for CFA to review targets and baselines and create new ones, where appropriate. FSIM supports the use of evidence-based targets and baselines where appropriate, as they can provide a level of accountability, clarify organisational expectations, and enable a more comprehensive understanding of the impact of interventions or activities. This in turn can make reporting more comprehensive and useful for future use.
FRV’s outcomes framework 2023/24
This quarter, FRV incorporated a new measure ‘3.4.1b: Number of incidents outside of FRV district’ under the indicator ‘3.4.1: Increase in collaboration, consultation and advocacy with Country Fire Authority and other sector stakeholders to continually improve community safety and emergency management.’ FRV considers that this measure provides an indication of cooperation with the CFA in the Country Area of Victoria (CAoV).
From Q1 2023-24, FRV has discontinued reporting on the quarterly measure ‘% WorkCover claims who ceased work with no return to work’ (formerly 3.1.1a). FRV report that it does not believe it was a meaningful indicator of the impact of FRV’s return to work activities due to the way the indicator is measured. FRV report that under its Health Model program, work is underway to establish a health, safety and wellbeing reporting framework informed by metrics that demonstrate progress in a more meaningful way.
FRV advise that they are intending to undertake a fulsome review of their Outcomes Framework once a dedicated performance management specialist has been onboarded. Recruitment for this position will commence in January 2024, with the review expected to be completed within a 12-month period. All measures within the current outcomes framework will be subject to this review, as are other FRV performance measures deemed within scope. The review will determine the scale of potential changes to the outcomes framework.
There is an opportunity for the review to consider FSIM’s observations as part of this process and clarify timelines for when the review will be finalised and when FRV’s revised outcomes framework will be available. As noted with CFA’s review, FSIM also sees an opportunity for FRV to consider the use of evidence-based targets and baselines where appropriate, to provide additional transparency on FRV’s progress.
FRV cyber-attack impacts on outcomes framework reporting
FRV report that the cyber-attack in December 2022 continues to have a severe impact on its information and computer technology systems. FRV is still unable to access the Australian Incident Reporting System (AIRS) (a national database that collects data on fire services reported incidents and response) and FireCode (a service management tool that captures the scheduling of fire protection maintenance of fire safety assets) and cannot provide results on operational indicators and subsequent measures for:
- 1.1.1 Decrease in preventable fires in homes
- 1.1.3 Increase in homes with working smoke alarms
- 1.2.1 Decrease in preventable fires in higher-risk buildings
- 1.2.2 Decrease in impact from fire due to early detection and suppression systems
- 1.2.4 Number of services provided by Fire Equipment Services (FES) to maintain fire protection equipment and essential safety measures in the built environment.
- 3.3.1 Increase in use of and participation in research, data analytics and evaluation to support evidence-based decision making and improvements to community and firefighter safety.
FRV also advise that due to the cyber-attack, it cannot report on measure ‘2.1.3a Percentage of response to Road rescue responses within 13.5 minutes (target 90 per cent)’, which informs the indicator ‘2.1.3 Improved readiness for all emergencies, including large or complex incidents, for specialist rescue, and for times of surge demand’.
FRV confirm that it is not able to retrospectively populate metrics as there has been no extensive post-incident data recording while AIRS and FireCode have been unavailable. FRV reports that AIRS is likely to be restored by the end of 2023, and FireCode will likely be restored by April 2024. Once AIRS and FireCode have been restored, FRV can commence reporting on associated metrics.
In the interim, FRV has obtained data from the Emergency Services Telecommunications Authority (ESTA) Computer Aided Dispatch (CAD) system as an alternative to AIRS data. FRV confirms that ESTA CAD data has been validated to ensure its integrity and comparability. FRV reported that a comparative analysis of both data sets, for a period before the cyber-attacks, confirmed that the difference between the two data sets did not materially affect the results. FRV found that incident volumes and response times were similar for Structure Fire, EMR and Code 1 classifications in both data sets.
FRV reported five operational measures using ESTA CAD data in Q1 2023-24:
- Number of properties with six or more false alarms on different days (rolling 12 months)
- Number of structure fire incidents
- Number of incidents other than structure fires and other non-fire related incidents
- Percentage of response to structure fire incidents within 7.7 minutes
- Percentage of response to EMR (emergency medical response) within 9.2 minutes.
Industrial action impacting FRV data quality
FRV advised that FRV staff undertook industrial action during the periods of 27 October 2022 to 20 April 2023 and on 12 August 2023. From this industrial action, when FRV staff responded to ‘False alarm’, ‘False Call’ or ‘Good intent calls’, they reported such calls as ‘Other’. This has had an impact on FRV’s ability to reliably report on indicator ‘1.2.3 Decrease in unwanted false alarms to reduce unnecessary emergency response’ and the ‘False Alarms and False Calls’, ‘Good intent calls and ‘Other’ categories for measure ‘1.3.3: Decrease in preventable non-structure fires and other hazard related incidents’ during reporting periods: Q2, Q3, Q4 2022-23 and Q1 2023-24.
Due to a separate ongoing protected industrial action undertaken by some CFA staff, FRV have advised that it has modified data reporting to reflect results for metropolitan appliances only when reporting on measure ‘2.1.3b: Total operational fleet availability’. FRV report that since the start of this industrial action in the end of March 2023, reporting on fleet availability has been limited due to regional appliance information being unavailable.
Updated