This Fire Services Outcomes Framework – Progress Report provides the Country Fire Authority (CFA) and Fire Rescue Victoria (FRV) Quarter 1 (Q1) FY 2022-23 fire services performance indicators.
The purpose of this report is to meet FSIM’s legislative requirements under section 141 of the Fire Rescue Victoria Act 1958 (FRV Act) to publish a report that sets out the fire service agencies quarterly outcomes update and relevant, supporting data from CFA and FRV.
This report provides quarterly results provided by CFA and FRV against their respective performance measures and agency commentary on quarterly performance. FSIM has included historical data against measures provided by agencies, where available, dating back to the start of reform (1 July 2020) as a reference for the reader.
Reporting on Outcomes
Section 140(1) of the FRV Act requires CFA and FRV to each prepare an outcomes framework that sets out their respective outcomes-based fire service performance measures. In developing their outcomes frameworks, CFA and FRV each determined their own strategic organisational priorities and the associated indicators and measures of performance to be included in those frameworks.
Publishing agencies’ quarterly outcome indicator data provides a level of transparency of the fire services organisational trends and outputs as the work of the agencies continues. Two years into reporting on outcomes, both CFA and FRV have improved indicator selection and data identification and collection. Explanations of baselines, targets, and supporting data for many of the agencies’ indicators is progressing and some indicators and measures continue to be refined.
Agencies are not yet able to show clear linkages between actions taken or programs implemented and a resulting impact on the baseline/target, service delivery indicator or broader outcome. FSIM has noted this issue previously and recognises that it will take some time and agencies are actively considering how their program measurement can provide a story of progress or impact as their data and analysis further matures.
Tracking the progress of agencies towards their outcomes is a long-term proposition and many trends or patterns are only observable over an extended period of time. For this reason, FSIM intends to provide an annual assessment of agency outcome progress in its Q4 FY 2022-23 report. For quarters one to 3, FSIM notes particular indicators in the summary by exception where a result:
- shows a continuing off-track trend over repeated quarters
- demonstrates a notable achievement
- is a significant change from the previous quarter (positive or negative).
FSIM notes that some performance measures are influenced by factors outside the control of CFA or FRV and in some instances, agencies may not be able to provide substantiated reasons for changes to results within the 3 categories listed above. FSIM will continue to work with agencies to explore clearer linkages and explanations of contributing factors to results as their outcomes reporting continues to develop.
Maturing outcomes frameworks
The Outcomes Frameworks of CFA and FRV signal the strategic priorities of each organisation, and each provides a set of measures to demonstrate progress towards strategic goals. The ongoing review and refinement of both Outcomes Frameworks is important to assist agencies to assess progress towards delivering against their objectives and support informed, evidence-based decision making.
Across Q1 FY 2022-23, CFA has been developing its Year 3 Outcomes Framework, with this work focusing on the continuous improvement of metrics used (rather than a substantive change to structure or approach) and intends to report against the Year 3 framework in Q2 FY 2022-23.
FRV finalised its Year 2 Outcomes Framework in FY 2021-22 but continues to report against a selection of performance outputs in the absence of a full suite of measures against its year 2 framework. FSIM has included the targets FRV set in year one for these output measures as a historical reference for the reader, however, FSIM notes that FRV does not intend to set updated targets for these measures. FRV advises that it intends to report against all Year 2 Outcomes Framework indicators in Q2 FY 2022-23, using output measures as a proxy. This is because data to support meaningful outcome measures is not available for some of the indicators. FRV is therefore reporting on relevant output measures in their place, as these outputs are expected to contribute to achieving the intended outcomes.
Outcomes framework Q1 results commentary overview
Data quality
FSIM notes that in Q1 FY2022-23, FRV identified data issues, which resulted in a revision of previously reported data from FY 2021-22. The data issues affected 14,000 records relating to 8 measures[1] as a result of an IT incident occurring 1-3 September 2022. FRV detected the issue on 12 September 2022 and subsequently updated the relevant data. FRV has implemented new practices, including automated responses, 30-day error log for references and scheduled back-ups of the database to minimise the risk of repeated occurrences of this issue.
Service Level Agreements (SLAs)
FSIM has commented in previous quarters on the need to progress and finalise SLAs (also known by both agencies as Service Level Deeds of Agreements), that completion of clear, agreed, and implemented SLAs between agencies is fundamental to achieving complementary fire service and underpin the broader success of the Fire Services Reform. These SLAs provide a foundation for role clarity and functions for volunteers and staff in both agencies and are required to harmonise procedures. While FRV continues to report on the progress of SLAs (as part of its Outputs Performance report / year one framework and noting that updated targets for this indicator will not be set), the reported 88% relates to agreements agreed “in principle”, not executed agreements.
FSIM has noted in past reports that service level agreements have been challenging and complex and agencies have developed interim work arounds where a documented solution is yet to be agreed. Three years into reform, these foundational agreements – which set out operational and service delivery expectations and requirements – are not finalised. FSIM is aware of ongoing work by both CFA and FRV to negotiate, progress and finalise agreements and note that there is in principle agreement on a proportion of SLAs. However, the ongoing delays, and in some cases inability to finalise agreements, has impacted service delivery efficiency.
Safety and well-being
CFA reported a fourth consecutive quarter of not meeting its baseline for hazard reporting with a result of 34 reports against a 5-year baseline of 106 for the quarter. FSIM notes that as part of reform, CFA staff reduced by approximately one third and it will take time for the 5-year baseline to more accurately reflect CFA’s current staffing numbers and associated hazard reporting. CFA also notes that the key contributing factor to this result was the impact of COVID-19 related restrictions on brigade activities, including requirements to self-isolate. CFA is working to implement a new hazard reporting checklist to assist with targeting key hazards for identification and control and FSIM looks forward to CFA providing further detail about how the new checklist impacts CFA’s ability to provide a safer workplace, once implemented.
CFA did not meet its baseline this quarter for its decrease in emergency response injuries indicator. CFA reported an emergency response lost time injury rate of 79.8 against a baseline of 71.7 for this quarter. CFA intends to review its Occupational Health and Safety Management system including incident reporting and investigation procedures in FY 2022-23 to increase in the reliability of data and underpin more informed insights. CFA also did not meet its baseline for unplanned absences and correlates this result with an increase in COVID-19 related absences during the winter months compared to 2021.
Community preparedness
CFA has significantly increased the number of homes where members have installed a smoke alarm as part of their smoke alarm installation program, compared to Q1 FY 2021-22. CFA is also tracking well in all three related indicators, comprising installation of smoke alarms at homes, replacement of inoperable smoke alarms and increase in the number of vulnerable community members provided with a fire prevention advice.
FRV reported an increase in engagements for structured programs, largely due to Fire Ed sessions being delivered to schools. FRV also reported positive results for its outcome measure of number of people engaged by Fire Equipment Services which it attributes to the resumption and increased take-up (following the easing of COVID-19 related restrictions) of in-person emergency management training.
Capability
FRV continues to report below target results for both its specialist capability and core maintenance skills output indicators. FRV has updated its Skills Maintenance database (SMDB) to now include Division B data and reported a result of 86% against a target of 95%. FRV noted that performance was lower than expected primarily due to staffing issues such as unfilled vacancies and sick leave and changes to the database settings which negatively skewed the data. FRV has identified the issue and aims to rectify it by next quarter.
FRV reported a result of 43% against a target of 100% for its specialist capability indicator. FRV notes that skills maintenance activities for specialist capabilities are not conducted every quarter and, as such, Marine and Urban Search and Rescue Response (USAR), reported 0% for Q1 which reduced the overall percentage to 43%. FRV proposes to adjust the target to reflect the planned delivery schedule, rather than an even distribution across the 4 quarters which may provide a more accurate representation of specialist skills maintenance across the year. FRV also note that operational staffing requirements limit the number of staff that can be released ‘off shift’ for skills maintenance which is also impacting the result. To address this, FRV is conducting a review of all department positions to ensure there are enough operational positions and investigating strategies to encourage greater uptake of training positions. FRV is also working toward development of a learning management system, which should assist in addressing reporting shortfalls.
Footnote
[1] The eight measures are: 1.1.1a The number and rate of preventable residential structure fires attended; 1.1.1b Number of preventable residential structure fires attended; 1.1.1c Number of preventable residential structure fires by common causes and locations; 1.1.3 Percentage of residential structure fires with smoke alarms by type of residence; 1.2.1 Number of preventable fires in higher risk buildings (class 5-9); 1.2.2 Percentage of structure fires in non-residential buildings with smoke alarms or sprinklers; 1.3.3 Number of incidents other than structure fires and other non-fire related incidents; 2.1.1b Proportion of preventable residential structure fires contained to room of origin by time of day
Updated