2. Methodology

FSIM’s year three methodology

FSIM’s Annual Report FY 2022-23 objectives

FSIM is required under section 131(1) of the FRV Act to:

• report on the progress of CFA and FRV in completing implementation plan actions
• assess the effectiveness of methods CFA and FRV adopted in completing implementation plan actions
• assess the impact of the fire services reforms upon the financial sustainability of CFA and FRV
• assess ongoing efforts to improve the interaction between CFA and FRV.

This report provides a summary of Year Two to Five Plan action progress from the period 1 July 2022 and 30 June 2023. Specifically, the assessment acquits FSIM’s legislative requirements under section 131(1)(a)(i) of the FRV Act.

By assessing and reporting on the progress of Year Two to Five Plan actions, FSIM provides transparency and assurance to the public that government and fire services agencies are held accountable for the ongoing implementation of Victoria’s fire services reforms.

This report contains:

• an update on implementation of recommendations made in FSIM’s Annual Report 2021/22 (section three)
• potential barriers to reform impacting implementation of key Year Two to Five Plan actions (section four)
• a status update of all 41 Year Two to Five Plan actions as reported to the Minister (section five)
• a progress summary assessment of the 41 Year Two to Five Plan actions as at 30 June 2023, by priority area (section six)
• an assessment of impact of the reform on financial sustainability of CFA and FRV (section seven)
• FSIM’s concluding remarks for this Year Three report (section eight).

Assessing progress

FSIM monitors the implementation of actions by assessing progress against actions set out in the Year Two to Five Plan. In FY 2022-23, FSIM received progress updates from CFA, DJCS and FRV for all outstanding actions. In requesting documentation from agencies, FSIM requested evidence relating to how each action was being delivered, delivery against action milestones (where provided), risks impacting action delivery, and risks of the action not meeting its objective. FSIM communicated with nominated representatives of agencies to clarify information or request additional documentation. FSIM also referred to a range of publicly available information in preparing this report.

FSIM provides a context statement and progress summary to assess each action, and includes information relating to project management, communication, interaction between CFA, FRV and other organisations, reform activities transition to BAU, or governance where relevant.

Following each assessment of action progress, FSIM provides a ‘status’: progressing, implemented, partially implemented, or closed, along with findings on risks and issues identified. Table 2 describes the status that FSIM assigns for each action assessed.

Table 2: FSIM progress monitoring and assessment definitions

Assessing effectiveness

FSIM has legislative provisions, under s131(1)(ii) of the FRV Act, to assess the effectiveness of methods used or activities undertaken by agencies in completing an implementation plan action and meeting the action’s objective.

In FY 2022-23, FSIM did not evaluate the effectiveness of completed implementation plan actions against their intended objectives, as CFA, DJCS, and FRV were completing actions during the reporting period. FSIM considers the effectiveness assessment as a longer-term proposition given that embedding implementation plan actions is complex with multifaceted impacts that unfold over time.

FSIM intends to identify completed actions to be assessed for effectiveness in future reporting based on set criteria. In FY 2022-23, FSIM sought information from agencies, such as performance measures and supporting evidence, to demonstrate progress towards an action’s objective. FSIM will build on the information received to date and through engagement with relevant stakeholders to support evidence-based effectiveness assessments in future reports.

Information collection

FSIM collected information via multiple channels and stakeholders to inform the FY 2022-23 assessment. FSIM reviewed and analysed documents, attended events and meetings, and conducted interviews to inform this report. FSIM is a standing observer at the weekly HoA and the Fire Services Strategic Executive Committee (SEC).

FSIM’s approach to the collection, use and disclosure of information is underpinned by its obligations under section 138 of the FRV Act not to use or disclose confidential information obtained or received in the course of, or as a result of, the performance of FSIM’s functions except as permitted by the FRV Act.

FRV experienced a cyber-attack in December 2022, which severely impacted its information technology (IT) systems. In FY 2022-23, FRV undertook a significant program of work to investigate the cyber-attack and restore the IT environment. FRV advised that it developed and implemented interim solutions to maintain system functionality and data collection functions while developing a longer-term recovery solution.

FRV further implemented business continuity measures, workarounds, and contingencies to ensure service delivery continued. During this period, FRV crews continued to turn out and respond to incidents using mobile phones, pagers, and radios, and FRV maintained a range of corporate support functions. FSIM notes that the cyber-attack impacted FRV’s progress against some Year Two to Five Plan actions either through deploying resources formerly allocated to an action to address IT issues or through direct impacts on action progress. FSIM has noted in individual action progress summaries where the cyber-attack impacted action delivery.

While FRV was able to continue delivering services, the cyber-attack significantly impacted on its ability to provide documentation to support FSIM’s assessment against some actions. FSIM uses the terminology ‘FRV advises’ in instances where FRV has provided commentary on its progress against an action but could not provide supporting evidence due to the inability to access documentation.

Assurance principles

FSIM’s assurance activities are guided internally by an adapted set of assurance principles outlined in Table 3. FSIM adapted the principles outlined in the Assurance Framework for Emergency Management^[5] to incorporate two additional principles of independence and transparency that reflect FSIM’s functions. These principles guide FSIM’s approach in delivering a consistent and coordinated assessment of CFA’s, DJCS’s and FRV’s fire services reform program implementation.

Table 3: FSIM assurance principles

Footnotes

[5] Inspector General for Emergency Management, 2019, Assurance Framework for Emergency Management, Victorian Government, viewed 8 August 2023, https://www.igem.vic.gov.au/publications/igem-reports/assurance-framework-for-emergency-management