The scope of the evaluation was developed by PWC in collaboration with FSV and focused on the following five key questions which addressed systems, practices and processes at The Orange Door:
- What changes can be made to establishment approaches to strengthen future implementation and operations.
- Intended operations, systems and processes at The Orange Door.
- The impact of the initial service offering for improving client experience and client and system outcomes[1].
- Ensuring The Orange Door workforce has the resources, capacity and specialist expertise to undertake key functions.
- Supporting better integration and coordination at the local level.
FSV notes the overarching evaluation findings and key opportunities for improvement cover a range of cross-cutting themes. In addition, responsibility for many of the opportunities for improvement are shared between FSV and the partner agencies of The Orange Door.
In considering its response, FSV acknowledges the interconnected nature of the evaluation findings and has grouped its response by the following ten themes:
- The Orange Door foundational model
- Client experience
- Governance
- Connection with the broader service sector
- Establishment preparation
- Demand management
- Workforce
- Risk assessment, management and information sharing
- Physical infrastructure
- IT systems
At Appendix 1, a breakdown of FSV’s response in respect of the individual findings is also provided.
The Orange Door foundational model
The evaluation identified that clearer operational guidance is required for The Orange Door workforce to better understand and operationalise the foundational model. The workforce would benefit from refinement of practice guidance and foundational documents which were intended to support commencement of The Orange Door and initial steps towards achieving the reform intent described in the Support and Safety Hubs Statewide Concept.
In addition, ongoing development of consistent tools, practices and processes would support implementation
While The Orange Door was designed to improve perpetrator accountability within the family violence system, the evaluation found there was a need to strengthen integration of perpetrator services into The Orange Door. Although men’s/perpetrator services is a comparatively less mature sector, the evaluation identified the potential to leverage the skills and experiences of Aboriginal services to build workforce capacity in more integrated, holistic ways of working to better keep perpetrators in view.
Assist the workforce to understand and operationalise the model.
To strengthen the understanding of key foundational concepts and their practical application amongst The Orange Door workforce, FSV is undertaking a range of practice development activities to embed consistent and effective practice within the operational model. These include activities focusing on specific areas of practice (e.g. assessment and case closure) and working with practice and team leaders to support ongoing implementation. The latter will occur through establishment of a Practice Development Working Group.
The induction program for The Orange Door workforce has also been refreshed based on initial feedback from practitioners, with expanded modules and further opportunities for practical learning tailored to The Orange Door service context and operational priorities. This has been developed with input from peak bodies and sector experts.
Greater need to focus on the integration of perpetrator services into The Orange Door.
The Orange Door enables a more comprehensive ‘view’ of the perpetrator than has previously been available to inform risk assessment and risk management across the family violence system. FSV is taking a number of steps to support the ongoing integration of perpetrator services into The Orange Door including:
- developing a perpetrator accountability plan as part of responding to the Expert Advisory Committee on Perpetrator Interventions Final Report. The perpetrator accountability response will be underpinned by the Multi-Agency Risk Assessment and Management (MARAM) Framework and its supporting tools and resources.
- leveraging existing skills and experience in some areas, such as the holistic healing and integrated approach used by Aboriginal services when responding to people who use violence to better keep perpetrators in view, and
- trialling several perpetrator interventions projects to provide an evidence base to support the development and maturation of this sector
Governance
While partnership formation and governance arrangements in the early implementation stage were found to be contributing to positive practices such as information sharing to support service delivery and risk management, the evaluation also identified inherent complexities around collective leadership. This included challenges associated with bringing together numerous partners who may have competing interests, different practice philosophies and expectations.
Developing a shared vision and collective accountability within a multi-disciplinary and multi-agency setting.
FSV is committed to embedding collective leadership within governance arrangements in each of the areas and continues to work with sector representatives and peak bodies at a statewide level to support a collaborative approach to meeting the stated aims of The Orange Door. Delivered in partnership between government, community services organisations (CSOs) and Aboriginal services, the partnership approach is formalised through a Partnership Agreement.
As part of the planned review of the Partnership Agreement, FSV has undertaken significant consultation with Hub Leadership Groups (HLGs) and Operations Leadership Groups (OLGs) in each area to discuss the effectiveness of the partnership approach, governance structures and the terms of the Agreements which has informed revised Agreements for operational areas and new Agreements for the next areas to be established.
Work is underway to more clearly articulate accountability within the governance structures for The Orange Door and the relationship with pre-existing local area governance structures.
FSV is also developing Quality Governance guidance for The Orange Door to provide structured operational guidance and to support the effective implementation of the Department of Health and Human Services (DHHS) Community Services Quality Governance Framework. The Framework outlines key features of quality governance to better ensure consistency across the sector and applies to all community services that are delivered, funded or regulated by DHHS.
Work with partner agencies to define a practical shared measurement/performance framework for The Orange Door.
Although the Partnership Agreement outlines the commitment to developing shared performance and accountability measures for the governance of The Orange Door, FSV recognises the need to better align the outcomes expected of the partner organisations with the outcomes of The Orange Door.
Work is currently underway to develop an outcomes-oriented performance framework for The Orange Door, core and partner services delivering family violence and family services. The new performance framework will be oriented towards the achievement of outcomes for people and the service system and promote collective accountability and integrated service delivery. It will be aligned with the whole of Victorian government Family Violence Outcomes Framework and other relevant government frameworks.
Establishment preparation
The evaluation found the scale and complexity of the family violence reforms and the associated roll out of The Orange Door had significant impact on the ‘volume of change in practice’ for the workforce and recommended that the sequencing of implementation and culture change required greater consideration prior to operational commencement.
Improve the sequencing of activities required for the establishment of The Orange Door prior to service commencement.
FSV has since updated its ‘pre-conditions’ for service commencement of The Orange Door in new areas to reflect the workforce, infrastructure and IT requirements identified in the evaluation. Implementation planning for the next areas has also considered the lessons learnt in relation to time needed for critical establishment activities such as recruitment and infrastructure fit out. In order to address the significant change in practice for The Orange Door workforce, FSV is developing a change strategy to better support implementation, which will focus on supporting individuals to successfully transition to the new model.
Co-development of practical operational procedures in conjunction with practitioners.
FSV is working with partners to strengthen consistent alignment across workflows, processes and practice in The Orange Door. To support this, a Practice Development Reference Group has been established, with membership including practitioners, team leaders and practice leaders from The Orange Door as well as managers from partner agencies.
Workforce
The evaluation identified a need for greater clarification around the standardisation of roles and responsibilities across partner agencies at The Orange Door. This was particularly in relation to the matrix management model and the associated decision-making and accountability considerations.
Given the variation in pre-existing skills, knowledge and training across workforce, multi-disciplinary training and professional development were also identified as critical for establishing a level of consistency in integrated practice at The Orange Door. FSV has started to address these issues through The Orange Door workforce strategy.
The workforce desires clearer guidance on roles, responsibilities, and accountabilities.
FSV is currently developing an accountability framework that aims to clarify roles and responsibilities for the key functions of The Orange Door. Furthermore, a specific review of Practitioner and Team Leader roles is planned as part of the workforce strategy to support greater consistency in role descriptions across partner agencies. Providing clearer guidance on roles, responsibilities and accountabilities will also be supported through implementation of the Family Violence Multi-Agency Risk Assessment and Management Framework (MARAM) and supporting practice guides
More training and development opportunities, tailored to the context of The Orange Door, are needed.
FSV continues to work with relevant peak bodies and sector experts to develop comprehensive foundation training, including evidence-based theoretical frameworks and best practice principles in the context of The Orange Door. To increase awareness across sectors and encourage opportunities for integrated practice, foundational training for practitioners on each of the three specialisms has been expanded (specialist family violence, child and families and men’s/perpetrators). The Inclusion Action Plan and the Workforce Strategy for The Orange Door will include commitments to training The Orange Door workforce in working with clients with diverse needs.
The workforce strategy[2], developed in consultation with peak bodies and partner agencies, will also support partner agencies across the employee ‘life-cycle’ of recruitment, training and development, retention and exit. The Orange Door workforce will also be supported in aligning family violence risk assessment and management practices to MARAM through rollout of the MARAM training modules.
Physical infrastructure
Physical infrastructure issues impacted the initial commencement of operations and colocation at some of The Orange Door areas.
A range of physical infrastructure issues impacted operations.
Since the commencement of operations, FSV has continued to assess and rectify physical infrastructure issues identified and has revised its pre-conditions for service commencement to reflect the lessons learned in establishing and fitting out the physical premises. Plans for the physical design of premises in the new areas have been updated with the operational experience gained in the first four areas.
FSV engaged expert advice to inform the security infrastructure and design in the operational areas and is currently reviewing implementation of these elements. Security assessments were undertaken pre and post-service commencement and a number of improvements have been implemented to ensure client and staff safety, with continued improvement of security policy, practice and infrastructure planned.
Client experience and outcomes and related data quality.
Although the evaluation was unable to assess client outcomes and experiences in depth due to a lack of data collected directly from clients, it was able to draw findings around client choice and agency through interviews with a limited number of clients. Direct consultations with practitioners and their engagement with clients also informed this area of the evaluation.
Overall, practitioners demonstrated a strong commitment to a client-centric approach to practice, and this was particularly apparent in allowing for the voices of Aboriginal clients to determine service choice and support options.
It was recommended that future evaluations use direct client feedback to determine client experience and outcomes, including the need to consider how responsive and accessible The Orange Door is for diverse client groups and whether the workforce has the skills, resources and capacity to meet these needs.
Client choice and agency is at times limited by a range of factors.
FSV is building on its current program of work relating to client experience and bringing the voice of clients into The Orange Door through the development of a client partnership strategy which will work in partnership with clients across design, development and operations. FSV is also currently introducing a client voice process to collect and analyse feedback from clients on their experience of The Orange Door.
Improved data collection and reporting in the Client Relationship Management (CRM) system is required to monitor accessibility and responsiveness for diverse communities.
FSV is committed to the collection and use of high-quality data, and to the development of a strong evidence-base that will enable us to better understand the impact of our services for clients and the service system. To enable this, FSV is identifying ways to improve data collection, access, relevance and reporting processes across The Orange Door, specialist family violence and sexual assault services and perpetrator interventions, in alignment with work led by DHHS for child and family services.
The Orange Door was designed to meet the needs of people from diverse backgrounds and be safe and accessible for both clients and the workforce. Since it was established, FSV has implemented some and will implement a further range of initiatives and strategies to meet the intersectional needs of clients and improve related data collection, including:
- the Intersectionality Capacity Building Project for family violence and universal services workforces
- development of an Inclusion Action Plan to embed inclusion, access and equity in The Orange Door services and policies
FSV continues to make enhancements to the CRM system to enable improved reporting. Work is underway to produce clearer reporting guidance to improve the completeness and consistency of data collected through the CRM.
Connections with the broader service sector
While this initial evaluation focused on partnerships within The Orange Door, it identified that further work needed to be done on the connections between The Orange Door, clients and the broader service sector. It also recognised that the establishment of these connections would remain a long-term activity and that FSV was taking steps to facilitate this through the development of roles such as the Service System Navigators.
Connection and networks with the broader service sector could be strengthened.
As part of the establishment of these connections, FSV is taking a phased approach to developing interfaces with other services at a statewide level, with some already in place, such as those with Victoria Police, Child Protection and Family Services, and some are currently in development, including legal services and housing and homelessness services.
The roles of Hub Managers and Service System Navigators have also been appointed in the next areas at an earlier point ahead of commencement to support the development of connections with existing area-based services, networks and groups, and referral pathways ahead of rollout.
In recognition that a strong service system is required to support the successful implementation of The Orange Door, Service System Networks are also being established in operational areas. These provide a mechanism for local services to engage with The Orange Door to support service delivery tailored to local needs and will help to support shared responsibility for interfaces.
Perception that The Orange Door is not sufficiently focused on child wellbeing and development.
FSV is working with stakeholders to address perceptions that The Orange Door is disproportionately focused on family violence services, including through communications, cultural change and practice development.
The implementation of The Orange Door is supported by the ‘Roadmap for Reform: Strong Families,
Safe Children’, the strategy for the reform of children, youth and family services in Victoria. Messaging about The Orange Door is regularly updated to reflect these reforms as they continue to be rolled out.
Build on existing systems and processes such as those currently in place in Child FIRST alliances during the establishment phase of new areas.
FSV is committed to building on the significant work Child FIRST/Family Services Alliances and other networks have already achieved to support good service linkages, appropriate information sharing and timely referrals. Implementation managers for new areas of The Orange Door are engaging early with Family Services and Child FIRST Alliances, Family Violence Regional Integration Committees and Dhelk Dja Action Groups in relation to:
- access and understanding existing arrangements
- workforce readiness to inform The Orange Door establishment
The early recruitment of Hub Managers and Service System Navigators in new areas will further support opportunities for strengthening the networks within areas as they transition to The Orange Door.
Determine if any clarification or awareness raising of The Orange Door is required within the community.
FSV pursued a strategy of not undertaking widespread public promotion of The Orange Door during the implementation phase to ensure partners could embed new processes and arrangements while managing demand in a gradual and sustainable manner. There were also considerations to ensure The Orange Door brand was visible but discrete for the safety of clients. Communications targeted key stakeholder groups and existing referrers to support transition.
The Orange Door website – www.orangedoor.vic.gov.au – was introduced to make it easier to find engaging, clear and easy-to-understand information about the service. It provides information about where and how to seek help, including for people who face particular barriers to getting help.
Development of the website was supported by rigorous user research and informed the language, tone, content and accessibility of the website.
Demand management
The evaluation found that a combination of high demand, lower than expected staffing (i.e. due to recruitment difficulties), high administrative burden of a new IT system and a lack of clarity about how to operationalise some processes initially resulted in significant wait times for some clients.
Other qualitative data from practitioners indicated there are variations between family violence services and family services in service provision and this was impacting on demand management. However due to the limited availability of data in CRM the evaluation was unable to verify this variation.
Addressing constraints that are impacting on demand and client wait times.
FSV is developing a demand management framework to consolidate and improve current practices across The Orange Door, family violence services (including specialist family violence services and perpetrator services) and family services. It will provide guidance on basic principles underpinning equitable access to services and will identify strategies to manage demand, supporting client pathways from initial contact to exit from the service system. It will encourage shared responsibility, open and transparent decision making and meaningful demand management collaboration across service delivery agencies and sectors.
Workflow processes are being refined through practice, data and system capability improvements. System enhancements and regular upgrades to CRM have, and will continue to, reduce administrative burden for practitioners and impact on demand management.
In areas that have experienced significant demand, specific strategies have been implemented to reduce waiting time, including use of dedicated teams. This has had a significant impact in reducing the number of cases awaiting assignment to practitioners in The Orange Door.
Provide guidance to practitioners on appropriate time-length for cases to better manage demand and projections for new areas.
There is an opportunity to consider variations for different client cohorts at The Orange Door and the demand management framework will support this. FSV is currently considering the most appropriate mechanisms for monitoring service intensity and duration within The Orange Door – such as analysing hours of client-related service delivery. This data will provide detailed information about the amount of time taken to support different client cohorts and may provide a foundation for guidance to practitioners. However, this must be balanced with qualitative data about client experience, which may indicate client preferences to continue engaging with The Orange Door rather than transition to other services.
To better plan for initial workflow, FSV to provide expected mix of demand sources prior to commencement.
FSV used existing data to inform initial planning however, given the significant change in the system introduced by The Orange Door, the ability to accurately apply this was limited. FSV will explore providing the OLGs in new areas with access to historical information on service demand from sources such as L17s, third-party referrals and direct contact, as well as support planning and implementation activities. Although, this information should be treated with caution as trends in service access are likely to change with the introduction of The Orange Door in areas.
Risk management and information sharing.
While the evaluation recognised that The Orange Door practitioners have access to a range of tools to assess and manage risk, there was still a need for better alignment of the MARAM framework with The Orange Door and broader services to improve consistency and the integration of risk assessment and management. The evaluation recommended that future evaluations look at the adequacy of risk assessment and risk management for clients over time. It also noted that a standardised assessment tool for child wellbeing was not yet available across The Orange Door areas.
Risk and needs assessments are undertaken inconsistently within The Orange Door.
The continued implementation of the MARAM and practice development activities are expected to improve the consistency of risk assessment and risk management within The Orange Door and beyond.
FSV is undertaking a range of work to support The Orange Door to further integrate and align their policies, procedures practice guidance and tools to the MARAM framework. Guidance to further clarify the expectations around completion of assessments has been provided and will be further supported through the release and ongoing implementation of the MARAM Framework practice guides.
A MARAM alignment strategy specific to The Orange Door is currently being developed. The strategy will support the embedding of MARAM in The Orange Door – across policies, procedures, practice guidance, tools and training.
To support child wellbeing practice, DHHS continues work to redevelop the Best Interests Case Practice Model to better align with the child and family and family violence reforms and consider the need for additional resources and tools to support the broader workforce in their work with children and families.
Ongoing evaluation and review processes across the broader service sector.
FSV will continue to review implementation of the MARAM and the Family Violence Information Sharing Scheme (FVIS Scheme) across all prescribed services, including The Orange Door through the MARAM and FVIS Scheme contracted evaluations. FSV will also continue to review The Orange Door change strategy to ensure that MARAM alignment and information sharing is effectively implemented.
IT Systems
There were functionality challenges associated with the establishment of key systems used at The Orange Door, and the first iteration of the CRM was found to be difficult to operate, burdensome and impacted negatively on service delivery. The evaluation recognised that the first iteration of any new IT system often poses challenges for users and that practitioners had reported that ongoing upgrades were already improving functionality. However, it also noted that further adaptations were required, specifically for data collection, monitoring and reviewing.
While the Central Information Point (CIP) also had minor initial establishment challenges, the general feedback was related to the positive contribution the system was making to information sharing for risk assessment and management at The Orange Door.
The CRM system could benefit from improvements to useability and functionality,
Updates are released approximately quarterly and are informed by practitioner feedback. The changes implemented through CRM Release 6, which commenced on July 2019, included:
The CRM system could benefit from improvements to useability and functionality,
Updates/improvements to the CRM are released approximately quarterly and are informed by practitioner feedback. The following changes have been implemented through CRM Release 6, which commenced on July 2019, and Release 7 which commenced in November 2019:
Release 6
- enhanced reporting capabilities,
- updates to minimise data entry and improve access to information, and
- updates to CIP requests and reports to improve information sharing between The Orange Door and CIP Operations. Release 7
- introduction of notifications being sent to practitioners (CRM Dashboard and email) including:
- a TRAM assessment is due to expire in 3 days
- a new screening record has been created for an existing case - delivery of a CIP report,
- greater integration with other systems to minimise duplication of data entry
- enhancements to the layout of fields to improve the usability of the system
- ability to share activities, risk assessments and close cases across multiple cases within a case group • spell check functionality
- improvements to the CIP request form
Comprehensive training, support and guidance materials are provided to The Orange Door workforce to support their use of the CRM system at induction and following the deployment of incremental system changes. These supports have been welcomed by practitioners.
[3] While the first evaluation was unable to draw conclusions on client experience and outcomes due to limited data during the early stages of implementation, it is expected that these areas will be addressed in future evaluations.
[4] The Orange Door workforce strategy is designed to complement and align with broader workforce development strategies.
Updated