Appendix C: List of tools and resources

Guidance on place-based approaches

• Framework for Place-Based Approaches
  The Victorian Government’s Framework for Place-Based Approaches describes a way of thinking about place that will better enable VPS staff to effectively communicate across government.
• Place-Based Guide
  This guide provides a contemporary, evidence-informed and practical collection of ideas, advice, case studies, tools and resources to support the effective design, implementation and evaluation of place-based approaches.
• Funding toolkit
  This toolkit brings together existing and new tools and resources that VPS staff and managers can pick up and use when designing new, or managing existing, funding agreements with place-based approaches
• What works for place-based approaches in Victoria research report
  Prepared by researchers from Jesuit Social Services (JSS) Centre for Just Places, RMIT University and the Centre for Community Child Health, Murdoch Children's Research Institute (MCRI), this paper identifies the principles, enabling conditions and barriers for the success of place-based approaches. It is complemented with practice learnings from case studies of Victorian place-based initiatives.
• Boundary spanning to improve community outcomes
  This report provides guidance to VPS staff working in collaborative initiatives across government and with community stakeholders and place-based initiatives. The report presents a framework which describes the key supporting conditions required for effective collaboration.

---

Guidance on MEL for place-based approaches

• Framework and Toolkit for evaluating place-based delivery approaches (2019)
  Developed by Clear Horizon for the Federal Department of Social Services, the framework and toolkit developed to allow governments, communities, evaluators and other organisations to build evidence regarding the appropriateness, effectiveness, efficiency, impact and sustainability of place-based approaches.
• Principles for evaluating systems change
  A paper prepared by Mark Cabaj and Tamarack Institute outlines 15 principles to guide the evaluation of systems change efforts. These principles span across the various stages of evaluation: framing, designing, capturing outcomes and learning and accountability.
• Evaluating systems change results: An Inquiry Framework (PDF, 862 KB)
  The framework can used as guidance when developing a high-level theory of change for capturing population or system level change. It provides guidance on the questions that can be asked when assessing systems change efforts and can help define what constitutes a ‘result’ when developing a high level theory of change.
• The Top 10 Questions: A Guide to Evaluating Place-Based Initiatives
  A short, ten-step guide outlining some key considerations for evaluating place-based approaches.

---

General resources to assist with scoping and planning MEL

• Rainbow Framework
  Developed by Better Evaluation, this framework lists the standard criteria to consider when developing an evaluation plan.

---

Victorian Government specific resources to assist with scoping and planning MEL

• (Former) Victorian Department of Health and Human Services Evaluation Guide (PDF, 276 KB)
  Developed by the former Victorian Department of Health and Human Services, provides guidance to support staff planning and commissioning an evaluation.
• Crime Prevention Evaluation Toolkit
  Developed by the Victorian Department of Justice and Community Safety, this toolkit offers resources that can be used across different stages of the evaluation process such as planning, managing, measuring changes etc.

---

Defining the objectives of MEL

• Evaluability Assessment
  Developed by the Juvenile Justice Evaluation Centre, this guide aims to assist with implementing evaluability assessment to ensure that initiatives are ready for evaluation. Although it is written for programmatic contexts and does not consider a developmental approach looking to conduct a program evaluation, the concepts and ideas may help VPS staff to identify considerations for whether certain evaluative approaches are suited to the context of the place-based approach.
• Tool - Developing evaluations that are used (PDF, 354 KB)
  Short document developed by Tamarack Institute, this short paper outlines key considerations to ensure the useability of evaluation findings.

---

Guidance on developmental evaluation

• Developmental Evaluation Primer (PDF, 1,037 KB)
  A primer developed by Jamie Gamble for the J.W. McConnell Family Foundation. It provides an introduction to developmental evaluation and how to implement it.
• Developmental Evaluation
  Better Evaluation introduces developmental evaluation and outlines key differences between traditional and developmental evaluation.

---

Identifying role(s) of government

• The Shared Power Principle (PDF, 922 KB)
  The Centre for Public Impact provides guidance to governments on how power can be shared with communities by identifying four patterns of power sharing used by governments around the world to create positive outcomes for communities.

---

Developing a theory of change or outcomes logic

• Guidance on developing a theory of practice
  Better Evaluation provides an overview of the different ways that you can present a theory of change.
• The Water of Systems Change
  The seminal work by FSG outlining six conditions needed to advance systems change.
• ‘Thinking Big: How to use Theory of Change for Systems Change’
  Report by New Philanthropy Capital and Lankelly Chase examining usefulness of theory of change in supporting systems change (2018).
• Methodological Brief – Theory of Change
  A briefing note commissioned by UNICEF and written by Patricia Rogers summarising the role of theory of change in evaluation, and challenges and considerations for developing quality theories (2014).

---

Resourcing

• Levels of resourcing table (pp 30-32), National Place-Based Evaluation Framework (PDF, 3.7 MB)
  Clear Horizon provides a table with different levels of resourcing based on scope and purpose of evaluation

---

Engaging with stakeholders

• Spectrum of community led approaches (PDF, 324 KB)
  Tamarack Institute provides a list of community-led approaches that can be used for evaluation purposes. This tool can help determine what level of community leadership is most appropriate, and what kind of engagement approaches are needed.
• AES First Nations Cultural Safety Framework
  Developed by the Australian Evaluation Society, the framework outlines key principles that support culturally safe evaluations, and provides guidance on the roles and responsibilities of different stakeholders involved in evaluation
• Section 3 on ‘Evaluating with Community’ (pp.11- 17), National Framework for Place-Based evaluations (PDF, 3.7 MB)
  The framework provides key considerations for evaluations involving diverse cohorts and ethical considerations that are relevant across MEL activities.
• Practical strategies for culturally competent evaluation (PDF, 706 KB)
  Developed by Centres for Diseases Control and Prevention in the U.S.A, the guide highlights the prominent role of culture in evaluation. It is designed for program staff and evaluators and provides important strategies for approaching an evaluation with a critical cultural lens to ensure that evaluation efforts have cultural relevance and generate meaningful findings that stakeholders.
• Came et al. (2019), Māori and Pasifika leaders’ experiences of government health advisory groups in New Zealand, Kōtuitui: New Zealand, Journal of Social Sciences Online, 14:1, 126-135
  This journal article is based is based on a qualitative study that explores the experiences of six Māori and Pasifika leaders on health policy-making advisory committees. It points to the need for deeper engagement and more genuine recognition of the knowledge that First Nations leaders and communities have for addressing inequities in their communities.

---

Embedding a strategic learning culture and structures into government

• Building a positive evaluation culture: Key considerations for managers in the families and children services sector
  This resource provides tips and considerations for managers who wish to build an organisational culture where evidence and evaluation are valued.
• Most Significant Learning (MSL)
  MSL involves collecting stories to help surface learnings, trigger deep reflection and capture developmental moments in the innovation journey.
• Reflection Workshop/Evaluation Parties
  A structured large-group workshop that facilitates collaborative sense-making and systemic critical thinking to give meaning to data and emerging knowledge to be able to act on it. Participants can interrogate the evidence and assess performance during the workshop.
• Yarning Circle
  The Yarning Circle provides a safe space for stakeholders involved in MEL to be heard and to respond. It involves deep listening and encourages respectful and honest interactions between stakeholders, sharing of knowledge and can be used to foster shared accountability.

---

Additional tools and methods for monitoring and evaluation

• Overview of value and role of rubrics in evaluation and a short paper on the usefulness of rubrics in evaluation
  Rubrics can be used in evaluation as an alternative way to establish performance and collectively assess progress against these standards. Rubrics allow all stakeholders to see the evaluation criteria (which can often be complex and subjective).
• Addressing attribution through contribution analysis
  A paper by John Mayne and published in the Canadian Journal of Program Evaluation, describes the steps necessary to produce a credible contribution story.
• Most Significant Change approach (MSC)
  Most Significant Change approach involves generating and analysing personal accounts of change and deciding on the significance of these stories.

---

MEL within First Nations contexts

• The Victorian Government Self-Determination Reform Framework
  The framework is intended to guide public service action to enable self-determination in line with government’s commitments in the Victorian Aboriginal Affairs Framework (2018-2023)
• AES First Nations Cultural Safety Framework (PDF, 4.6 MB)
  Developed by the Australian Evaluation Society, the framework outlines key principles, and guidance on the roles and responsibilities of different stakeholders involved in evaluation.
• OCCAAARRS Framework
  Developed by Skye Trudgett, OCCAAARS is a conceptual framework for researching, evaluating and designing Aboriginal programs, initiatives and organisations with Aboriginal data sovereignty and governance as the guiding intent.
• Ngaa-bi-nya (Yulang Indigenous Evaluation)
  Ngaa-bi-nya is a framework that offers a practical guide for the evaluation of Aboriginal and Torres Strait Islander health, social and justice programs and may be a helpful scoping tool.
• First Nations Ripple Mapping
  This tool is for any First Nations collective working toward place-based systems change. The tools seeks to support collectives in identifying progress toward agency and self-determination in collective efforts of change making.
• How can evaluation better recognise Indigenous self-determination?
  This article by Social Ventures Australia, explores the role of data and evidence of what works in delivering improved outcomes for an with First Nations peoples. It is based on the perspectives of First Nations leaders, evaluators and VPS staff on how evaluation practices need to change so that they are consistent with the right of First Nations self-determination.
• Vaughan et al. (2017), 'Hey, we are the best ones at dealing with our own': embedding a culturally competent program for Maori and Pacific Island children into a mainstream health service in Queensland, Australia.
  The report presents the results of one component of an external evaluation of Good Start Program (GSP), a community-based program for the prevention of chronic disease among Maori and Pacific Island (MPI) communities living in the state of Queensland, Australia. An evaluation of the GSP was undertaken using a mixed methods approach. This paper reports on the qualitative component which used Talanoa, a culturally tailored research methodology.
• Chilisa, B (2021) Indigenous Made in Africa Evaluation Frameworks: Addressing Epistemic Violence and Contributing to Social Transformation
  This journal article explores approaches and methods used to evaluate international initiatives that seek to achieve social transformation in the developing world. The article focuses on two approaches; Indigenous paradigmatic framework and the Made in Africa approach to evaluation.