Executive Summary
Our knowledge synthesis responds to the following question: What are the most promising approaches to evaluate community change impacts of comprehensive community initiatives (CCIs) in promoting Aboriginal child and family well-being and how might Indigenous ways of knowing and approaches to research assist in such evaluation?
This question is the result of a recognition that Aboriginal child and family well-being are dependent on a host of highly interconnected social and structural factors that can only be address through comprehensive community initiatives (CCIs); approaches that bring together and coordinate the efforts of multiple community sectors to achieve the social and structural change that is beyond the capacity of any one of them working in isolation. It is also in recognition that evaluation approaches to such initiatives are being developed across a range of fields and there has been no systematic review or synthesis of the literature to date, and more importantly, there has been no examination of the contribution of Indigenous research paradigms to evaluation of CCIs. If the aspirations of Canada’s Aboriginal peoples to foster sustainable,, vibrant communities is to be met, then surely strategies to evaluate such efforts must incorporate culturally relevant approaches that resonate in those communities?
Methods
This knowledge synthesis was separated into two parts: first, a systematic review and synthesis of literature from 2006 – 2017 related to evaluation of community change initiatives and second, a synthesis of Indigenous ways of knowing and research methods. These two parts were brought together at the conclusion of the synthesis.
The first part of the review began by generating and utilizing a list of search terms; in conducting the review a decision was made to include both peer-reviewed and grey literature in order to include insights from agencies or organizations with ground-level expertise whose work may not appear in strict academic searches. After screening all relevant hits and discarding duplicates, those that did not offer full-text, and those that did not have the required foci, a total of 34 documents were selected for full review including 19 peer reviewed articles, one full-text book chapter and 14 grey literature documents.
Results
Thematic mapping identified the five following themes or major categories within the literature: challenges of evaluation, principles for evaluation, methodologies for evaluation, case studies of evaluation, and methods and tools for evaluation. The challenges of evaluating the impacts of CCIs are enormous particularly within the current focus on evidence based policy to enhance accountability and enable decision making. The literature discussed at length the challenges of associating specific outcomes to interventions that are highly contextualized. In terms of principles there was some indication of the turmoil in this field; three areas of principles where there were clear differences in approaches advocated or showcased were purpose and time frame of evaluation, the need for contextual sensitivity, and the issue of stakeholder participation. The nature of these differences resonates throughout the review and synthesis of part 1 so these issues will be discussed further below.
With respect to methodologies, debates were again present and related to debates around principles. Three main approaches to evaluation were identified, though it should be noted that these are not discrete approaches; there is a certain amount of methodological influence and overlap among them. The three approaches are Developmental/Adaptive approaches, Generalizable approaches, and Participatory approaches. Developmental/adaptive approaches have as a primary goal, the facilitation of internal learning regarding the creation of community level change to enhance well-being. Within developmental/adaptive approaches ‘Theory of Change’ (TOC) methodologies are dominant. TOC consists of a five or six stage process of ‘backward mapping’ that begins with the long-term change(s) or vision that is the goal of the initiative. Once these are determined, pre-conditions or requirements necessary to their achievement are identified along with relevant contextual knowledge or assumptions. Interventions to create the required conditions are defined along with indicators to assess their achievement. In many cases a narrative is developed to explain the theory of change; the narrative explains why the pre-conditions are necessary to achieve the goal as well as how particular interventions will help in the achievement of the pre-conditions and why the indicators are relevant measures. The result is a ‘pathway’ or theory of change that identifies assumptions, illustrates what is to be done as well as why, outlines a timeframe for each step and specifies indicators to be measured. It is suggested that the final product acts as a tool for evaluation that facilitates internal validity in evaluation without sacrificing contextual sensitivity as the theory can be revisited and changed or adapted based on findings and changes in contextual aspects.
Generalizable approaches emphasize the need for scientific rigor in evaluation of CCIs to enhance the establishment of an evidence base for translation across contexts and significant research is being done in the field of evaluation, behavioral medicine and population health to name a few, to develop approaches to the evaluation of community change efforts that can claim scientific rigor. Stepped Wedge Cluster Random Control Trials, longitudinal studies with standardized continuous measurement, interrupted time-series designs, multiple baseline designs, cross-case methodology, cost-benefit analysis, a range of neighborhood data, from cluster detection to socio-economic indicators, asset mapping, spatial patterning and others, along with mixed methods and approaches to measure social levels of social capital.
Participatory approaches range from stakeholder participation in evaluation design, data collection and analysis as well as dissemination and decision making. In developmental/ adaptive approaches participation is essential to facilitate the long term learning and growth central to these approaches. There were however, significant differences in the ways in which participation was conceptualized with ‘stakeholder participation’ taking on many meanings.
Eighteen case studies showcased the variety of evaluation methodologies, principles, and challenges along with a variety of methods and tools for data collection and analysis. While the majority were case studies of American initiatives, there were a number of Canadian initiatives and one case study of an Australian initiative. The case studies highlight the debates over principles and methodologies but they also showcase the research that is being done and the progress that is being made in terms of development of the various approaches. A literature that engages more robustly with the various debates would be helpful along with literature that focuses explicitly on Canadian contexts and the inclusion of various levels of government.
The second part of the review identified a strong and coherent literature regarding Indigenous ways of knowing and approaches to research. However there were no case studies that applied this knowledge to Aboriginal CCIs. Only two of the eighteen case studies were related to Aboriginal CCIs –one in an off-reserve American Indian community and one an urban Aboriginal initiative in Canada. It is our contention that developmental/adaptive approaches to evaluation of CCIs with their focus on systems understanding, ongoing change and contextual sensitivity are most congruent with Indigenous ways of knowing and research approaches with their emphasis on interconnectedness, circularity, and balance and harmony. Moreover developmental/adaptive approaches suggest considerable flexibility for methodologies grounded in local protocols and practices and guided by Elders. Principles of respect reciprocity, responsibility, and relationship may also be a good fit with these Indigenous approaches due to their emphasis on local input into and control over evaluation design. However, this review also identified areas of tension and dissonance between developmental/adaptive approaches and Indigenous ways of knowing and approaches to research, i.e. the expectation that evaluation design should include identification of specific timelines for community level changes.
Ultimately there is a need for more research that engages CCIs focused on Aboriginal well-being in examining evaluation approaches and determining what is of most use.
Knowledge Mobilization
Knowledge mobilization will encompass social media and University websites, institutional and community presentations including with members of Aboriginal community change initiatives, social work educators and indigenous educators, the Urban Aboriginal Knowledge Network, the Prairie Child Welfare Consortium biennial conference. Presentations will also occur to funders, policy-makers, and advocates in child welfare and Aboriginal governance. We aim for 1-2 scholarly publications in national and international open access journals. But we contend that the most important knowledge mobilization will occur through the development of research that will bring together multi-sector community change initiatives aimed at enhancing Aboriginal Child and Family well-being to develop strategies for evaluation that honor Indigenous ways of knowing and approaches to research. While the priority should be on evaluation to enable learning and development through culturally respectful protocols and practices, the goal should also be to involve funders and relevant policy-makers in reflecting on their relevance to evidence-based policy development.
