Strengths:
This research began with the question: What are the most promising approaches to evaluate community change impacts of comprehensive community initiatives (CCIs) in promoting Aboriginal child and family well-being and how might Indigenous ways of knowing and approaches to research assist in such evaluation?
In focusing on knowledge developed over the past ten years that helps to answer these this question, a number of strengths are noted. The first two, which we have labelled Truth and Reconciliation and Collective Impact, were not part of this knowledge synthesis, but do represent knowledge developed within the past ten years that has been foundational to our research agenda and the knowledge developed through this synthesis.
Truth and Reconciliation: A process of truth has begun (as opposed to secrecy and denial), first with the 1996 Report of the Royal Commission on Aboriginal Peoples, followed by the various reports of the Truth and Reconciliation Commission of Canadian that not only acknowledges the traumatic legacy of colonial practices for Aboriginal health, education, employment, justice, child welfare, and more, but begins to illustrate how complex and inextricably intertwined these issues are. Without changes on multiple fronts, we are unlikely to see significant or lasting changes on any.
Collective Impact: The concept of “collective impact” has drawn our attention to the fact that complex social issues (including addressing Aboriginal well-being) cannot be achieved by any single community organization or government department working in isolation. “No single organization is responsible for any major social problem, nor can any single organization cure it” (Kania & Kramer, 2011, pp. 38-39). Required instead are comprehensive community initiatives; the ‘joined-up’ efforts of local representatives of senior and municipal government departments to enable the boundary spanning cooperation, coordination, and communication that appear key to successful social change.
Approaches to Evaluation of CCIs: Literature identifying the specific challenges of evaluating community change initiatives and the even greater challenges of evaluating multi-sector or comprehensive community change initiatives predates this synthesis but continues to be present. What our review uncovered is the work being done to meet these challenges. Numerous methodological approaches are developing accompanied by principles and case studies that showcase their application to specific community or multilevel change initiatives. Alongside of this, methods and tools for data collection and analysis are also being developed.
Our review identified 3 primary methodological approaches: developmental/adaptive aimed primarily at examining the impacts of community change initiatives within specific contexts in order to facilitate long-term, place-based learning, adaptation, and growth; Generalizable aimed at promoting internal and external validity to enhance learning and inform broader policy development by developing greater generalizability and translatability of interventions across contexts; and participatory aimed at involving various stakeholders in these efforts whether to build evaluative capacity or to empower marginalized individuals and groups. The debates over these different aims are significant and – in our opinion – are testimony to the strength of the developing literature in this field. In particular, there are efforts to bridge the divides and develop approaches to evaluation of CCIs that promote both contextual learning and generalizability and translatability across contexts.
Indigenous Ways of Knowing and Approaches to Research: There is a significant and coherent literature concerning Indigenous ways of knowing and approaches to research and many Indigenous researchers highlight the use of these in specific contexts. Much of the literature is coming from Indigenous scholars in Canada who are internationally recognized for their contributions. The Sources tab identifies the formal sources we drew on in articulating Indigenous ways of knowing and approaches to research. It should also be noted that the literature consistently highlights the fluid and organic nature of Indigenous research methods that develop through the relationship between researcher and researched, and with attention to the values, traditions, practices, and aspirations of the community as well as the guidance of local Elders.
In our review of the evaluation literature, we see the possibility for some areas of congruence between Indigenous approaches to research and certain aspects of developmental/adaptive approaches to evaluation. Both emphasize the participation of community members in identifying a vision for the community and in identifying community conditions or dynamics that can be addressed to enable the vision. And both emphasize a fluid process of learning and adaptation. Developmental/adaptive approaches offer considerable flexibility that with guidance from Elders, could incorporate methods congruent with local protocols, traditions, and ceremonies. However, there are also some clear tensions between some developmental/adaptive approaches and Indigenous ways of knowing and approaches to research. One obvious example is the use of timelines within developmental/adaptive approaches and the expectation that timelines of impact can and should be developed by participants in CCIs; such an expectation is significantly at odds with Indigenous worldviews.
Attempts to enhance the scientific rigor and thus generalizability/translatability of evaluations of CCIs may be much more problematic as such approaches appear to require more standardized protocols for implementation of ‘interventions,’ collection of ‘data,’ and measures of ‘indicators;’ the language and processes imply a lack of respect for local knowledge, values aspirations, protocols, and processes.
Weaknesses and Gaps
Approaches to Evaluation of CCIs: At this point, the literature on evaluation of community change initiatives is minimal; our synthesis identified only 34 documents that met our criteria for inclusion; that number would have been much smaller if we had confined our search to methodological studies. And if we had required studies that met standards of scientific rigor, we would have synthesized only a handful of documents. Instead, maintaining a broader scope allowed us to examine a variety of emerging ideas and methodologies. It is hoped that a much more robust literature will begin to develop that engages more fully with and across the debates between these methodologies, particularly debates of contextual sensitivity and enabling of development and adaptation versus the primacy of generalizable and translatable evaluation findings.
There is a particular need for more literature that focuses on Canadian contexts and that examines the role and perspectives of federal and provincial policy-makers. Only five Canadian studies were identified in this review and only two of these incorporated the role and perspectives of the federal government in evaluation of community change initiatives. Moreover, none of them examined the role or perspectives of Canada’s provincial governments and government departments, despite the significant role of provincial governments in policy and funding. There were also no studies incorporating the role and perspectives of regional and municipal government representatives; while these levels of government lack the power and policy-making punch of federal or provincial governments, they are nonetheless highly significant for CCIs given their contextual focus.
While we had hoped for at least some literature regarding the role of Indigenous ways of knowing and approaches to research in evaluation of CCIs aimed at enhancing Aboriginal child and family well-being, we had anticipated that this would be a significant gap. We had planned to supplement the literature review through 2-3 focus groups with participants in CCIs aimed at enhancing Aboriginal well-being. However as the research only began in January, the review of relevant literature was not completed until the end of May and the CCIs targeted for focus groups do not meet over the summer months. Nonetheless, meetings have occurred with several of these initiatives and with Chairs and Coordinators of all of the initiatives. All have shared that no formal evaluation of these initiatives is occurring and all groups have expressed interest in presentation of the research findings and further discussion of evaluation strategies within their initiatives.
