This knowledge synthesis was separated into two parts: first, a synthesis of literature from 2006 – 2017 related to evaluation of community change initiatives and second, a synthesis of literature related to Indigenous ways of knowing and research methods. The remainder of this report discusses the process of review and synthesis within each of these two parts and then presents findings from these two separate literature reviews. Following these results we examine the strength of the literature and identify gaps in knowledge. We also note key areas of tension and debate and discuss the implications of these. The final components of the report discuss knowledge mobilization activities that will be occurring over the next six months as well as ways that we will be addressing gaps and tensions in knowledge of evaluation methods and the role of Indigenous ways of knowing and approaches to research.
Part 1: Evaluation of Community Change Initiatives
The first part of our review began by generating a list of search terms that was developed by two members of the research team in conjunction with a librarian who provided training in systematic literature review strategies. In conducting this first part of the literature review a decision was made to include both peer-reviewed and grey literature. The primary reason for this was to gain insight from agencies or organizations that might be conducting evaluative work in the area of comprehensive community initiatives. Their work offers ground-level expertise but often does not appear within academic databases.
Searches of academic databases utilized key words and combinations of key words while the strategy used for search engines such as Google scholar was controlled through a narrowing technique that utilized full phrases as opposed to key words. For example, during the first Google Scholar search, the actual research question was used and yielded eleven results. As with the academic databases a snowball approach was also utilized; material found through the Google Scholar searches was examined for additional relevant material for review.
The discussion below highlights the initial to final selection of literature used in the synthesis followed by a Bibliography of sources used in the synthesis.
Stage 1: Identification of relevant literature
Forty-seven articles were chosen from academic databases after duplicates or articles that were deemed to be outside the research purview were discarded. Thirty-two documents were chosen from grey literature databases and search engines after duplicates and literature deemed to be outside the research purview were discarded. Thus a total of 79 documents were selected for initial review.
Stage 2: Initial Review
Upon review of abstracts and/or methods sections of the initial 79 documents, forty-one were discarded as they were found to lack a research focus, to lack an evaluative focus, or lacked any focus on evaluation of community level changes/impacts/outcomes. Thus a total of thirty-eight documents were selected for final review and coding.
Stage 3: Final Review and Coding
The 38 articles were then subjected to a final review and coding for themes by the PI and the two research assistants who conducted the searches. During this coding process an additional 3 documents were discarded as they were found to not meet inclusion criteria while 1 was found to be a duplicate of another article. Consequently, 34 documents from Part One of the literature review were included in this knowledge synthesis.
Findings
Thematic mapping identified the five following themes or major categories within the literature: challenges of evaluation, principles for evaluation, methodologies for evaluation, case studies of evaluation, and methods and tools for evaluation. The challenges of evaluating the impacts of CCIs are enormous particularly within the current focus on evidence based policy to enhance accountability and enable decision making. The literature discussed at length the challenges of associating specific outcomes to interventions that are highly contextualized. In terms of principles there was some indication of the turmoil in this field; three areas of principles where there were clear differences in approaches advocated or showcased were purpose and time frame of evaluation, the need for contextual sensitivity, and the issue of stakeholder participation. The nature of these differences resonates throughout the review and synthesis of part 1 so these issues will be discussed further below.
With respect to methodologies, debates were again present and related to debates around principles. Three main approaches to evaluation were identified, though it should be noted that these are not discrete approaches; there is a certain amount of methodological influence and overlap among them. The three approaches are Developmental/Adaptive approaches, Generalizable approaches, and Participatory approaches. Developmental/adaptive approaches have as a primary goal, the facilitation of internal learning regarding the creation of community level change to enhance well-being. Within developmental/adaptive approaches ‘Theory of Change’ (TOC) methodologies are dominant. TOC consists of a five or six stage process of ‘backward mapping’ that begins with the long-term change(s) or vision that is the goal of the initiative. Once these are determined, pre-conditions or requirements necessary to their achievement are identified along with relevant contextual knowledge or assumptions. Interventions to create the required conditions are defined along with indicators to assess their achievement. In many cases a narrative is developed to explain the theory of change; the narrative explains why the pre-conditions are necessary to achieve the goal as well as how particular interventions will help in the achievement of the pre-conditions and why the indicators are relevant measures. The result is a ‘pathway’ or theory of change that identifies assumptions, illustrates what is to be done as well as why, outlines a timeframe for each step and specifies indicators to be measured. It is suggested that the final product acts as a tool for evaluation that facilitates internal validity in evaluation without sacrificing contextual sensitivity as the theory can be revisited and changed or adapted based on findings and changes in contextual aspects.
Generalizable approaches emphasize the need for scientific rigor in evaluation of CCIs to enhance the establishment of an evidence base for translation across contexts and significant research is being done in the field of evaluation, behavioral medicine and population health to name a few, to develop approaches to the evaluation of community change efforts that can claim scientific rigor. Stepped Wedge Cluster Random Control Trials, longitudinal studies with standardized continuous measurement, interrupted time-series designs, multiple baseline designs, cross-case methodology, cost-benefit analysis, a range of neighborhood data, from cluster detection to socio-economic indicators, asset mapping, spatial patterning and others, along with mixed methods and approaches to measure social levels of social capital.
Participatory approaches range from stakeholder participation in evaluation design, data collection and analysis as well as dissemination and decision making. In developmental/ adaptive approaches participation is essential to facilitate the long term learning and growth central to these approaches. There were however, significant differences in the ways in which participation was conceptualized with ‘stakeholder participation’ taking on many meanings.
Eighteen case studies showcased the variety of evaluation methodologies, principles, and challenges along with a variety of methods and tools for data collection and analysis. While the majority were case studies of American initiatives, there were a number of Canadian initiatives and one case study of an Australian initiative. The case studies highlight the debates over principles and methodologies but they also showcase the research that is being done and the progress that is being made in terms of development of the various approaches. A literature that engages more robustly with the various debates would be helpful along with literature that focuses explicitly on Canadian contexts and the inclusion of various levels of government.
Part 2: Indigenous Ways of Knowing and Approaches to Research in Evaluation of Community Change Initiatives
The second part of this knowledge synthesis examined the congruence between evaluation methods and Indigenous ways of knowing and approaches to research. In doing this, our goal was not to conduct a systematic review of the literature on Indigenous ways of knowing or Indigenous approaches to research. Instead, we approached this aspect of the knowledge synthesis differently, talking to one another about the ideas of Indigenous scholars, and the learning that had come to us through our relationships with Indigenous people. We used our sources (focusing particularly on the work of Canadian Indigenous scholars) and one another to guide us to new sources and continued to discuss these. Once we were no longer identifying new concepts or ideas, we began to put these ideas to paper, consistently checking with one another and with our sources. We have honored the perspective that Indigenous ways of knowing cannot be attributed to any particular individual, however we have also acknowledged the specific words, thoughts, ideas, or experiences with Indigenous ways of knowing and approaches to research that have been shared by particular individuals. (See the Bibliography of Sources in the References tab.)
Despite a diversity of cultures, practices, and locally grounded perspectives there are a number of shared worldviews that inform Indigenous epistemologies and research approaches and distinguish them from others (Wilson, 2008). Yet it is important to note that Indigenous people are connected to these worldviews, cultures, and traditions in different ways and to different degrees. With respect to Canada’s Indigenous peoples, there were – and still are – concerted efforts to destroy adherence to Indigenous worldviews, cultural traditions and practices. Confinement on reserves, outlawing of cultural practices, and destruction of sacred sites and cultural artifacts are only some of the ways this occurred. Perhaps the most widespread was the development of residential schools and the enforced enrollment of Aboriginal children in these schools. In these settings not only were children required to adhere to Western worldviews, traditions and practices, but also to view their own with suspicion and derision. Children were punished for speaking their own language or engaging in traditional practices. The traumatic impacts of colonization and residential schooling on Aboriginal individuals, families, and communities cannot be overstated and developing an understanding of this trauma and turmoil is fundamental to understanding the journeys and knowledges of Canada’s Aboriginal peoples.
One of the most significant outcomes of colonial and neo-colonial practices concerns the loss of Indigenous languages. According to Norris (2011), only a minority of Indigenous people in Canada speak or understand the language of their ancestors. Yet language is an essential aspect of Indigenous ways of knowing; it is through language that worldviews are represented and transmitted, that traditions are taught, and identities and relationships are understood (Battiste, 2002). “When an Aboriginal language dies, a whole way of thinking is lost, both to the community and to humanity” (Castellano, Davis & Lahache, 2001, p. 26).
Our review of the literature identified the following core ontological perspectives or worldviews:
Interconnectedness
Fundamental to Indigenous ontologies is the notion of interconnectedness. Interconnectedness is part of a broader spirituality that includes the interconnectedness of humans to one another as well as to past and future generations, but also the interconnectedness of oneself with animals, plants, and all parts of the land and larger cosmos. This interconnectedness to both people and places is core to personal and collective identity, values and beliefs and thus relationship is a foundational component of an Indigenous worldview.
Circularity
Circularity is viewed as part of the natural order of creation and as such is also linked to spirituality; the cyclical nature of life is seen in the seasons of the year, the cycles of the moon, ocean tides, etc. With no beginning and no end, the circle signifies transformation and movement as well as continuity and interconnectedness. Circles also represent tension and flow between each individual and the whole. Communities are circles of individuals within a whole.
Balance and Harmony
Balance and harmony are closely related to one another and to each of the principles above. As everything is interconnected, related to a whole, each part has a role in the creation of balance and harmony for the whole. If one part is out of balance or harmony, it will struggle to fulfill its role and the whole will be out of balance and harmony. Similarly, if one relationship is out of balance and harmony, the balance and harmony of the whole is impacted.
The above have numerous epistemological implications. First, knowledge is relational. As everything is interconnected, knowledge only develops within these interconnections – indeed knowledge is these interconnections. Second, knowledge is personal. Congruent with the perspective that each of us is a unique part within the whole, is the belief that each of us has particular, specific knowledge such that there is no one ‘truth’ but rather many truths. Third, knowledge is developmental. Learning, or knowledge, like everything else, is cyclical and is a process rather than a product; over time, within relationship, and by paying attention to balance and harmony, and being guided by the spiritual, knowledge and understanding grow. Fourth, space is viewed as more important than time; the ‘here’ matters more than the ‘now’ and knowledge represents learning to living well in this place so as to contribute to the balance and harmony of the whole.
|
Judy’s Story: Indigenous Worldviews and Decision Making I was conducting semi-structured interviews with members of an Aboriginal Multi-sector Initiative and asking about decision making processes. In response to my questions several members shared the same story: They had, for a number of meetings, discussed engaging in a form of social action to draw attention to a particular issue impacting Aboriginal people living in a nearby community. As the discussion progressed it became clear to each of them that there were a variety of opinions over whether or not the action should take place. As a result the group did not proceed with the action. Over and over I was told that “the time was not right,” “it was not the right time,” “we decided to wait until the time was right.” Steeped in Western culture with its notions of ‘majority rule’ and ‘just do it,’ I found their words puzzling but I was aware that a different worldview was being shared with me. Over the years, I have come to appreciate that I was learning firsthand about how harmony within the group is paramount and how each individual’s opinions and perspectives make up the whole. I was also hearing and learning how actions are decided as a group and how time is viewed as circular. A few years later, the group revisited the decision and, with the support of all, the action occurred. The right time had come. |
Four interrelated terms are consistently present in the literature on Indigenous approaches to research: Respect, Reciprocity, Responsibility, and Relationship. The concepts denote the importance of participatory approaches that respect the knowledge that each individual has to offer to the research process but that also respect the whole community, its values, traditions, and beliefs as well as its ideas and aspirations and ensure that research is meaningful to these. And while each participant is viewed as equally important, there is particular respect for the knowledge and insight of Elders. Respect also requires honesty and transparency by all participants in the research process. Reciprocity requires a mutual exchange of benefits; those in the role of researcher both give to and receive from the community and its members. Benefits should not flow only one way. Responsibility encompasses accountability; researchers are accountable not just to the immediate community or participants, but are engaged and accountable to a much broader constituency. Wilson (2008) suggests that responsibility and accountability extend to the relationship the researcher has with the world around him or her; it requires researchers to be accountable to ‘all my relations’ – past, present, and future; land, plants, animals and human. Responsibility also extends to research participants and encompasses trust, honesty, and openness. Ultimately, research must honor and respect the interrelatedness of researcher and researched. Engaging in research with respect, reciprocity, and responsibility is the relationship.
Aboriginal approaches to research reflect the above perspectives and implications. As noted by Linda Tuhiwai Smith (2001), “Indigenous methodologies tend to approach cultural protocols, values and behaviours as an integral part of methodology” (p. 15). However, rather than relying on a prescribed set of methods, approaches are organic, guided by practices and protocols grounded in specific places and cultural traditions and practices. There are however, several approaches to research methods that are worth noting:
Talking Circles: The circle is a model used for group interaction in both the symbolic realm (to convey teachings and promote the development of individual and community values) and on the practical level (to use as a structure for ceremonies, discussions and problem solving). Talking circles allow everyone to get to know each other on a deeper level, co-constructing knowledge. This method of sharing is a traditional, and sacred ceremonial, means of learning, gaining knowledge and establishing relationships and has proven successful in Indigenous societies for generations. As a research method, talking circles reflect the equal relationship of each participant to the process of knowledge development within the whole (Wilson, 2008).
The Medicine Wheel: While not universal, the Medicine Wheel is considered one of the oldest symbols of First Nations spirituality (Kovach 2005). A circle divided into four quadrants, the Medicine Wheel can represent Aboriginal worldviews, illustrate the human journey through life, and explain relationships between various aspects of creation, both seen and unseen. Within research, the Medicine Wheel can serve as a framework for gathering information, organizing, or representing knowledge.
Ceremony: While Shawn Wilson (2008) suggests that research is ceremony, it is also important to note that Indigenous approaches to research typically incorporate ceremony. It is through ceremony that connections are made with the spiritual world and this enables stronger relationships and truer knowledge. How can we learn and understand if we are not fully connected? (Thus Wilson’s insight that research is ceremony.)
The Importance of Elders: Within Aboriginal communities Elders are leaders, educators, spiritual directors and healers. They see the world through the eyes of the ancestors and interpret it through lessons passed down from those ancestors. They are recognized and respected for knowing, for living, and for teaching traditional knowledge, ceremonies, and the stories of all our relations. They act as a bridge between the past and the present, between the spiritual and the physical worlds, and between the past memory and future vision. Within the research process, Elders play an essential part in guiding research and in the development and use of knowledge.
|
Jason’s Story: The Teachings of the Elders Whenever I conduct research, I am always reminded of the teaching and stories that the Elders used to share with me. One teaching relates to knowledge. An Elder told me that whenever I seek knowledge or obtain knowledge, to keep in mind that I am borrowing this knowledge, it is not mine to keep. The Elder explained that it is my responsibility to continue the cycle of knowledge and share it with my people |
While the second part of the review identified a strong and coherent literature regarding Indigenous ways of knowing and approaches to research, there were no case studies that applied this knowledge to Aboriginal CCIs. Only two of the eighteen case studies were related to Aboriginal CCIs –one in an off-reserve American Indian community and one an urban Aboriginal initiative in Canada. It is our contention that developmental/adaptive approaches to evaluation of CCIs with their focus on systems understanding, ongoing change and contextual sensitivity are most congruent with Indigenous ways of knowing and research approaches with their emphasis on interconnectedness, circularity, and balance and harmony. Moreover developmental/adaptive approaches suggest considerable flexibility for methodologies grounded in local protocols and practices and guided by Elders. Principles of respect reciprocity, responsibility, and relationship may also be a good fit with these Indigenous approaches due to their emphasis on local input into and control over evaluation design. However, this review also identified areas of tension and dissonance between developmental/adaptive approaches and Indigenous ways of knowing and approaches to research, i.e. the expectation that evaluation design should include identification of specific timelines for community level changes.
Ultimately there is a need for more research that engages CCIs focused on Aboriginal well-being in examining evaluation approaches and determining what is of most use.
Knowledge Mobilization
Knowledge mobilization will encompass social media and University websites, institutional and community presentations including with members of Aboriginal community change initiatives, social work educators and indigenous educators, the Urban Aboriginal Knowledge Network, the Prairie Child Welfare Consortium biennial conference. Presentations will also occur to funders, policy-makers, and advocates in child welfare and Aboriginal governance. We aim for 1-2 scholarly publications in national and international open access journals. But we contend that the most important knowledge mobilization will occur through the development of research that will bring together multi-sector community change initiatives aimed at enhancing Aboriginal Child and Family well-being to develop strategies for evaluation that honor Indigenous ways of knowing and approaches to research. While the priority should be on evaluation to enable learning and development through culturally respectful protocols and practices, the goal should also be to involve funders and relevant policy-makers in reflecting on their relevance to evidence-based policy development.
