Get 20M+ Full-Text Papers For Less Than $1.50/day. Subscribe now for You or Your Team.

Learn More →

Building a culture of impact in an international agricultural research organization: Process and reflective learning

Building a culture of impact in an international agricultural research organization: Process and... Abstract Most agricultural research organizations strive to address societal challenges and contribute to positive societal impacts. Fulfilling this ambition involves embedding a culture of impact in organizational culture, which, in our view, entails three main elements: understanding the role of the research community in contributing to impacts over the long term within the systems in which it operates; equipping researchers to support positive change; and implementing strategies that allow the culture of impact to percolate at various levels of the organization. To build just such a culture, in the past 8 years, Cirad, the French Agricultural Research Center for International Development, embarked on a transformational process, from which we draw key lessons. Building a culture of impact requires fostering transdisciplinary dialogue on the multiple roles of researchers, on their contribution to societal impacts, and on the relevance of this reflection. This involves adapting from pre-existing visions, interactions, and practices. Formalization in the organization’s strategy and the action of leading change agents foster its institutionalization. Strengthening capacity to build shared visions of change and collective processes in research design, implementation, and evaluation while respecting the diversity of profiles and approaches in the organization favors appropriation. This requires adequate funding at the project, institutional and funding bodies level, and targeted communication to ensure buy-in by internal and external change agents. We argue that a culture of impact is a reflective culture and long-term dynamics that aims to overcome the dichotomy between research and development and bring agricultural research closer to societal needs. 1. Introduction Within an organization, culture involves shared assumptions, values, beliefs, customs, experience, knowledge, and traditions that evolve over time, feeding the organization’s identity and mission (Schein 1988). Culture builds progressively, based on the organization’s history, interactions between resources, processes, routines, governance, and values (Ekboir et al. 2009). Building culture also relies on the organization’s capacity to provide the appropriate framework for the behavioral changes of its members (Mintzberg and Westley 1992). In this transformational process, structural and strategic reconfigurations occur, redesign, and reprogramming of systems and procedures and, beyond that, changes in operations, facilities, and professional profiles (Mintzberg and Westley 1992; Chapman 2002; Horton 2003). This introspective exercise can also reveal differences between what an organization communicates about its actions (the ‘espoused theory’), and the way it actually works (the ‘theory in use’) (Argyris 2004), and how those two levels interact over time. Moreover, enabling organizational change at various scales depends on the nature and scope of the organization, appropriate strategies for leveraging change, and change agent roles (Chapman 2002), but also on the type and phase of the organizational change process (Mintzberg and Westley 1992). For any organization keen on contributing to societal impacts, embedding in organizational culture the learning on how such impacts are generated (Douthwaite et al. 2003; Joly and Matt 2017) is important to improve research practices (Watts et al. 2007) and build a culture of impact. A culture of impact is not one that focuses on the efficiency or efficacy of the results of the organization but of the impacts it helps generate in the system in which it operates. However, to integrate it in organizational culture, the institution first needs to be convinced of the value of building a culture of impact. This conviction may have a number of reasons, including fulfilling its mission, responding to external pressures, accountability, learning, and improving its members’ vision of impact generation. These objectives can also be in tension (Joly et al. 2016). Additionally, the factors that drive an organization to analyze and evaluate its impacts can influence how it approaches evaluation (Penfield et al. 2014). Indeed, the increasing pressure from society and donors on agricultural research for development (AR4D) organizations to ‘prove their worth’, produces different responses. As an example, some organizations have responded through large-scale assessments focusing on economic impacts (Hazell 2008; Walker and Alwang 2015; Midmore 2017) or comparative assessments covering comprehensive impacts and impact generation processes (Joly et al. 2015; Temple et al. 2018). Researchers in AR4D sometimes resist recognizing that their role goes beyond producing sound science and knowledge and includes contributing to developmental impacts (Faure et al. 2010). This is a particularly divisive issue: should research only be held accountable for producing outputs, something on which researchers usually exert a fair degree of control? Or should it also promote output appropriation, leading to outcomes, which depend on multiple interactions between research and other stakeholders? Or should it be involved in ensuring that development actually takes place (impacts), even though researchers might not be legitimate in this role and impacts are mainly long term? Some authors respond that AR4D should be a bridge between the two (Thornton et al. 2017) or that it should be embedded in development praxis (Coe, Sinclair and Barrios 2014). Under these premises, the objective of this article is to analyze the process and strategies that can strengthen the capacity of agricultural research organizations to build a full-fledged culture of impact embedded in organizational culture. We analyze how improving impact assessment practices and increasing awareness of research impacts within the organization can lead to a different perception of the role and responsibilities of research in the innovation process and to advancing research practices. We take stock of the lessons learned so far from an empirical case based on the 8-year long ongoing experience of the French Agricultural Research Center for International Development (Cirad), a publicly funded AR4D organization with 800 researchers working in partnership with scientists and other actors in developing countries. We describe the phases that led Cirad from developing ex post evaluation practices of its members through to feeding lessons learned from impact assessment case studies into an ex ante approach, tools, and capacity strengthening to building impact pathways and shared visions of change between different stakeholders. The analysis relies on a review of the documents, methods, approaches, and tools produced, the various internal and external arenas in which these were discussed, and the various capacity strengthening and evaluation support activities carried out. Finally, we discuss how research organizations can foster a culture of impact, and outline the opportunities and challenges that such an endeavor entails. 2. Why does agricultural research for development need to develop and integrate a culture of impact? Development, in its many manifestations, is at the core of the mission of many national and international agricultural research organizations such as the CGIAR consortium and the national agricultural research systems of most developing countries. Cirad’s motto explicitly puts research for development at the center, as ‘contributing to sustainable development of rural areas and agricultural supply chains in southern countries, with particular emphasis on fairness and on the world’s poorest populations’ (Cirad DG-DRS 2012). Its research actions aim to support the achievement of the Sustainable Development Goals for and through sustainable agriculture. Certainly, to be able to fulfill such development-oriented missions, agricultural research organizations need to be adequately equipped to program, design, monitor, and evaluate their research, with development as the core long-term target. In other words, they need to build a ‘culture of impact’ (Hainzelin, Barret and Faure 2016; Leeuwis, Klerkx and Schut 2018; Hainzelin et al. 2017). All the while acknowledging that neither all research activities nor all researchers in a diverse and complex research organization can contribute directly to generating societal impacts, we deem that a culture of impact entails at least three elements. First, understanding the links between research activity and the changes, positive or negative, intended or unintended, that it can (ex ante), or did (ex post), generate at various scales, and formulating an organizational vision of the approach to analyze those processes. Learning change processes allows researchers to critically reflect on, and if necessary question, their role—among the roles of other stakeholders—in the research-innovation-impact continuum. While a researcher’s main role is the production of knowledge, we need to acknowledge that the research community plays multiple roles that contribute to impact generation (Matt et al. 2017, Faure et al. 2018). However, tensions can arise within the organization with its academic targets and from the evaluation of researchers through their scientific production as opposed to managing the increasing pressure to demonstrate societal impacts. Moreover, the multiple roles of researchers support, not replace, those who are legitimate actors of change. Therefore, transforming learning into real change depends on the transparent definition of the ‘rules of the game’ and the adherence of the different players to the process and its application. A second element of the culture of impact is adequately equipping researchers to support the processes that increase positive change and mitigate negative change: from design, implementation, and interaction with different actors and contexts, to evaluation. This entails developing common values, attitudes, and concepts, and providing researchers with a portfolio of approaches, tools, and methods to improve individual and collective research strategies and practices toward enabling positive societal change. In the context of agricultural research for development carried out in partnerships, this reflective learning should involve research and development partners in a similar effort of learning to foster change. Finally, building a culture of impact means implementing strategies that allow it to percolate at various scales, from the individual researcher, to the institution as a whole, via teams and partners and by adopting and adapting rules and actions that facilitate working toward impact, improving practices and interactions. This means that, to change the collective mindset about how it contributes to societal impact, the research organization should have an explicit vision of how it defines impact, development, and the innovation process. At Cirad, the process of building a culture of impact was driven more by internal motivation to learn and improve than to respond to external pressure from donors. This has fostered an internal debate on the most appropriate approaches and methods for ex post evaluation of the impacts of research activities in complex settings. Rather than measuring economic impacts of the adoption of technologies (Weißhuhn, Helming and Ferretti 2018) through mainstream evaluation methods (de Janvry, Dunstan and Sadoulet 2011), the institution developed its own impact evaluation framework, inspired by comprehensive approaches (Earl, Carden and Smutylo 2001; Douthwaite et al. 2003; Patton and Horton, 2009; Mayne 2012; Joly et al. 2015; Riley et al. 2018). The following section describes this process and its consequences for building a culture of impact at Cirad. 3. Results: The different phases of building a culture of impact at Cirad 3.1 Before the formalization of the culture of impact Cirad has a long experience in AR4D but until recently had no formal mechanisms to assess the impact of its research. Between the 1960s and the 1980s, the founding member institutes of Cirad focused a significant proportion of their actions on promoting and implementing agricultural development in developing countries that had recently become independent. Contributing to development not necessarily through research but rather through ‘development engineering’ was part and parcel of what many Cirad staff did and were expected to do by French and international public donors, and by national governments. From the early 2000s on, Cirad gradually underwent major structural changes to better align its functioning and staff profile with that typical of more academically oriented research organizations. This led the institution to increase the number of PhD level researchers and to encourage its numerous technically oriented staff to pursue more research actions and to produce scientific knowledge. In so doing, peer-reviewed, disciplinary science became increasingly important as reflected in the motto ‘Science at the Center’ used by Cirad higher-level management to communicate about this change. The emergence and expansion of a more academic culture in Cirad, particularly prevalent among the younger generation of researchers, gradually called into question the previously ‘organic’ relationship between research activities and their potential impact on development. It was in this context that in 2010, Cirad embarked on a reflective learning process about its impacts. This process was based on three principles: (1) acquiring a better understanding of how innovation systems in Southern countries unfold, taking into account the contributions of all stakeholders; (2) building an approach based on ‘impact pathways’ to influence scientific and research partnership planning, and to organize collective experience-sharing and learning capacity by making underlying theories of change explicit; and (3) reviewing existing research practices to build more effective projects in partnership, and simultaneously formalizing the building of the culture of impact. Between 2010 and 2018, three main phases can be distinguished, each of which contributed to developing this culture of impact (Fig. 1). Figure 1. Open in new tabDownload slide The sequence of formally building an impact culture at Cirad from 2010 on. Figure 1. Open in new tabDownload slide The sequence of formally building an impact culture at Cirad from 2010 on. 3.2 The ‘reflection and prototyping phase’: 2010–14 From 2010 to 2014, a small inter-institutional task force from four French research organizations, including representatives of Cirad, and the French Development Agency, sets the process in motion with a critical literature review of existing impact evaluation methods (Saint-Martin et al. 2011). The working group concluded that, in the context of agricultural research in developing countries, attributing change to a specific action would be challenging. Challenges concerned the lack of data, institutional weaknesses in partner structures, the partnership modus operandi of Cirad, the constraints on evaluating research outputs, and the delays between research action and change. Additionally, the report remarked that standard evaluation methods tend to underestimate negative effects and are blind to unexpected effects that do not stem from the planned logic of an intervention. The working group recommended testing alternative approaches and to analyze the causal links between action and change, and the role of different actors, beyond linear measures of economic efficiency. Small-scale testing (four case studies) of a prototype ex post impact assessment method deemed appropriate to Cirad’s modus operandi followed. This phase also saw Cirad includes impact assessment as part of its 5-year strategic vision in 2012. 3.3 The ‘testing’ phase: 2014–16 Between 2014 and mid-2016, an impact task force was established to apply and validate a further iteration of the prototyped method, named ImpresS (Impacts of Research in the South) (Barret et al. 2018). ImpresS ex post is an iterative and participatory evaluation method that includes systematization, validation, data collection, analysis, and actor feedback. Figure 2 summarizes the five phases of ImpresS ex post. The method was applied to 13 case studies selected among 90 candidate case studies submitted by Cirad research units, and involved around 30 Cirad researchers and their partners in developing countries. Several publications stemmed from these assessments, including a peer-reviewed special issue (Gaunand, Temple and Trouche 2018). Figure 2. Open in new tabDownload slide The five phases of the ImpresS ex post method. Figure 2. Open in new tabDownload slide The five phases of the ImpresS ex post method. Lessons from the in-depth case studies underlined (1) the long term-nature and multiplicity of impacts to which research contributes (Triomphe et al. 2016; Hainzelin et al. 2017; Temple et al. 2018), (2) the importance of co-producing outputs and outcomes with a variety of actors to generate impacts (Temple et al. 2018), (3) the key role of all stakeholders’ capacities in innovation processes leading to impacts (Toillier et al. 2018), (4) the role of interactions with public actors and policy makers in design or scaling, which help generate outcomes (Dabat and Grandjean 2018), and (5) the multiple roles played by the research community in co-developing outcomes with other actors of the innovation process (Faure et al. 2018). 3.4 The ‘strengthening the culture of impact’ phase (mid-2016 to present) The wealth of lessons learned from the cross-analysis of ex post studies motivated the Cirad impact team and top management to question how research actions are planned and to equip researchers and their partners with a suitable approach to co-build plausible ex ante impact pathways in the design of complex interventions (Earl, Carden and Smutylo 2001; Douthwaite et al. 2007; Mayne 2015). A second wave of methodological development followed, resulting in the ImpresS ex ante approach (Blundo-Canto et al. 2018). The aim was to reinforce the culture of impact by developing guidelines, by strengthening researchers’ and support staff’s capacities, and by accompanying projects and program design. The approach, summarized in Fig. 3, is participatory and iterative, and is a flexible process that can be adapted to the needs of each project. Figure 3. Open in new tabDownload slide A stepwise iterative approach to build plausible impact pathways. Figure 3. Open in new tabDownload slide A stepwise iterative approach to build plausible impact pathways. ImpresS ex ante focuses on the importance of interactions between intervention partners and stakeholders to facilitate their appropriation of outputs, leading to changes in visions, perspectives, practices, and behaviors. The approach stresses the need to make explicit the strategies envisaged to overcome potential obstacles to appropriation. With ImpresS ex ante becoming a key element in building a culture of impact, new staff were recruited, and training sessions, seminars, and dissemination events were organized for researchers and support units to strengthen capacity and buy-in throughout the institution. For instance, an ad hoc 3-day training session engaging all members of project management support units was organized to integrate ex ante reflection into their practices. The design of this module revealed other training needs focused on more specific tools for project design and management, and the integration of ImpresS ex ante concepts in the ‘project leader training’ sessions periodically organized for Cirad researchers. Born from the lessons of the ex post case studies and the evaluation literature, ImpresS ex ante played a key role in strengthening the culture of impact by structuring the design of projects and programs around plausible impact pathways emerging from collective reflection and shared visions. Finally, during this 8-year period, multiple interactions between the impact team and Cirad’s higher management made it possible to align methodological development and learning from impact assessment with the organization’s strategy. Four ‘culture of impact’ seminars were organized during the annual meetings, when most Cirad researchers come together for a week, to disseminate and discuss methodological development and findings. Three lectures on the culture of impact were given involving international discussants and key funding bodies, enabling external dissemination and lobbying. Additionally, members of the impact team took part in international workshops to get feedback from experts in impact assessment as well as from the agricultural research community. To date, 14 project design workshops that include an ImpresS ex ante component have been conducted in different countries (France, Senegal, Costa Rica, Burkina Faso, Madagascar, Tunisia, and Peru), involving 14 different projects of different scope, scale, and topics, and more than 100 researchers and representatives from partner institutions. 4. Discussion: First lessons in building a culture of impact in agricultural research organizations As the experience we describe in this article shows, building a culture of impact in a research organization implies assimilating broader organizational culture. This involves nested changes, from broader and conceptual changes to concrete strategic, structural, professional, and operational actions. Eight years into this process at Cirad, notable progress has been made, but internal resistance and new challenges have emerged. For a process to be truly transformative, it needs be sustained over time and to overcome internal resistance in the form of defensive behaviors against changes that may be perceived as threatening (Argyris 2004). In Cirad’s experience, to foster long-term sustainability and to reduce and overcome internal tensions, doting the institution with adequate resources to establish dialogue, but also evaluating the changes in behavior and in practices that this culture of impact entails, remains essential. It is important to state that adhering to a culture of impact does not mean distorting the core business of most researchers (i.e. producing new knowledge and research outputs). Rather, it implies enriching their thinking, practices, and interactions, through methods, approaches, and tools to choose strategies that plausibly increase the appropriation of research outputs by stakeholders. The transformative process of building an impact culture at Cirad was not linear. Stemming from the original intention of using impact evaluation to understand how research contributes to generate societal impacts, acknowledging the long trajectory of Cirad’s own contribution, it evolved into applying this learning to build and equip researcher’s practices and interactions with other stakeholders. Even if this process is still in its early stages, the lessons already learned provide valuable clues on how to integrate a culture of impact in agricultural research for development organizations. The institutional capacity to move toward this culture of impact can enhance the performance and societal contribution of these research organizations. Integrating the culture of impact also increases researchers’ and managers’ ability to learn from the organization’s functioning, and to use this learning to evolve and adapt to external and internal factors of change. In the following section, we present six key lessons we believe are valid beyond Cirad’s own experience. 4.1 Fostering transdisciplinary understanding for the evaluation of complex problems Complexity theory acknowledges that changes do not occur linearly in dynamic, transformational complex systems, in which multiple actors interact (Byrne 1998). This challenges the notion of narrowly measuring predetermined outcomes following a linear cause–effect logic (Patton 2006). Complexity science is increasingly promoted for the analysis of dynamic and adaptive agricultural innovation systems (Ekboir 2003; Patton and Horton 2009; Douthwaite and Gummert 2010; Mayne 2012), but the challenges of evaluating a diversity of impacts at different scales (Joly et al. 2015) and beyond science and economic impacts (Joly et al. 2016; Gaunand et al. 2015) need to be taken into account. Open debates about the role of scientists in contributing to societal impacts and in assessing the impacts of the innovation processes to which they contribute are needed (Temple et al. 2018). As a corollary, innovative approaches to evaluate researchers, programs, and teams taking into account transdisciplinary and social impacts (Belcher et al. 2016) should be devised. Additionally, the approaches and methods to evaluate impacts in complex systems, where heterogeneous effects occur in multiple dimensions, require cross-discipline points of view and interactions. Research and impact evaluation teams should consequently be equipped to solve complex problems through transdisciplinary interactions. When building a culture of impact, transdisciplinarity involves scientists from different disciplines and/or fulfilling a variety of roles and functions in innovation processes, as well as managers and support staff, but also research and development partners, in creating shared solutions (Klerkx et al. 2010) to societal problems. However, methodological and epistemological principles and practices differ, and the same terminology can express different concepts (Belcher et al. 2016). The development of a culture of impact therefore requires creating a common lexicon around impact, to allow diverse people to communicate and interact. As mentioned above, at Cirad, different types of events and arenas encouraged diversity of expressions and contributed to a shared vision of what a culture of impact is. As a corollary, the very use of the terminology of impact culture allowed the circulation of concepts formerly restricted to certain parts of the organization. Building relevance, credibility, and internal and external legitimacy (Hansson and Polk 2018) within this transdisciplinary dialogue and interactions was crucial. 4.2 Adaptability Favoring change processes should not be seen as an abrupt change in direction: changing attitudes and beliefs to change practices involves building on and adapting existing ones. Adaptability means (1) using potential levers and addressing barriers to appropriation of a culture of impact by managers, researchers, and their partners, and (2) adopting a participatory, adaptive, and iterative process in both methodological development and implementation. For instance, ImpresS ex ante was deliberately conceived to incorporate existing project design approaches and was non-normative in its application. While the approach provides a structured way of thinking about project design and planning, how it is actually implemented depends on the needs of each project team. Building a common language so that researchers from different disciplines and their partners can interact to create shared visions of change is a key step in this adaptability. Sharing concepts and language makes it possible to create and strengthen interest, buy-in, and appropriation. The seminars, guidelines, workshops, and ongoing support provided to research teams, all aim at creating this shared language and interest. At Cirad, the increasing number of researchers, teams, and support staff who approach the impact team for guidance and support is a sign of ongoing appropriation. The permanent two-way learning loop that is created through this support also enables continuous improvement in the ImpresS approach, increasing its relevance and adaptability to users’ needs and purposes. Of course, other factors also contribute to this interest and appropriation, such as increasing pressure from donors to prove the societal value of research, the internal motivation of research teams, researchers, and support staff, or emerging demands from partners and stakeholders. 4.3 Institutional buy-in Chapman (2002) synthesized how transformational change benefits from strong leadership, the ability to involve those affected by change as change agents, and external facilitators. As shown by Cirad’s experience, support and encouragement from upper management plays a crucial role in building a culture of impact. Leadership buy-in, also formalized in strategic documents, provides an enabling context for methodological development, integration in research and management practices, and capacity strengthening. In innovation systems thinking, one refers to the role of lobbying activities and interest groups to counter resistance and to increase the legitimacy of an innovation (Hekkert et al 2007). Change agents at the upper management level, who lead debate and actions to strengthen the culture of impact, together with brokering individuals, enable and support two-way communication at all levels of the institutional structure. This facilitates managers’ appropriation of results and proposals related to the culture of impact. This appropriation can occur through participation in methodological development, testing, and validation; targeted capacity strengthening in evaluation; and participation in case studies. Certainly, the long-term durability of the drive toward a culture of impact will depend on its widespread validation and appropriation by researchers, support staff, and managers at all levels, fostered by increased project effectiveness, but also by an improved ability to interact in transdisciplinary research settings, to which we refer above. 4.4 Capacity development and network building The process of building a culture of impact relies to a great extent on supporting researchers’ understanding of how research contributes to impacts and on the implementation of strategies that support the emergence of positive change. Such a strategy aims to ‘pro-actively create awareness amongst innovation network actors as regards their position vis-à-vis their institutional environment’ (Klerkx et al. 2010: 399). To put it in another way, it makes it possible to design research actions in a participatory manner that takes advantage of collective intelligence to increase their success. However, this often requires new skills and new ways of thinking through collective processes. It also highlights the different functions played by research teams with multiple competences at different stages of the impact pathway. Fostering a culture of impact may also be perceived as threatening by researchers whose schedules are already stretched to breaking point and who have multiple demands on their time. Providing support structures is therefore indispensable: this includes capacity strengthening, making resource persons available to accompany them, along with access to relevant in-house guidelines. Providing training in different formats that are appropriate for researchers, their partners, and support staff plays a key role. Another approach is creating extensive network of coaches and resource persons. These actions create legitimacy for the culture of impact and aim to overcome typical resistance to change. Additionally, new types of professionals are needed who have the capacity to build bridges between researchers and managers, support staff, donors, and other stakeholders. This new professionalism is advocated for agricultural research (Douthwaite et al. 2017). Key resource persons inside or outside the organization, who are willing and who are able to play this bridging role at the interface of science, management, and participatory methods of project development should be identified, as they play a key role in scaling the culture of impact. The complementary skills this new professional requires include a combination of planning, monitoring and evaluation skills with systems and complexity thinking; the capacity to accompany researchers in these processes; and the ability to work with different stakeholders, including researchers from the natural, political, and social sciences and their partners. 4.5 Financial mechanisms Developing and strengthening a culture of impact requires significant funding. In practical terms, at the project planning level, this should include a budget for partners’ and stakeholders’ workshops to iteratively build a shared impact pathway and strategy. At the research organization level, management can earmark funds for the construction of large-scale projects ex ante; for the implementation of ex post evaluations; for the functioning of methodological groups; for capacity development; for network building; or for new staff in impact assessment research and support activities. At the level of funding bodies, a key challenge is to ensure research funding supports longer project cycles and takes into account the fact that impacts emerge in the long term, usually through clusters of projects, rather than as the result of the work accomplished in one project alone. Funding bodies can support this in several ways: by favoring bridging and continuity between short-term project cycles and incentivizing interactions between clusters of projects; by promoting monitoring and evaluation frameworks that allow for transparent adaptive management instead of rigid reporting mechanisms; or by financing longer inception phases during which context-relevant and more effective strategies can be defined with local stakeholders. This could include providing a budget to develop and/or validate impact pathways at the end of the first year of a project, after a thorough appraisal and partnership building. This also requires a fair amount of lobbying of and communication with funding bodies by the research community. 4.6 External and internal communication and partnerships Understanding and learning from the different functions played by research, and the contributions it can make toward achieving impacts, should become an integral part of individual, collective, and institutional reflection. In Cirad’s experience, adequate internal communication during organization-wide events has been important to support the emergence of a transdisciplinary understanding. In addition, external and scientific communication about the culture of impact increases its legitimacy in the eyes of other stakeholders, particularly funders. These communication channels include international conferences, events for funding bodies, and participation in global platforms on agricultural research, but also research partnerships with other research organizations to share experience in and lessons learned from impact generation and the role of research. Ongoing actions at Cirad support long-term strengthening of the culture of impact through capacity strengthening, communication, monitoring and evaluation, and research on organizational learning and change in partnership with other research organizations. 5. Conclusion This article takes stock of the first lessons learned concerning the processes and approaches to build a culture of impact in agricultural research for development organizations, based on the experience gained by Cirad. A transformational process was set in motion by the need to increase knowledge on how research contributes to societal impacts and to identify appropriate evaluation methods for complex interventions. This started with the development, testing, and application of a participatory ex post evaluation method (ImpresS). The results achieved and the lessons learned from the application of ImpresS ex post generated new knowledge on impact generation processes and revealed the need to better equip researchers to plan and to act for impact in an ex ante fashion. To this end, an ad hoc approach for reflective, participatory, and structured project design and planning was developed. Building a culture of impact in a research organization implies individual, collective, and institutional understanding of how impact generation processes work in complex systems. To thrive, the culture of impact should permeate all levels of the organization, from top management to researchers and project support staff. A culture of impact means shifting from an exclusive focus on research outputs to a wider focus on how to encourage their appropriation, by identifying strategies and interactions that foster perception, practice, and behavioral changes. However, one of the main challenges is building awareness and acknowledgment of the contribution of research to societal impacts among all the organizations’ members, while respecting the diversity of profiles and approaches, and the wide range of outputs that are produced, as not everything needs to generate impact. It also involves accepting that there is not only one way to achieve and to evaluate the Research-Impact link, even if an institution promotes a certain approach. In this sense, by acknowledging the multiple roles the research community plays in interaction with other actors, a culture of impact can help overcome the dichotomy between research and development. To do so, reflection on impact should be embedded in the research process, that is, be an integral part of it, rather than a separate task. As we have shown, building a culture of impact implies a conceptualization of impact culture embedded in organizational culture and collective experience, supported by transdisciplinary dialog to share an understanding of impact culture. This requires adaptability in building on existing practices, along with institutional support from leading change agents and recognition of the impact culture through formalization in the organization’s strategy. Appropriation of the culture of impact relies on shared language, acknowledgment, and understanding of the role of the research community in contributing to societal impacts. It also entails adequately equipping researchers and their partners and strengthening their capacity to support collective processes in research design, implementation, and evaluation. Finally, this requires adequate funding at the project, institutional and funding bodies level, and targeted communication to ensure buy-in by internal and external change agents. Ultimately, a culture of impact is a reflective culture oriented toward actionable organizational learning: understanding impact generation and the contribution of research is part of a long-term dynamics that aims to bring agricultural research for development closer to societal needs. Acknowledgements The authors would like to thank Boru Douthwaite for his insightful recommendations in the development of the ImpresS approach. The authors would also like to thank all the researchers, their partners, and the support staff who participated and continue to participate in building a culture of impact at Cirad. References Argyris C. ( 2004 ) Double-Loop Learning and Organizational Change, in Boonstra J. J. (ed.) Dynamics of Organizational Change and Learning , pp. 389 – 402 . Hoboken, NJ : John Wiley & Sons Ltd . https://doi.org/10.1002/9780470753408.ch19 Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Barret D. et al. ( 2018 ) ImpresS Ex Post. Methodological Guide to Ex Post Evaluation of Agricultural Research in Developing Countries . France : CIRAD, Montpellier . https://doi.org/10.19182/agritrop/00006 Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Belcher B. M. et al. ( 2016 ) ‘ Defining and Assessing Research Quality in a Transdisciplinary Context ’, Research Evaluation , 25 : 1 – 17 . Google Scholar Crossref Search ADS WorldCat Blundo-Canto G. et al. ( 2018 ) ImpresS Ex Ante. An Approach for Building Ex Ante Impact Pathways . France : CIRAD, Montpellier . https://doi.org/10.19182/agritrop/00013 Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Byrne D. S. ( 1998 ) Complexity Theory and the Social Sciences: An Introduction . New York, NY : Routledge, London . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Chapman J. A. ( 2002 ) ‘ A Framework for Transformational Change in Organisations ’, Leadership and Organization Development Journal , 23 : 16 – 25 . Google Scholar Crossref Search ADS WorldCat CIRAD DG-DRS. ( 2012 ) Strategic Vision 2012-2022. Making Research a Real Tool for Development , pp. 6 . Montpellier, France : Cirad . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Coe R. , Sinclair F., Barrios E. ( 2014 ) ‘ Scaling up Agroforestry Requires Research ‘In’ Rather than ‘For’ Development ’, Current Opinion in Environmental Sustainability , 6 : 73 – 7 . Google Scholar Crossref Search ADS WorldCat Dabat M. H. , Grandjean A. ( 2018 ) ‘ Comment Les Acteurs Publics Contribuent à L’impact de la Recherche Dans Les Pays en Développement ’, Cahiers Agricultures , 27 . https://doi.org/10.1051/cagri/2018003 Google Scholar OpenURL Placeholder Text WorldCat de Janvry A. , Dunstan A., Sadoulet E. ( 2011 ) Recent Advances in Impact Analysis Methods for Ex-Post Impact Assessments of Agricultural Technology: Options for the CGIAR . Rome, Italy : ISPCC . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Douthwaite B. et al. ( 2007 ) ‘ Participatory Impact Pathways Analysis: A Practical Application of Program Theory in Research-for-Development ’, Canadian Journal of Program Evaluation , 22 : 127 – 59 . Google Scholar OpenURL Placeholder Text WorldCat Douthwaite B. , Gummert M. ( 2010 ) ‘ Learning Selection Revisited: How Can Agricultural Researchers Make a Difference? ’, Agricultural Systems , 103 : 245 – 55 . https://doi.org/10.1016/j.agsy.2010.01.005 Google Scholar Crossref Search ADS WorldCat Douthwaite B. et al. ( 2017 ) ‘ A New Professionalism for Agricultural Research for Development ’, International Journal of Agricultural Sustainability , 15 / 3 : 238 – 52 . Google Scholar Crossref Search ADS WorldCat Douthwaite B. et al. ( 2003 ) ‘ Impact Pathway Evaluation: An Approach for Achieving and Attributing Impact in Complex Systems ’, Agricultural Systems , 78 : 243 – 65 . Google Scholar Crossref Search ADS WorldCat Earl S. , Carden F., Smutylo T. ( 2001 ) Outcome Mapping: Building Learning and Reflection into Development Programs . Ottawa : International Development Research Centre . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Ekboir J. ( 2003 ) ‘ Why Impact Analysis Should Not Be Used for Research Evaluation and What the Alternatives Are ’, Agricultural Systems , 78 : 166 – 84 . Google Scholar Crossref Search ADS WorldCat Ekboir J. M. et al. ( 2009 ). Successful Organizational Learning in the Management of Agricultural Research and Innovation: The Mexican Produce Foundations, IFPRI Research Report. Washington, DC : IFPRI . https://doi.org/10.2499/9780896291713RR162 Faure G. et al. ( 2018 ) ‘ How Different Agricultural Research Models Contribute to Impacts: Evidence from 13 Case Studies in Developing Countries ’, Agricultural Systems , 165 : 128 – 36 . Google Scholar Crossref Search ADS WorldCat Faure G. et al. ( 2010 ) Innover avec les acteurs du monde rural. La recherche-action en partenariat, Agricultures tropicales en poche . Versailles : Editions Quæ . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Gaunand A. et al. ( 2015 ) ‘ How Does Public Agricultural Research Impact Society? A Characterization of Various Patterns ’, Research Policy , 44 : 849 – 61 . Google Scholar Crossref Search ADS WorldCat Gaunand A. , Temple L., Trouche G. ( 2018 ) ‘ Évaluer Les Impacts Des Recherches en Agriculture Sur la Société: Outils, Méthodes, Études de Cas ’, Cahiers Agricultures , 27 : 3400. https://www.cahiersagricultures.fr/component/toc/?task=topic&id=872 Google Scholar OpenURL Placeholder Text WorldCat Hainzelin E. , Barret D., Faure G. ( 2016 ) Agriculture Research in Developing Countries: From a “Culture of Promise” to a “Culture of Impact ”. France : CIRAD, Montpellier . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hainzelin E. et al. ( 2017 ). Agricultural Research in the Global South: Steering Research beyond Impact Promises . France : CIRAD, Montpellier . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hazell P. ( 2008 ). An Assessment of the Impact of Agricultural Research in South Asia since the Green Revolution . Rome, Italy : Science Council Secretariat . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hansson S. , Polk M. ( 2018 ) ‘ Assessing the Impact of Transdisciplinary Research: The Usefulness of Relevance, Credibility, and Legitimacy for Understanding the Link between Process and Impact ’, Research Evaluation , 27 : 132 – 44 . Google Scholar Crossref Search ADS WorldCat Hekkert M. P. et al. ( 2007 ) ‘ Functions of Innovation Systems: A New Approach for Analysing Technological Change ’, Technological Forecasting and Social Change , 74 : 413 – 32 . Google Scholar Crossref Search ADS WorldCat Horton D. (ed.), ( 2003 ) Evaluating Capacity Development: Experiences from Research and Development Organizations around the World . The Hague; Ottawa; Wageningen : ISNAR, CTA, IDRC . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Joly P.-B. et al. ( 2015 ) ‘ ASIRPA: A Comprehensive Theory-Based Approach to Assessing the Societal Impacts of a Research Organization ’, Research Evaluation , 24 : 440 – 53 . Google Scholar Crossref Search ADS WorldCat Joly P.-B. et al. ( 2016 ) Agricultural Research Impact Assessment: Issues, Methods and Challenges . OECD Food, Agriculture and Fisheries Papers, No. 98, Paris: OECD Publishing. http://dx.doi.org/10.1787/5339e165-en Joly P.-B. , Matt M. ( 2017 ) ‘ Towards a New Generation of Research Impact Assessment Approaches’ , The Journal of Technology Transfer . https://doi.org/10.1007/s10961-017-9601-0 Google Scholar OpenURL Placeholder Text WorldCat Klerkx L. , Aarts N., Leeuwis C. ( 2010 ) ‘ Adaptive Management in Agricultural Innovation Systems: The Interaction between Innovation Networks and Their Environment ’, Agricultural Systems , 103 : 390 – 400 . Google Scholar Crossref Search ADS WorldCat Leeuwis C. , Klerkx L., Schut M. ( 2018 ) ‘ Reforming the Research Policy and Impact Culture in the CGIAR: Integrating Science and Systemic Capacity Development ’, Global Food Security , 16 : 17 – 21 . Google Scholar Crossref Search ADS WorldCat Matt M. et al. ( 2017 ) ‘ Opening the Black Box of Impact–Ideal-Type Impact Pathways in a Public Agricultural Research Organization ’, Research Policy , 46 : 207 – 18 . Google Scholar Crossref Search ADS WorldCat Mayne J. ( 2012 ) ‘ Contribution Analysis: Coming of Age? ’, Evaluation , 18 : 270 – 80 . Google Scholar Crossref Search ADS WorldCat Mayne J. ( 2015 ) ‘ Useful Theory of Change Models ’, Canadian Journal of Program Evaluation , 30 : 119 – 42 . https://doi.org/10.3138/cjpe.230. Google Scholar Crossref Search ADS WorldCat Midmore P. ( 2017 ) ‘ The Science of Impact and the Impact of Agricultural Science ’, Journal of Agricultural Economics , 68 : 611 – 31 . Google Scholar Crossref Search ADS WorldCat Mintzberg H. , Westley F. ( 1992 ) ‘ Cycles of Organizational Change ’, Strategic Management Journal , 13 : 39 – 59 . Google Scholar Crossref Search ADS WorldCat Patton M. Q. ( 2006 ) ‘ Evaluation for the Way we Work ’, Nonprofit Quaterly Spring , 28 : 33 . Google Scholar OpenURL Placeholder Text WorldCat Patton M. Q. , Horton D. ( 2009 ) Utilization-Focused Evaluation for Agricultural Innovation . Institutional Learning and Change (ILAC) Initiative Brief No. 22. Bioversity, Rome: ILAC. Penfield T. et al. ( 2014 ) ‘ Assessment, Evaluations, and Definitions of Research Impact: A Review ’, Research Evaluation , 23 / 1 : 21 – 32 . Google Scholar Crossref Search ADS WorldCat Riley B. L. et al. ( 2018 ) ‘ Using Contribution Analysis to Evaluate the Impacts of Research on Policy: Getting to ‘Good Enough ’, Research Evaluation , 27 / 1 : 16 – 27 . Google Scholar Crossref Search ADS WorldCat Saint-Martin G. et al. ( 2011 ) Evaluation de L’impact de la Recherche au Cirad: Rapport du Groupe de Travail . France : CIRAD, Montpellier . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Schein E. H. ( 1988 ) ‘Organizational Culture: What It Is and How to Change It’, in Human Resource Management in International Firms , pp. 56 – 82 . London : Palgrave Macmillan . https://doi.org/10.1007/978-1-349-11255-5_4 Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Temple L. et al. ( 2018 ) ‘ Assessing Impacts of Agricultural Research for Development: A Systemic Model Focusing on Outcomes ’, Research Evaluation , 27 : 157 – 70 . Google Scholar Crossref Search ADS WorldCat Thornton P. et al. ( 2017 ) ‘ Responding to Global Change: A Theory of Change Approach to Making Agricultural Research for Development Outcome-Based ’, Agricultural Systems , 152 : 145 – 53 . Google Scholar Crossref Search ADS WorldCat Toillier A. et al. ( 2018 ) ‘ Comprendre la Contribution de la Recherche à L’innovation Collective Par L’exploration de Mécanismes de Renforcement de Capacité ’, Cahiers Agricultures , 27 : 15002. Google Scholar Crossref Search ADS WorldCat Triomphe B. et al. ( 2016 ) ‘ Mieux évaluer et accompagner l'innovation agricole en Afrique. Leçons d'une analyse transversale de 13 cas d'études ’, Cahiers Agricultures , 25 : 64003 , DOI: 10.1051/cagri/2016050. Google Scholar Crossref Search ADS WorldCat Walker T.S. , Alwang J. (eds), ( 2015 ) Crop Improvement, Adoption, and Impact of Improved Varieties in Food Crops in Sub-Saharan Africa . Croydon, UK : CGIAR Consortium of International Agricultural Research Centers and CAB International . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Watts J. et al. ( 2007 ) ‘ Institutional Learning and Change: An Introduction ’, ILAC Working Papers , 3 : 19. http://hdl.handle.net/10568/70607 Google Scholar OpenURL Placeholder Text WorldCat Weißhuhn P. , Helming K., Ferretti J. ( 2018 ) ‘ Research Impact Assessment in Agriculture—a Review of Approaches and Impact Areas ’, Research Evaluation , 27 : 36 – 42 . Google Scholar Crossref Search ADS WorldCat © The Author(s) 2018. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contactjournals.permissions@oup.com © The Author(s) 2018. Published by Oxford University Press. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Research Evaluation Oxford University Press

Building a culture of impact in an international agricultural research organization: Process and reflective learning

Loading next page...
 
/lp/oxford-university-press/building-a-culture-of-impact-in-an-international-agricultural-research-u80yQNjuCc

References (58)

Publisher
Oxford University Press
Copyright
Copyright © 2022 Oxford University Press
ISSN
0958-2029
eISSN
1471-5449
DOI
10.1093/reseval/rvy033
Publisher site
See Article on Publisher Site

Abstract

Abstract Most agricultural research organizations strive to address societal challenges and contribute to positive societal impacts. Fulfilling this ambition involves embedding a culture of impact in organizational culture, which, in our view, entails three main elements: understanding the role of the research community in contributing to impacts over the long term within the systems in which it operates; equipping researchers to support positive change; and implementing strategies that allow the culture of impact to percolate at various levels of the organization. To build just such a culture, in the past 8 years, Cirad, the French Agricultural Research Center for International Development, embarked on a transformational process, from which we draw key lessons. Building a culture of impact requires fostering transdisciplinary dialogue on the multiple roles of researchers, on their contribution to societal impacts, and on the relevance of this reflection. This involves adapting from pre-existing visions, interactions, and practices. Formalization in the organization’s strategy and the action of leading change agents foster its institutionalization. Strengthening capacity to build shared visions of change and collective processes in research design, implementation, and evaluation while respecting the diversity of profiles and approaches in the organization favors appropriation. This requires adequate funding at the project, institutional and funding bodies level, and targeted communication to ensure buy-in by internal and external change agents. We argue that a culture of impact is a reflective culture and long-term dynamics that aims to overcome the dichotomy between research and development and bring agricultural research closer to societal needs. 1. Introduction Within an organization, culture involves shared assumptions, values, beliefs, customs, experience, knowledge, and traditions that evolve over time, feeding the organization’s identity and mission (Schein 1988). Culture builds progressively, based on the organization’s history, interactions between resources, processes, routines, governance, and values (Ekboir et al. 2009). Building culture also relies on the organization’s capacity to provide the appropriate framework for the behavioral changes of its members (Mintzberg and Westley 1992). In this transformational process, structural and strategic reconfigurations occur, redesign, and reprogramming of systems and procedures and, beyond that, changes in operations, facilities, and professional profiles (Mintzberg and Westley 1992; Chapman 2002; Horton 2003). This introspective exercise can also reveal differences between what an organization communicates about its actions (the ‘espoused theory’), and the way it actually works (the ‘theory in use’) (Argyris 2004), and how those two levels interact over time. Moreover, enabling organizational change at various scales depends on the nature and scope of the organization, appropriate strategies for leveraging change, and change agent roles (Chapman 2002), but also on the type and phase of the organizational change process (Mintzberg and Westley 1992). For any organization keen on contributing to societal impacts, embedding in organizational culture the learning on how such impacts are generated (Douthwaite et al. 2003; Joly and Matt 2017) is important to improve research practices (Watts et al. 2007) and build a culture of impact. A culture of impact is not one that focuses on the efficiency or efficacy of the results of the organization but of the impacts it helps generate in the system in which it operates. However, to integrate it in organizational culture, the institution first needs to be convinced of the value of building a culture of impact. This conviction may have a number of reasons, including fulfilling its mission, responding to external pressures, accountability, learning, and improving its members’ vision of impact generation. These objectives can also be in tension (Joly et al. 2016). Additionally, the factors that drive an organization to analyze and evaluate its impacts can influence how it approaches evaluation (Penfield et al. 2014). Indeed, the increasing pressure from society and donors on agricultural research for development (AR4D) organizations to ‘prove their worth’, produces different responses. As an example, some organizations have responded through large-scale assessments focusing on economic impacts (Hazell 2008; Walker and Alwang 2015; Midmore 2017) or comparative assessments covering comprehensive impacts and impact generation processes (Joly et al. 2015; Temple et al. 2018). Researchers in AR4D sometimes resist recognizing that their role goes beyond producing sound science and knowledge and includes contributing to developmental impacts (Faure et al. 2010). This is a particularly divisive issue: should research only be held accountable for producing outputs, something on which researchers usually exert a fair degree of control? Or should it also promote output appropriation, leading to outcomes, which depend on multiple interactions between research and other stakeholders? Or should it be involved in ensuring that development actually takes place (impacts), even though researchers might not be legitimate in this role and impacts are mainly long term? Some authors respond that AR4D should be a bridge between the two (Thornton et al. 2017) or that it should be embedded in development praxis (Coe, Sinclair and Barrios 2014). Under these premises, the objective of this article is to analyze the process and strategies that can strengthen the capacity of agricultural research organizations to build a full-fledged culture of impact embedded in organizational culture. We analyze how improving impact assessment practices and increasing awareness of research impacts within the organization can lead to a different perception of the role and responsibilities of research in the innovation process and to advancing research practices. We take stock of the lessons learned so far from an empirical case based on the 8-year long ongoing experience of the French Agricultural Research Center for International Development (Cirad), a publicly funded AR4D organization with 800 researchers working in partnership with scientists and other actors in developing countries. We describe the phases that led Cirad from developing ex post evaluation practices of its members through to feeding lessons learned from impact assessment case studies into an ex ante approach, tools, and capacity strengthening to building impact pathways and shared visions of change between different stakeholders. The analysis relies on a review of the documents, methods, approaches, and tools produced, the various internal and external arenas in which these were discussed, and the various capacity strengthening and evaluation support activities carried out. Finally, we discuss how research organizations can foster a culture of impact, and outline the opportunities and challenges that such an endeavor entails. 2. Why does agricultural research for development need to develop and integrate a culture of impact? Development, in its many manifestations, is at the core of the mission of many national and international agricultural research organizations such as the CGIAR consortium and the national agricultural research systems of most developing countries. Cirad’s motto explicitly puts research for development at the center, as ‘contributing to sustainable development of rural areas and agricultural supply chains in southern countries, with particular emphasis on fairness and on the world’s poorest populations’ (Cirad DG-DRS 2012). Its research actions aim to support the achievement of the Sustainable Development Goals for and through sustainable agriculture. Certainly, to be able to fulfill such development-oriented missions, agricultural research organizations need to be adequately equipped to program, design, monitor, and evaluate their research, with development as the core long-term target. In other words, they need to build a ‘culture of impact’ (Hainzelin, Barret and Faure 2016; Leeuwis, Klerkx and Schut 2018; Hainzelin et al. 2017). All the while acknowledging that neither all research activities nor all researchers in a diverse and complex research organization can contribute directly to generating societal impacts, we deem that a culture of impact entails at least three elements. First, understanding the links between research activity and the changes, positive or negative, intended or unintended, that it can (ex ante), or did (ex post), generate at various scales, and formulating an organizational vision of the approach to analyze those processes. Learning change processes allows researchers to critically reflect on, and if necessary question, their role—among the roles of other stakeholders—in the research-innovation-impact continuum. While a researcher’s main role is the production of knowledge, we need to acknowledge that the research community plays multiple roles that contribute to impact generation (Matt et al. 2017, Faure et al. 2018). However, tensions can arise within the organization with its academic targets and from the evaluation of researchers through their scientific production as opposed to managing the increasing pressure to demonstrate societal impacts. Moreover, the multiple roles of researchers support, not replace, those who are legitimate actors of change. Therefore, transforming learning into real change depends on the transparent definition of the ‘rules of the game’ and the adherence of the different players to the process and its application. A second element of the culture of impact is adequately equipping researchers to support the processes that increase positive change and mitigate negative change: from design, implementation, and interaction with different actors and contexts, to evaluation. This entails developing common values, attitudes, and concepts, and providing researchers with a portfolio of approaches, tools, and methods to improve individual and collective research strategies and practices toward enabling positive societal change. In the context of agricultural research for development carried out in partnerships, this reflective learning should involve research and development partners in a similar effort of learning to foster change. Finally, building a culture of impact means implementing strategies that allow it to percolate at various scales, from the individual researcher, to the institution as a whole, via teams and partners and by adopting and adapting rules and actions that facilitate working toward impact, improving practices and interactions. This means that, to change the collective mindset about how it contributes to societal impact, the research organization should have an explicit vision of how it defines impact, development, and the innovation process. At Cirad, the process of building a culture of impact was driven more by internal motivation to learn and improve than to respond to external pressure from donors. This has fostered an internal debate on the most appropriate approaches and methods for ex post evaluation of the impacts of research activities in complex settings. Rather than measuring economic impacts of the adoption of technologies (Weißhuhn, Helming and Ferretti 2018) through mainstream evaluation methods (de Janvry, Dunstan and Sadoulet 2011), the institution developed its own impact evaluation framework, inspired by comprehensive approaches (Earl, Carden and Smutylo 2001; Douthwaite et al. 2003; Patton and Horton, 2009; Mayne 2012; Joly et al. 2015; Riley et al. 2018). The following section describes this process and its consequences for building a culture of impact at Cirad. 3. Results: The different phases of building a culture of impact at Cirad 3.1 Before the formalization of the culture of impact Cirad has a long experience in AR4D but until recently had no formal mechanisms to assess the impact of its research. Between the 1960s and the 1980s, the founding member institutes of Cirad focused a significant proportion of their actions on promoting and implementing agricultural development in developing countries that had recently become independent. Contributing to development not necessarily through research but rather through ‘development engineering’ was part and parcel of what many Cirad staff did and were expected to do by French and international public donors, and by national governments. From the early 2000s on, Cirad gradually underwent major structural changes to better align its functioning and staff profile with that typical of more academically oriented research organizations. This led the institution to increase the number of PhD level researchers and to encourage its numerous technically oriented staff to pursue more research actions and to produce scientific knowledge. In so doing, peer-reviewed, disciplinary science became increasingly important as reflected in the motto ‘Science at the Center’ used by Cirad higher-level management to communicate about this change. The emergence and expansion of a more academic culture in Cirad, particularly prevalent among the younger generation of researchers, gradually called into question the previously ‘organic’ relationship between research activities and their potential impact on development. It was in this context that in 2010, Cirad embarked on a reflective learning process about its impacts. This process was based on three principles: (1) acquiring a better understanding of how innovation systems in Southern countries unfold, taking into account the contributions of all stakeholders; (2) building an approach based on ‘impact pathways’ to influence scientific and research partnership planning, and to organize collective experience-sharing and learning capacity by making underlying theories of change explicit; and (3) reviewing existing research practices to build more effective projects in partnership, and simultaneously formalizing the building of the culture of impact. Between 2010 and 2018, three main phases can be distinguished, each of which contributed to developing this culture of impact (Fig. 1). Figure 1. Open in new tabDownload slide The sequence of formally building an impact culture at Cirad from 2010 on. Figure 1. Open in new tabDownload slide The sequence of formally building an impact culture at Cirad from 2010 on. 3.2 The ‘reflection and prototyping phase’: 2010–14 From 2010 to 2014, a small inter-institutional task force from four French research organizations, including representatives of Cirad, and the French Development Agency, sets the process in motion with a critical literature review of existing impact evaluation methods (Saint-Martin et al. 2011). The working group concluded that, in the context of agricultural research in developing countries, attributing change to a specific action would be challenging. Challenges concerned the lack of data, institutional weaknesses in partner structures, the partnership modus operandi of Cirad, the constraints on evaluating research outputs, and the delays between research action and change. Additionally, the report remarked that standard evaluation methods tend to underestimate negative effects and are blind to unexpected effects that do not stem from the planned logic of an intervention. The working group recommended testing alternative approaches and to analyze the causal links between action and change, and the role of different actors, beyond linear measures of economic efficiency. Small-scale testing (four case studies) of a prototype ex post impact assessment method deemed appropriate to Cirad’s modus operandi followed. This phase also saw Cirad includes impact assessment as part of its 5-year strategic vision in 2012. 3.3 The ‘testing’ phase: 2014–16 Between 2014 and mid-2016, an impact task force was established to apply and validate a further iteration of the prototyped method, named ImpresS (Impacts of Research in the South) (Barret et al. 2018). ImpresS ex post is an iterative and participatory evaluation method that includes systematization, validation, data collection, analysis, and actor feedback. Figure 2 summarizes the five phases of ImpresS ex post. The method was applied to 13 case studies selected among 90 candidate case studies submitted by Cirad research units, and involved around 30 Cirad researchers and their partners in developing countries. Several publications stemmed from these assessments, including a peer-reviewed special issue (Gaunand, Temple and Trouche 2018). Figure 2. Open in new tabDownload slide The five phases of the ImpresS ex post method. Figure 2. Open in new tabDownload slide The five phases of the ImpresS ex post method. Lessons from the in-depth case studies underlined (1) the long term-nature and multiplicity of impacts to which research contributes (Triomphe et al. 2016; Hainzelin et al. 2017; Temple et al. 2018), (2) the importance of co-producing outputs and outcomes with a variety of actors to generate impacts (Temple et al. 2018), (3) the key role of all stakeholders’ capacities in innovation processes leading to impacts (Toillier et al. 2018), (4) the role of interactions with public actors and policy makers in design or scaling, which help generate outcomes (Dabat and Grandjean 2018), and (5) the multiple roles played by the research community in co-developing outcomes with other actors of the innovation process (Faure et al. 2018). 3.4 The ‘strengthening the culture of impact’ phase (mid-2016 to present) The wealth of lessons learned from the cross-analysis of ex post studies motivated the Cirad impact team and top management to question how research actions are planned and to equip researchers and their partners with a suitable approach to co-build plausible ex ante impact pathways in the design of complex interventions (Earl, Carden and Smutylo 2001; Douthwaite et al. 2007; Mayne 2015). A second wave of methodological development followed, resulting in the ImpresS ex ante approach (Blundo-Canto et al. 2018). The aim was to reinforce the culture of impact by developing guidelines, by strengthening researchers’ and support staff’s capacities, and by accompanying projects and program design. The approach, summarized in Fig. 3, is participatory and iterative, and is a flexible process that can be adapted to the needs of each project. Figure 3. Open in new tabDownload slide A stepwise iterative approach to build plausible impact pathways. Figure 3. Open in new tabDownload slide A stepwise iterative approach to build plausible impact pathways. ImpresS ex ante focuses on the importance of interactions between intervention partners and stakeholders to facilitate their appropriation of outputs, leading to changes in visions, perspectives, practices, and behaviors. The approach stresses the need to make explicit the strategies envisaged to overcome potential obstacles to appropriation. With ImpresS ex ante becoming a key element in building a culture of impact, new staff were recruited, and training sessions, seminars, and dissemination events were organized for researchers and support units to strengthen capacity and buy-in throughout the institution. For instance, an ad hoc 3-day training session engaging all members of project management support units was organized to integrate ex ante reflection into their practices. The design of this module revealed other training needs focused on more specific tools for project design and management, and the integration of ImpresS ex ante concepts in the ‘project leader training’ sessions periodically organized for Cirad researchers. Born from the lessons of the ex post case studies and the evaluation literature, ImpresS ex ante played a key role in strengthening the culture of impact by structuring the design of projects and programs around plausible impact pathways emerging from collective reflection and shared visions. Finally, during this 8-year period, multiple interactions between the impact team and Cirad’s higher management made it possible to align methodological development and learning from impact assessment with the organization’s strategy. Four ‘culture of impact’ seminars were organized during the annual meetings, when most Cirad researchers come together for a week, to disseminate and discuss methodological development and findings. Three lectures on the culture of impact were given involving international discussants and key funding bodies, enabling external dissemination and lobbying. Additionally, members of the impact team took part in international workshops to get feedback from experts in impact assessment as well as from the agricultural research community. To date, 14 project design workshops that include an ImpresS ex ante component have been conducted in different countries (France, Senegal, Costa Rica, Burkina Faso, Madagascar, Tunisia, and Peru), involving 14 different projects of different scope, scale, and topics, and more than 100 researchers and representatives from partner institutions. 4. Discussion: First lessons in building a culture of impact in agricultural research organizations As the experience we describe in this article shows, building a culture of impact in a research organization implies assimilating broader organizational culture. This involves nested changes, from broader and conceptual changes to concrete strategic, structural, professional, and operational actions. Eight years into this process at Cirad, notable progress has been made, but internal resistance and new challenges have emerged. For a process to be truly transformative, it needs be sustained over time and to overcome internal resistance in the form of defensive behaviors against changes that may be perceived as threatening (Argyris 2004). In Cirad’s experience, to foster long-term sustainability and to reduce and overcome internal tensions, doting the institution with adequate resources to establish dialogue, but also evaluating the changes in behavior and in practices that this culture of impact entails, remains essential. It is important to state that adhering to a culture of impact does not mean distorting the core business of most researchers (i.e. producing new knowledge and research outputs). Rather, it implies enriching their thinking, practices, and interactions, through methods, approaches, and tools to choose strategies that plausibly increase the appropriation of research outputs by stakeholders. The transformative process of building an impact culture at Cirad was not linear. Stemming from the original intention of using impact evaluation to understand how research contributes to generate societal impacts, acknowledging the long trajectory of Cirad’s own contribution, it evolved into applying this learning to build and equip researcher’s practices and interactions with other stakeholders. Even if this process is still in its early stages, the lessons already learned provide valuable clues on how to integrate a culture of impact in agricultural research for development organizations. The institutional capacity to move toward this culture of impact can enhance the performance and societal contribution of these research organizations. Integrating the culture of impact also increases researchers’ and managers’ ability to learn from the organization’s functioning, and to use this learning to evolve and adapt to external and internal factors of change. In the following section, we present six key lessons we believe are valid beyond Cirad’s own experience. 4.1 Fostering transdisciplinary understanding for the evaluation of complex problems Complexity theory acknowledges that changes do not occur linearly in dynamic, transformational complex systems, in which multiple actors interact (Byrne 1998). This challenges the notion of narrowly measuring predetermined outcomes following a linear cause–effect logic (Patton 2006). Complexity science is increasingly promoted for the analysis of dynamic and adaptive agricultural innovation systems (Ekboir 2003; Patton and Horton 2009; Douthwaite and Gummert 2010; Mayne 2012), but the challenges of evaluating a diversity of impacts at different scales (Joly et al. 2015) and beyond science and economic impacts (Joly et al. 2016; Gaunand et al. 2015) need to be taken into account. Open debates about the role of scientists in contributing to societal impacts and in assessing the impacts of the innovation processes to which they contribute are needed (Temple et al. 2018). As a corollary, innovative approaches to evaluate researchers, programs, and teams taking into account transdisciplinary and social impacts (Belcher et al. 2016) should be devised. Additionally, the approaches and methods to evaluate impacts in complex systems, where heterogeneous effects occur in multiple dimensions, require cross-discipline points of view and interactions. Research and impact evaluation teams should consequently be equipped to solve complex problems through transdisciplinary interactions. When building a culture of impact, transdisciplinarity involves scientists from different disciplines and/or fulfilling a variety of roles and functions in innovation processes, as well as managers and support staff, but also research and development partners, in creating shared solutions (Klerkx et al. 2010) to societal problems. However, methodological and epistemological principles and practices differ, and the same terminology can express different concepts (Belcher et al. 2016). The development of a culture of impact therefore requires creating a common lexicon around impact, to allow diverse people to communicate and interact. As mentioned above, at Cirad, different types of events and arenas encouraged diversity of expressions and contributed to a shared vision of what a culture of impact is. As a corollary, the very use of the terminology of impact culture allowed the circulation of concepts formerly restricted to certain parts of the organization. Building relevance, credibility, and internal and external legitimacy (Hansson and Polk 2018) within this transdisciplinary dialogue and interactions was crucial. 4.2 Adaptability Favoring change processes should not be seen as an abrupt change in direction: changing attitudes and beliefs to change practices involves building on and adapting existing ones. Adaptability means (1) using potential levers and addressing barriers to appropriation of a culture of impact by managers, researchers, and their partners, and (2) adopting a participatory, adaptive, and iterative process in both methodological development and implementation. For instance, ImpresS ex ante was deliberately conceived to incorporate existing project design approaches and was non-normative in its application. While the approach provides a structured way of thinking about project design and planning, how it is actually implemented depends on the needs of each project team. Building a common language so that researchers from different disciplines and their partners can interact to create shared visions of change is a key step in this adaptability. Sharing concepts and language makes it possible to create and strengthen interest, buy-in, and appropriation. The seminars, guidelines, workshops, and ongoing support provided to research teams, all aim at creating this shared language and interest. At Cirad, the increasing number of researchers, teams, and support staff who approach the impact team for guidance and support is a sign of ongoing appropriation. The permanent two-way learning loop that is created through this support also enables continuous improvement in the ImpresS approach, increasing its relevance and adaptability to users’ needs and purposes. Of course, other factors also contribute to this interest and appropriation, such as increasing pressure from donors to prove the societal value of research, the internal motivation of research teams, researchers, and support staff, or emerging demands from partners and stakeholders. 4.3 Institutional buy-in Chapman (2002) synthesized how transformational change benefits from strong leadership, the ability to involve those affected by change as change agents, and external facilitators. As shown by Cirad’s experience, support and encouragement from upper management plays a crucial role in building a culture of impact. Leadership buy-in, also formalized in strategic documents, provides an enabling context for methodological development, integration in research and management practices, and capacity strengthening. In innovation systems thinking, one refers to the role of lobbying activities and interest groups to counter resistance and to increase the legitimacy of an innovation (Hekkert et al 2007). Change agents at the upper management level, who lead debate and actions to strengthen the culture of impact, together with brokering individuals, enable and support two-way communication at all levels of the institutional structure. This facilitates managers’ appropriation of results and proposals related to the culture of impact. This appropriation can occur through participation in methodological development, testing, and validation; targeted capacity strengthening in evaluation; and participation in case studies. Certainly, the long-term durability of the drive toward a culture of impact will depend on its widespread validation and appropriation by researchers, support staff, and managers at all levels, fostered by increased project effectiveness, but also by an improved ability to interact in transdisciplinary research settings, to which we refer above. 4.4 Capacity development and network building The process of building a culture of impact relies to a great extent on supporting researchers’ understanding of how research contributes to impacts and on the implementation of strategies that support the emergence of positive change. Such a strategy aims to ‘pro-actively create awareness amongst innovation network actors as regards their position vis-à-vis their institutional environment’ (Klerkx et al. 2010: 399). To put it in another way, it makes it possible to design research actions in a participatory manner that takes advantage of collective intelligence to increase their success. However, this often requires new skills and new ways of thinking through collective processes. It also highlights the different functions played by research teams with multiple competences at different stages of the impact pathway. Fostering a culture of impact may also be perceived as threatening by researchers whose schedules are already stretched to breaking point and who have multiple demands on their time. Providing support structures is therefore indispensable: this includes capacity strengthening, making resource persons available to accompany them, along with access to relevant in-house guidelines. Providing training in different formats that are appropriate for researchers, their partners, and support staff plays a key role. Another approach is creating extensive network of coaches and resource persons. These actions create legitimacy for the culture of impact and aim to overcome typical resistance to change. Additionally, new types of professionals are needed who have the capacity to build bridges between researchers and managers, support staff, donors, and other stakeholders. This new professionalism is advocated for agricultural research (Douthwaite et al. 2017). Key resource persons inside or outside the organization, who are willing and who are able to play this bridging role at the interface of science, management, and participatory methods of project development should be identified, as they play a key role in scaling the culture of impact. The complementary skills this new professional requires include a combination of planning, monitoring and evaluation skills with systems and complexity thinking; the capacity to accompany researchers in these processes; and the ability to work with different stakeholders, including researchers from the natural, political, and social sciences and their partners. 4.5 Financial mechanisms Developing and strengthening a culture of impact requires significant funding. In practical terms, at the project planning level, this should include a budget for partners’ and stakeholders’ workshops to iteratively build a shared impact pathway and strategy. At the research organization level, management can earmark funds for the construction of large-scale projects ex ante; for the implementation of ex post evaluations; for the functioning of methodological groups; for capacity development; for network building; or for new staff in impact assessment research and support activities. At the level of funding bodies, a key challenge is to ensure research funding supports longer project cycles and takes into account the fact that impacts emerge in the long term, usually through clusters of projects, rather than as the result of the work accomplished in one project alone. Funding bodies can support this in several ways: by favoring bridging and continuity between short-term project cycles and incentivizing interactions between clusters of projects; by promoting monitoring and evaluation frameworks that allow for transparent adaptive management instead of rigid reporting mechanisms; or by financing longer inception phases during which context-relevant and more effective strategies can be defined with local stakeholders. This could include providing a budget to develop and/or validate impact pathways at the end of the first year of a project, after a thorough appraisal and partnership building. This also requires a fair amount of lobbying of and communication with funding bodies by the research community. 4.6 External and internal communication and partnerships Understanding and learning from the different functions played by research, and the contributions it can make toward achieving impacts, should become an integral part of individual, collective, and institutional reflection. In Cirad’s experience, adequate internal communication during organization-wide events has been important to support the emergence of a transdisciplinary understanding. In addition, external and scientific communication about the culture of impact increases its legitimacy in the eyes of other stakeholders, particularly funders. These communication channels include international conferences, events for funding bodies, and participation in global platforms on agricultural research, but also research partnerships with other research organizations to share experience in and lessons learned from impact generation and the role of research. Ongoing actions at Cirad support long-term strengthening of the culture of impact through capacity strengthening, communication, monitoring and evaluation, and research on organizational learning and change in partnership with other research organizations. 5. Conclusion This article takes stock of the first lessons learned concerning the processes and approaches to build a culture of impact in agricultural research for development organizations, based on the experience gained by Cirad. A transformational process was set in motion by the need to increase knowledge on how research contributes to societal impacts and to identify appropriate evaluation methods for complex interventions. This started with the development, testing, and application of a participatory ex post evaluation method (ImpresS). The results achieved and the lessons learned from the application of ImpresS ex post generated new knowledge on impact generation processes and revealed the need to better equip researchers to plan and to act for impact in an ex ante fashion. To this end, an ad hoc approach for reflective, participatory, and structured project design and planning was developed. Building a culture of impact in a research organization implies individual, collective, and institutional understanding of how impact generation processes work in complex systems. To thrive, the culture of impact should permeate all levels of the organization, from top management to researchers and project support staff. A culture of impact means shifting from an exclusive focus on research outputs to a wider focus on how to encourage their appropriation, by identifying strategies and interactions that foster perception, practice, and behavioral changes. However, one of the main challenges is building awareness and acknowledgment of the contribution of research to societal impacts among all the organizations’ members, while respecting the diversity of profiles and approaches, and the wide range of outputs that are produced, as not everything needs to generate impact. It also involves accepting that there is not only one way to achieve and to evaluate the Research-Impact link, even if an institution promotes a certain approach. In this sense, by acknowledging the multiple roles the research community plays in interaction with other actors, a culture of impact can help overcome the dichotomy between research and development. To do so, reflection on impact should be embedded in the research process, that is, be an integral part of it, rather than a separate task. As we have shown, building a culture of impact implies a conceptualization of impact culture embedded in organizational culture and collective experience, supported by transdisciplinary dialog to share an understanding of impact culture. This requires adaptability in building on existing practices, along with institutional support from leading change agents and recognition of the impact culture through formalization in the organization’s strategy. Appropriation of the culture of impact relies on shared language, acknowledgment, and understanding of the role of the research community in contributing to societal impacts. It also entails adequately equipping researchers and their partners and strengthening their capacity to support collective processes in research design, implementation, and evaluation. Finally, this requires adequate funding at the project, institutional and funding bodies level, and targeted communication to ensure buy-in by internal and external change agents. Ultimately, a culture of impact is a reflective culture oriented toward actionable organizational learning: understanding impact generation and the contribution of research is part of a long-term dynamics that aims to bring agricultural research for development closer to societal needs. Acknowledgements The authors would like to thank Boru Douthwaite for his insightful recommendations in the development of the ImpresS approach. The authors would also like to thank all the researchers, their partners, and the support staff who participated and continue to participate in building a culture of impact at Cirad. References Argyris C. ( 2004 ) Double-Loop Learning and Organizational Change, in Boonstra J. J. (ed.) Dynamics of Organizational Change and Learning , pp. 389 – 402 . Hoboken, NJ : John Wiley & Sons Ltd . https://doi.org/10.1002/9780470753408.ch19 Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Barret D. et al. ( 2018 ) ImpresS Ex Post. Methodological Guide to Ex Post Evaluation of Agricultural Research in Developing Countries . France : CIRAD, Montpellier . https://doi.org/10.19182/agritrop/00006 Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Belcher B. M. et al. ( 2016 ) ‘ Defining and Assessing Research Quality in a Transdisciplinary Context ’, Research Evaluation , 25 : 1 – 17 . Google Scholar Crossref Search ADS WorldCat Blundo-Canto G. et al. ( 2018 ) ImpresS Ex Ante. An Approach for Building Ex Ante Impact Pathways . France : CIRAD, Montpellier . https://doi.org/10.19182/agritrop/00013 Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Byrne D. S. ( 1998 ) Complexity Theory and the Social Sciences: An Introduction . New York, NY : Routledge, London . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Chapman J. A. ( 2002 ) ‘ A Framework for Transformational Change in Organisations ’, Leadership and Organization Development Journal , 23 : 16 – 25 . Google Scholar Crossref Search ADS WorldCat CIRAD DG-DRS. ( 2012 ) Strategic Vision 2012-2022. Making Research a Real Tool for Development , pp. 6 . Montpellier, France : Cirad . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Coe R. , Sinclair F., Barrios E. ( 2014 ) ‘ Scaling up Agroforestry Requires Research ‘In’ Rather than ‘For’ Development ’, Current Opinion in Environmental Sustainability , 6 : 73 – 7 . Google Scholar Crossref Search ADS WorldCat Dabat M. H. , Grandjean A. ( 2018 ) ‘ Comment Les Acteurs Publics Contribuent à L’impact de la Recherche Dans Les Pays en Développement ’, Cahiers Agricultures , 27 . https://doi.org/10.1051/cagri/2018003 Google Scholar OpenURL Placeholder Text WorldCat de Janvry A. , Dunstan A., Sadoulet E. ( 2011 ) Recent Advances in Impact Analysis Methods for Ex-Post Impact Assessments of Agricultural Technology: Options for the CGIAR . Rome, Italy : ISPCC . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Douthwaite B. et al. ( 2007 ) ‘ Participatory Impact Pathways Analysis: A Practical Application of Program Theory in Research-for-Development ’, Canadian Journal of Program Evaluation , 22 : 127 – 59 . Google Scholar OpenURL Placeholder Text WorldCat Douthwaite B. , Gummert M. ( 2010 ) ‘ Learning Selection Revisited: How Can Agricultural Researchers Make a Difference? ’, Agricultural Systems , 103 : 245 – 55 . https://doi.org/10.1016/j.agsy.2010.01.005 Google Scholar Crossref Search ADS WorldCat Douthwaite B. et al. ( 2017 ) ‘ A New Professionalism for Agricultural Research for Development ’, International Journal of Agricultural Sustainability , 15 / 3 : 238 – 52 . Google Scholar Crossref Search ADS WorldCat Douthwaite B. et al. ( 2003 ) ‘ Impact Pathway Evaluation: An Approach for Achieving and Attributing Impact in Complex Systems ’, Agricultural Systems , 78 : 243 – 65 . Google Scholar Crossref Search ADS WorldCat Earl S. , Carden F., Smutylo T. ( 2001 ) Outcome Mapping: Building Learning and Reflection into Development Programs . Ottawa : International Development Research Centre . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Ekboir J. ( 2003 ) ‘ Why Impact Analysis Should Not Be Used for Research Evaluation and What the Alternatives Are ’, Agricultural Systems , 78 : 166 – 84 . Google Scholar Crossref Search ADS WorldCat Ekboir J. M. et al. ( 2009 ). Successful Organizational Learning in the Management of Agricultural Research and Innovation: The Mexican Produce Foundations, IFPRI Research Report. Washington, DC : IFPRI . https://doi.org/10.2499/9780896291713RR162 Faure G. et al. ( 2018 ) ‘ How Different Agricultural Research Models Contribute to Impacts: Evidence from 13 Case Studies in Developing Countries ’, Agricultural Systems , 165 : 128 – 36 . Google Scholar Crossref Search ADS WorldCat Faure G. et al. ( 2010 ) Innover avec les acteurs du monde rural. La recherche-action en partenariat, Agricultures tropicales en poche . Versailles : Editions Quæ . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Gaunand A. et al. ( 2015 ) ‘ How Does Public Agricultural Research Impact Society? A Characterization of Various Patterns ’, Research Policy , 44 : 849 – 61 . Google Scholar Crossref Search ADS WorldCat Gaunand A. , Temple L., Trouche G. ( 2018 ) ‘ Évaluer Les Impacts Des Recherches en Agriculture Sur la Société: Outils, Méthodes, Études de Cas ’, Cahiers Agricultures , 27 : 3400. https://www.cahiersagricultures.fr/component/toc/?task=topic&id=872 Google Scholar OpenURL Placeholder Text WorldCat Hainzelin E. , Barret D., Faure G. ( 2016 ) Agriculture Research in Developing Countries: From a “Culture of Promise” to a “Culture of Impact ”. France : CIRAD, Montpellier . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hainzelin E. et al. ( 2017 ). Agricultural Research in the Global South: Steering Research beyond Impact Promises . France : CIRAD, Montpellier . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hazell P. ( 2008 ). An Assessment of the Impact of Agricultural Research in South Asia since the Green Revolution . Rome, Italy : Science Council Secretariat . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Hansson S. , Polk M. ( 2018 ) ‘ Assessing the Impact of Transdisciplinary Research: The Usefulness of Relevance, Credibility, and Legitimacy for Understanding the Link between Process and Impact ’, Research Evaluation , 27 : 132 – 44 . Google Scholar Crossref Search ADS WorldCat Hekkert M. P. et al. ( 2007 ) ‘ Functions of Innovation Systems: A New Approach for Analysing Technological Change ’, Technological Forecasting and Social Change , 74 : 413 – 32 . Google Scholar Crossref Search ADS WorldCat Horton D. (ed.), ( 2003 ) Evaluating Capacity Development: Experiences from Research and Development Organizations around the World . The Hague; Ottawa; Wageningen : ISNAR, CTA, IDRC . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Joly P.-B. et al. ( 2015 ) ‘ ASIRPA: A Comprehensive Theory-Based Approach to Assessing the Societal Impacts of a Research Organization ’, Research Evaluation , 24 : 440 – 53 . Google Scholar Crossref Search ADS WorldCat Joly P.-B. et al. ( 2016 ) Agricultural Research Impact Assessment: Issues, Methods and Challenges . OECD Food, Agriculture and Fisheries Papers, No. 98, Paris: OECD Publishing. http://dx.doi.org/10.1787/5339e165-en Joly P.-B. , Matt M. ( 2017 ) ‘ Towards a New Generation of Research Impact Assessment Approaches’ , The Journal of Technology Transfer . https://doi.org/10.1007/s10961-017-9601-0 Google Scholar OpenURL Placeholder Text WorldCat Klerkx L. , Aarts N., Leeuwis C. ( 2010 ) ‘ Adaptive Management in Agricultural Innovation Systems: The Interaction between Innovation Networks and Their Environment ’, Agricultural Systems , 103 : 390 – 400 . Google Scholar Crossref Search ADS WorldCat Leeuwis C. , Klerkx L., Schut M. ( 2018 ) ‘ Reforming the Research Policy and Impact Culture in the CGIAR: Integrating Science and Systemic Capacity Development ’, Global Food Security , 16 : 17 – 21 . Google Scholar Crossref Search ADS WorldCat Matt M. et al. ( 2017 ) ‘ Opening the Black Box of Impact–Ideal-Type Impact Pathways in a Public Agricultural Research Organization ’, Research Policy , 46 : 207 – 18 . Google Scholar Crossref Search ADS WorldCat Mayne J. ( 2012 ) ‘ Contribution Analysis: Coming of Age? ’, Evaluation , 18 : 270 – 80 . Google Scholar Crossref Search ADS WorldCat Mayne J. ( 2015 ) ‘ Useful Theory of Change Models ’, Canadian Journal of Program Evaluation , 30 : 119 – 42 . https://doi.org/10.3138/cjpe.230. Google Scholar Crossref Search ADS WorldCat Midmore P. ( 2017 ) ‘ The Science of Impact and the Impact of Agricultural Science ’, Journal of Agricultural Economics , 68 : 611 – 31 . Google Scholar Crossref Search ADS WorldCat Mintzberg H. , Westley F. ( 1992 ) ‘ Cycles of Organizational Change ’, Strategic Management Journal , 13 : 39 – 59 . Google Scholar Crossref Search ADS WorldCat Patton M. Q. ( 2006 ) ‘ Evaluation for the Way we Work ’, Nonprofit Quaterly Spring , 28 : 33 . Google Scholar OpenURL Placeholder Text WorldCat Patton M. Q. , Horton D. ( 2009 ) Utilization-Focused Evaluation for Agricultural Innovation . Institutional Learning and Change (ILAC) Initiative Brief No. 22. Bioversity, Rome: ILAC. Penfield T. et al. ( 2014 ) ‘ Assessment, Evaluations, and Definitions of Research Impact: A Review ’, Research Evaluation , 23 / 1 : 21 – 32 . Google Scholar Crossref Search ADS WorldCat Riley B. L. et al. ( 2018 ) ‘ Using Contribution Analysis to Evaluate the Impacts of Research on Policy: Getting to ‘Good Enough ’, Research Evaluation , 27 / 1 : 16 – 27 . Google Scholar Crossref Search ADS WorldCat Saint-Martin G. et al. ( 2011 ) Evaluation de L’impact de la Recherche au Cirad: Rapport du Groupe de Travail . France : CIRAD, Montpellier . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Schein E. H. ( 1988 ) ‘Organizational Culture: What It Is and How to Change It’, in Human Resource Management in International Firms , pp. 56 – 82 . London : Palgrave Macmillan . https://doi.org/10.1007/978-1-349-11255-5_4 Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Temple L. et al. ( 2018 ) ‘ Assessing Impacts of Agricultural Research for Development: A Systemic Model Focusing on Outcomes ’, Research Evaluation , 27 : 157 – 70 . Google Scholar Crossref Search ADS WorldCat Thornton P. et al. ( 2017 ) ‘ Responding to Global Change: A Theory of Change Approach to Making Agricultural Research for Development Outcome-Based ’, Agricultural Systems , 152 : 145 – 53 . Google Scholar Crossref Search ADS WorldCat Toillier A. et al. ( 2018 ) ‘ Comprendre la Contribution de la Recherche à L’innovation Collective Par L’exploration de Mécanismes de Renforcement de Capacité ’, Cahiers Agricultures , 27 : 15002. Google Scholar Crossref Search ADS WorldCat Triomphe B. et al. ( 2016 ) ‘ Mieux évaluer et accompagner l'innovation agricole en Afrique. Leçons d'une analyse transversale de 13 cas d'études ’, Cahiers Agricultures , 25 : 64003 , DOI: 10.1051/cagri/2016050. Google Scholar Crossref Search ADS WorldCat Walker T.S. , Alwang J. (eds), ( 2015 ) Crop Improvement, Adoption, and Impact of Improved Varieties in Food Crops in Sub-Saharan Africa . Croydon, UK : CGIAR Consortium of International Agricultural Research Centers and CAB International . Google Scholar Google Preview OpenURL Placeholder Text WorldCat COPAC Watts J. et al. ( 2007 ) ‘ Institutional Learning and Change: An Introduction ’, ILAC Working Papers , 3 : 19. http://hdl.handle.net/10568/70607 Google Scholar OpenURL Placeholder Text WorldCat Weißhuhn P. , Helming K., Ferretti J. ( 2018 ) ‘ Research Impact Assessment in Agriculture—a Review of Approaches and Impact Areas ’, Research Evaluation , 27 : 36 – 42 . Google Scholar Crossref Search ADS WorldCat © The Author(s) 2018. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contactjournals.permissions@oup.com © The Author(s) 2018. Published by Oxford University Press.

Journal

Research EvaluationOxford University Press

Published: Apr 1, 2019

There are no references for this article.