Summative evaluation of the Canadian Initiative for the Economic Diversification of Communities Reliant on Chrysotile

Purpose

The decline of the chrysotile asbestos industry in the Les Appalaches and Les Sources RCMs has had a major impact on the economies of these two communities, especially in terms of lost jobs. CED had initially begun helping the two RCMs address these challenges through its regular programs^Footnote3 before finally implementing a dedicated initiative when, following the closure of the last asbestos mine in 2012, the various levels of government announced targeted support for the industry. The Government of Quebec set up a $50M economic diversification fund in the Les Sources RCM, and the Government of Canada allocated $50M to CED to support the Les Sources and Les Appalaches RCMs in their economic transition.

Objective and term

The CEDICRC was in place from June 11, 2013, to March 31, 2020. It had a total budget of $50M to support the economic diversification of the Les Sources and Les Appalaches RCMs after the shutdown of the chrysotile industry.

The Initiative fell under the Strengthening Community Economies pillar, one of the three pillars of the Quebec Economic Development Program (QEDP). It had two components:

• Component 1 ($35M): Economic development projects (except public infrastructure projects) submitted by business and organizations; and
• Component 2 ($15M): Infrastructure projects

Intervention parameters

To be eligible, the projects had to take place in the Les Sources or Les Appalaches RCMs.

The authorized activities were those usually authorized under the Business development and Regional economic development pillars of the QEDP, although additional activities were authorized as part of an easing of the Initiative’s restrictions:

• Construction of a convention centre
• Fit-up (construction), expansion, modernization or conversion of unused industrial buildings into industrial motels
• Acquisition of existing buildings
• Tailings reclamation
• Broadband

The applicable assistance and stacking rates were the same as for the QEDP. CED also allowed for some easing of the funding conditions:

• A longer payment holiday (three years rather than two)
• A longer repayment period (seven years rather than five)

Snapshot of intervention

Mandate and scope

CED undertook to conduct a mid-term evaluation in 2016–2017, and to provide an update on the results in 2021–2022. The evaluation published in 2018, together with the current evaluation, complete CED’s commitment regarding the evaluation of the Initiative.

The evaluation covers the total amount of funding ($50.3 million) allocated under the Initiative from June 11, 2013, to March 31, 2020.

This exercise also aims to identify lessons learned from the implementation of this targeted initiative that could be useful to CED in situations involving initiatives implemented in similar contexts, i.e., initiatives that are geographically defined, have large budgets and involve the economic diversification of devitalized communities.

The current evaluation focuses on the three following evaluation issues:

Relevance

• To what extent did the Initiative meet the needs?

Effectiveness

• To what extent has the Initiative achieved the expected results? How sustainable will the results be? What are the impacts on the community?
• What impact has the implementation of the Initiative had on GBA+ target groups? Did the Initiative pose any accessibility barriers for individuals in the various target groups?

The issues addressed in this evaluation are intended to complement those addressed in the mid-term evaluation and have been prioritized in an effort to streamline evaluation resources. The issue of efficiency is being assessed as part of the current evaluation of the QEDP, which covers all the funding and initiatives delivered under that program.

Methodology

The evaluation is based on a document review; an analysis of administrative, program and socio-economic data; internal and external interviews with proponents and community stakeholders; and a survey of recipients.

Over the course of the Initiative, gender-based analysis plus (GBA+) became a government-wide priority applicable to all stages of a program’s life cycle. According to the Standards for Evaluation under the Policy on Results, GBA+ considerations must now be taken into account in the planning of evaluations. Although, for this Initiative, the number of proponents from target groups was limited, the evaluation took GBA+ considerations into account by targeting certain proponents in the survey and the interviews.

Methodological limitations

• Although the telephone survey included all of the Initiative’s funding recipients (except for the proponents interviewed), the number of survey respondents was low because of the limited number of proponents and the low survey response rate. This could affect the reliability of the results, especially when there are divergences of opinion. The interviews helped mitigate this limitation.
• Some of the indicators used to measure the Initiative’s performance can be considered more as outputs and therefore do not allow for an adequate assessment of the results generated by the funds invested. The qualitative data collection method (survey, interviews and document review) was used to mitigate this limitation.
• Since there was no economic diversification indicator identified when the Initiative was created to measure the response to diversification needs, the evaluation relies on economic indicators on employment and population, as well as on CED’s Economic Development Index (EDI)^Footnote5. This indicator has not been updated since 2016; however, the collection of qualitative data from CED advisors and external stakeholders, together with a review of internal economic analyses, has helped mitigate this limitation.
• For the period prior to 2018, the availability and reliability of project data on target groups in CED’s data systems is limited since this type of data was not systematically collected at that time. Over the past few years, CED has made a number of improvements to the quality of project data relating to target groups, notably by including a self-declaration form for recipients in 2018–2019 and adding definitions for the various target groups on the form in 2020–2021.
• The opinions and feedback stemming from this data collection process were obtained during the summer of 2021 and so may reflect the economic context and the respondents’ COVID-related experiences. The data collected served specifically to document the impact of the pandemic on the proponents, so as to isolate, insofar as possible, this effect on needs.

1. Findings pertaining to relevance

1.1 Initiative’s response to the needs of the regions

The purpose of this section is to determine whether the Initiative met the targeted needs in the two RCMs, and to identify the RCMs’ new needs.

Needs originally targeted

For several decades, the chrysotile industry was the main source of employment in the Les Sources and Les Appalaches RCMs. The industry started to decline in the 1980s in both RCMs, and the last mine closed for good in 2012. When the Initiative was implemented in 2013, the Les Sources and Les Appalaches RCMs were both facing a number of economic challenges, including a drop in the number of workers and lower aggregate income growth than in the rest of Quebec; however, the situation was not the same in the two RCMs.

According to the documentation and internal interviews, the Les Appalaches RCM was dealing with economic challenges, but was already gradually diversifying. Internal advisors noted that the support and the significant investments made in the decade leading up to the Initiative had already helped create an economic environment that was prepared to receive major funding such as that provided through the CDECTIC.

In the case of the Les Sources RCM, however, internal interviews and documentation point to a community that was facing numerous challenges and which, in 2013, had not succeeded in diversifying its activities. Various local stakeholders in the Les Sources RCM explained that, at the time the Initiative was launched, the community was in a position where it needed to build or rebuild its entire economic structure.

Extent to which the Initiative met the originally targeted needs

Intervention parameters

The parameters of the Initiative, in particular the eligibility of NPO and public infrastructure projects, were adequate in terms of meeting the various needs of the communities.

In the survey, 93% of respondents^Footnote6 said that the assistance had met their needs.

The interviews revealed that both the internal and external stakeholders agreed that the intervention parameters adequately supported the communities’ economic diversification approaches. It was also noted that the eligibility of NPO and public infrastructure projects was a key success factor that allowed for the funding of anchor projects in the communities. Most stakeholders said that without CED’s support, some municipalities would not have been able to make the necessary investments in public infrastructure to attract businesses and allow them to establish themselves in the communities in question.

A few NPO respondents stated that they had obtained 80% of their project funding through the Initiative, and that the required 20% was often difficult for their type of organization to come up with. A funding level of 90% would have made it easier for them to carry out forward-looking projects in their regions.

Term

Opinions were mixed as to the appropriateness of the Initiative’s term. Some stakeholders pointed out that economic diversification is a process that takes time, specifically noting that some of the projects receiving funding under the Initiative are large-scale and need time to be developed and implemented. With this in mind, several stakeholders indicated that the term of the Initiative was not optimal because it had put pressure on the proponents of some major projects to rapidly implement their projects to ensure the full use of the funding. However, according to CED respondents, the entire funding envelope was spent before the end of the Initiative, which demonstrates the capacity of the communities to carry out the projects within the allotted time frame.

Funding

CED provided a total of $50.3M in funding through the CEDICRC; of this amount, 82% was spent in the Les Appalaches RCM and 18% in the Les Sources RCM.

As shown by the total average annual investments^Footnote7, both prior to the Initiative and during its implementation^Footnote8, the CEDICRC resulted in an increase in CED investments in both RCMs.

A number of internal respondents pointed out in the interview that, to support economic diversification, it was useful to have had access to a significant amount of dedicated funding to finance riskier projects.

The majority of the external stakeholders interviewed felt that the funding available through the Initiative was insufficient to meet their community’s economic diversification needs, especially since they had not had access to other CED programs during the Initiative. CED respondents countered that many of the projects funded under the Initiative would not have obtained funding through regular programs because of different intervention priorities and competition among the projects submitted by the various regions served by the regional business office. Furthermore, from the standpoint of accountability and the of sound management of public funds, it ensured that the programs remained separate.

As concerns the breakdown of funding between the two RCMs, CED’s objective is to maximize the effectiveness of public funds by intervening in complementarity with the Government of Quebec^Footnote10. When interviewed, representatives of the Les Sources RCM said they found this decision to be unacceptable, and that it exacerbated the subsequent awarding of $50M in provincial funding to the Les Appalaches RCM in 2018.

Changing needs

During the period from 2016 to 2019, the economic indicators for the two RCMs generally pointed to positive growth, albeit anemic at times; on the other hand, between 2011 and 2016 a general downward trend was observed. Given that the Initiative was intended to support diversification to offset the loss of jobs resulting from end of the chrysotile asbestos industry, the fact that the number of workers ceased to decline and actually started to rise between 2016 and 2019 is a sign of positive economic development in line with the objectives of the Initiative.

However, there are still negative gaps when compared with Quebec as a whole, specifically as concerns the number of workers, average employment income and median age. And, even though the unemployment rates in both RCMs are lower than in Quebec as a whole, the participation and employment rates are also lower, which, respectively, point to a smaller working population and a smaller proportion of this working population. From 2016 to 2019, however, improvements to the data were observed in the Les Sources RCM.

According to the internal interviews, despite gaps in the economic indicators compared with those for Quebec as a whole, the Les Appalaches RCM now has a diversified industrial structure, a solid ecosystem and a very vibrant environment. The number of businesses in the RCM rose by over 50% between 2012 and 2019^Footnote11. When the CED Economic Development Index (EDI) was last updated in 2016, the Les Appalaches RCM came in 20th place among the 104 RCMs, up 23 places from 2011, and was no longer among the RCMs that CED considers to be “devitalized.” Many stakeholders in the RCM noted the community’s vitality, as witnessed by private investment projects involving the construction of a tourism site; the real-estate boom; and the growing popularity of the Cégep in the region.

In the case of the Les Sources RCM, internal respondents felt that diversification had been more difficult during the Initiative. Between 2012 and 2019, a 37%^Footnote12 increase in the number of businesses was reported in the RCM; however, CED’s internal economic analyses point to an industrial structure that remains devitalized and is made up primarily of small businesses. According to CED’s IDE, the RCM dropped from 78th to 89th place among the 104 RCMs between 2011 and 2016; according to the latest IDE update, it is now one of the 20 most devitalized RCMs in Quebec. Local stakeholders agree that the RCM has been unable to adequately diversify its economy to meet its needs; nevertheless, niches of economic diversification have been identified and businesses operating in various sectors have settled in the region in recent years and are helping the region move away from a single-industrial economy.

2. Findings pertaining to effectiveness

2.1 Achievement of the expected results

The purpose of this section is to determine the effectiveness of the Initiative from the viewpoint of the achievement of the expected results, other results generated by the Initiative, and satisfaction with the Initiative.

In response to a recommendation in the mid-term evaluation, a results chain was established for the Initiative in 2018, along with indicators and their targets. However, the targets were for the years 2018–2020 only, meaning that it was impossible to determine the extent to which the expected results were achieved over the entire term of the Initiative. To offset this situation, the evaluation relied on the collection of qualitative data to get a clearer picture of the overall results generated and the impact of the Initiative in the community.

Of the total funding provided by CED under this Initiative, almost 65% was used for the component targeting businesses and organizations, while approximately 36% went towards the Infrastructure component.

Businesses and organizations component

According to the performance measurement data, the results achieved are positive: the majority of the businesses supported (13/15) maintained or increased their annual sales, and all of them maintained or created jobs. Although the number of projects submitted by the two types of clients (SMEs and NPOs) was more or less the same, the majority of Component 1 funding went towards NPO projects^Footnote13.

Main activities of SME projects:

• Acquisition of equipment, IT tools and technologies
• Acquisition of capital assets

Main activities of NPO projects:

• Development or construction of facilities
• Strategy development (intelligence, specifications, studies, diagnoses or action plans)
• Business incubation

Examples of assistance provided under this component include funding for the completion of the King mine (KB3)  historical site facilities; the establishment of a materials innovation centre; and the acquisition of equipment for a technology transfer centre. Furthermore, it was through this component that CED was able to set up a $5M local investment fund with Capital expansion région Thetford (CERT) in the Les Appalaches RCM.

The documented results are consistent with the economic profile in the section on relevance, which shows an improvement in the economic situation in both RCMs in the later years of the Initiative’s implementation.

Finally, the interviewees and survey respondents were on the same page when it came to the sustainability of the Initiative’s results, supported in particular by the local investment fund in the Les Appalaches RCM. An investment fund was not proposed for the Les Sources RCM, something the various stakeholders in this RCM have criticized.

Infrastructure component

The performance measurement indicators for this component do not provide an indication of the results generated because they basically measure outputs produced by the funding rather than the results generated by the projects that received funding. The interviews provided a qualitative basis for the documentation of the results and impacts of this component.

CED’s investments under this component of the Initiative have helped fund public infrastructure projects that involve:

• Increasing wastewater treatment capacity
• Building or extending streets
• Constructing industrial buildings
• Redeveloping/expanding tourist facilities

The two natural gas service projects in the eligible RCMs, involving a total contribution of $7.7M from CED, were also funded through this component.

The interviews revealed that, in a number of cases, these projects made up the foundation of the RCMs’ economic diversification efforts by helping to keep businesses in the region or attract new ones, while also creating or maintaining numerous jobs, primarily in industrial parks. Investments in energy infrastructure were seen as being of particular importance in this regard. These views are consistent with an OECD report that confirms the importance of energy infrastructure for SMEs^Footnote15. Some good examples were brought up in the interviews, such as attracting the Canard du Lac Brome plant to the Les Sources RCM and keeping the St-Méthode bakery facilities in the Les Appalaches RCM.

Level of satisfaction

Satisfaction with the implementation of the Initiative was high among interviewees from the Les Appalaches RCM; however, community stakeholders in the Les Sources RCM expressed some dissatisfaction. In this regard, internal stakeholders noted that the breakdown of funding had resulted in more projects being rejected in the later years of the Initiative’s implementation, which could have contributed to the dissatisfaction with the Initiative.

Despite the fact that CED has an Office for Client Satisfaction, with contact information and a feedback form readily available online, stakeholders in the Les Sources RCM said they were unaware of CED’s mechanisms that would allow them to express their frustration or raise issues encountered during the implementation of the Initiative, and that this only increased their level of dissatisfaction. The internal document review found that the identification of CED’s feedback tool was improved when the CED website was revamped.

2.2 Gender-based analysis plus (GBA+)

The Auditor General of Canada’s 2015 report on the implementation of GBA+ noted that significant gender inequalities still existed, despite the Government’s commitment to eliminate them. In response to this report, the federal government renewed its commitment to GBA+ through its Action Plan on Gender-based Analysis (2016–2020)^Footnote16. To comply with the obligations stemming from this action plan, CED must take GBA+ considerations into account during the various phases of its programs, i.e., from their design and implementation to their evaluation. The Treasury Board of Canada Secretariat’s Standard on Evaluation (2016) reinforces this obligation to take GBA+ issues into account when conducting evaluations.

In line with these new obligations, CED adopted a statement of intent on GBA+ in 2018 to support the implementation of GBA+ across the organization. However, the Initiative was not subject to a gender-based analysis plus, nor were any specific performance indicators identified for target groups.

In 2018, as part of the authorities required to obtain funding for a pipeline project in the Les Appalaches RCM, CED carried out a more detailed GBA+, which concluded that the impact of the project would be equitable for both genders.

The surveys and interviews did not reveal any impacts on the target groups or barriers to the Initiative for these groups; this finding is limited, however, by the fact that potential target group proponents were not surveyed.