Evaluation Tasks

STEP 1: Obtain Resources

Conducting an evaluation involves allocating resources from the project budget for this task, hiring appropriate personnel, and budgeting staff time. If the evaluation is going to be conducted by an external evaluator, resources will be required for this individual or team. Additionally, if there is a fidelity component that will be conducted by an external team (e.g., through technical assistance from the Mental Health Commission of Canada), this will need to be arranged early on in the evaluation process.

Another resource to consider is data management. How will data be collected, stored, and accessed? This will be an important piece of the evaluation process. There is no “one size fits all” solution, but there are resources upon which you can draw. For instance, the Homelessness Partnering Strategy has developed a Homelessness Information System (HIFIS), which is a management information system. This system is particularly relevant for programs receiving federal Housing First funding, since the system allows them to report on certain outcome indicators that need to be measured. The case study on The Alex, a Calgary Housing First program, developed specifically tailored spreadsheets and information systems to make data management and decision making more efficient. Another issue that should be anticipated are the costs of participating borne by Housing First participants. Honoraria should be set aside for people with lived experience who participate in an evaluation.

STEP 2: Build a Culture of Evaluation

Building a culture of evaluation means helping your organization understand the importance of evaluation, and gradually building the capacity to do it well. A significant challenge here is building a commitment amongst staff and program leaders to examine their beliefs about how well the program is working and to create a community centred around learning. Another challenge is to understand that evaluation is about improving the program rather than making judgments about its worth. A key approach is to frame the evaluation as part of a culture of learning and continuous improvement.

Perhaps the most important challenge of building a culture of evaluation is to ensure that all stakeholders feel they are part of the process. This involves both planning and stakeholder engagement, so that all stakeholders understand the purpose of doing evaluation. Engaging stakeholders early is an important component of building a culture of evaluation. Stakeholder engagement will initially involve identifying stakeholders in your community who are involved in the implementation of your Housing First program.

Stakeholders might be broken into four broad groups:

• managers of implementation teams,
• direct service workers,
• persons with lived experience who participate in Housing First, and
• systems-level actors – these individuals are program sponsors and likely come from municipal or provincial government

Building a culture of evaluation also requires having some expertise and guidance to set things off in the right direction. Conducting an effective evaluation of a Housing First program will involve developing an evaluation plan and involving evaluators during the planning stage of program development. Having evaluation expertise early in the process will help you to tailor your evaluation approach to the stage of implementation.

In summary, it is important to build a culture of evaluation into your Housing First program to ensure that the evaluation is useful to a broad range of stakeholders who “buy in” to the process.

Another challenge is to understand that evaluation is about improving the program rather than making judgments about its worth.

STEP 3: Identify the Purpose of Evaluation and select Evaluation Components

STEP 5: Using Fidelity Indicators

The Pathways fidelity scale measures fidelity across five domains: (1) housing choice and structure; (2) separation of housing and services; (3) service philosophy; (4) service array; and (5) program structure. In At Home/Chez Soi, external teams performed site visits to conduct the fidelity assessments for each of the five project sites. Pathways to Housing has also developed a Housing First Self Assessment Survey for sites that wish to perform an internal fidelity assessment. The same five domains are covered as with the external fidelity measure. It is important to note that qualitative evaluation techniques are important in giving broader context and significance to quantitative fidelity measurements. This is an important consideration in planning this part of the evaluation.

A fidelity evaluation is a process that quantifies the degree to which implementation corresponds to the Housing First program model. Pathways to Housing in New York has developed a fidelity measure that was used in the At Home/Chez Soi project by an external team.

In developing a fidelity assessment process, the first task is to select one of the two measures of the key program domains noted above. While some sites may choose to have fidelity evaluated by external teams — likely through technical assistance provided by the Mental Health Commission of Canada (MHCC) — many sites will choose to utilize an internally led fidelity process (or start with an external review, and then develop the capacity to measure this internally).

Two excellent resources for internal fidelity are the HPS fidelity self-assessment tool and the Housing First Self-Assessment Survey developed by Pathways.

STEP 6: Finding Income Indicators

The Homelessness Partnering Strategy has outlined outcome indicators for measuring Housing First at the level of individuals. Outcome measures are broken into three categories:

• Housing First placement indicators;
• Self sufficiency indicators; and
• Prevention indicators.

Another useful set of outcome measures is the Ontario Ministry of Health and Long Term Care’s Community Mental Health Common Data Set (CDS). A final source of outcome indicators is the At Home/Chez Soi project, which used several outcome indicators that should be of interest to groups wanting to evaluate Housing First programs.

There are two components that use the measures to ascertain change. First, there should be a process of taking baseline measurements. Secondly, there should be a set of outcome measurements that show the impacts of the Housing First program on participants. The CDS was intended for the evaluation of outcomes across mental health programs in Ontario. The CDS is comprised of mandatory elements (p. 10) and optional elements (p. 11). The mandatory elements contain useful administrative, demographic, and clinical categories that capture basic information in each of these areas. The optional elements contain an “outcome” category that should be useful to individuals wanting to evaluate a Housing First program. In particular the outcomes category contains measures of baseline and current psychiatric hospitalizations, living arrangements, residence types, employment, education, and income. These measures are helpful in tracking life changes across time that are of importance in evaluating Housing First outcomes because they provide information pertinent to community integration and functioning, as opposed to more narrow measures like symptom severity.

The At Home/Chez Soi project included several outcome measures/instruments that received positive feedback from program staff. These measures included the Multnomah Community Ability Scale (MCAS) for community functioning; the Qoli-20 for quality of life; the Global Appraisal of Individual Needs (GAIN SPS); the Colorado Symptom Index (CSI); the EQ-5D for basic health status, and; the Service Satisfaction Scale (SSS-10). It should be noted that organizations can use these scales and compare against data from At Home/Chez Soi, including the treatment as usual group, for tracking.

STEP 7: Create an Evaluation Plan and Collect Data

A good strategy for keeping track of all of the evaluation tasks is to create a timeline or protocol that organizes data collection and makes clear which team members are responsible for specific tasks. An excellent worksheet to help guide the development of this protocol is available here.

STEP 8: Analyse the Data

Once data collection has been completed, data analysis should begin. There will likely be both qualitative and quantitative data to analyze that involves markedly different skills and might subsequently be assigned to different team members. Quantitative analysis will involve the use of statistical software to ascertain whether outcome measurement shows changes that are statistically significant. Qualitative ata will involve some form of “narrative analysis”, which can help understand whether and how any quantitative changes are significant to the participant outcomes. It might be helpful to use quantitative findings as a basis for doing exploratory qualitative analysis, particularly if the results are unclear or unexpected. It will be helpful to use a qualitative guidebook to guide the narrative analysis.

STEP 9: Develop an Reporting Process

It is important to consider the diverse audiences of evaluation data. Evaluation findings can be helpful in improving performance in clinical practice and might be communicated through staff coaching or training. Evaluation findings might also be useful in keeping community stakeholders current on the progress of the Housing First programs.

There are at least three stakeholder groups to consider that ideally should be involved in evaluation from the outset:

1. Service providers
2. Program leaders
3. Funders

Providing evaluation feedback to these groups is important for ensuring continual improvement and learning in your program.

Finally, evaluation results are a good way to engage landlords about the progress of the program and to develop relationships. Holding a breakfast or lunch in which results are shared with landlords can be an effective engagement strategy. Ideally, results should be synthesized into a one to three page document for this group.

Some concrete examples from At Home/Chez Soi might be helpful to illustrate the reporting process. One of the challenges of At Home/Chez Soi — as revealed through implementation evaluation — was the separation of housing and clinical service teams, and maintaining effective communication between them. This was an important finding for program leaders who needed to innovate new program structures that facilitated communication between these teams.