Making Evaluations of Evidence-Based Models Culturally Appropriate

mother holding newborn baby on a comfy chairOver the past decade, there has been growing interest in using evidence-based practice models to improve outcomes for clients in various settings. One area where we see more application of evidence-based practice is home visiting programs, in which trained professionals (such as nurses, social workers, or paraprofessionals) provide information and support to women and families during pregnancy and throughout a child’s first years. Evaluation ensures that providers deliver the evidence-based model’s supports and services with fidelity. In our work evaluating home visitation programs in very diverse communities in San Diego, we are learning how important it is to make thoughtful modifications to our evaluation protocols to fit the specific populations being served.

As part of our long-standing work with First 5 San Diego, we lead a countywide evaluation of First 5 First Steps, a home visitation program based on the Healthy Families America evidence-based model. The Healthy Families America model aims to reduce child maltreatment, improve parent-child interactions, and increase school readiness through home visits and routine screenings and assessments, including program evaluation tools.

The First 5 First Steps program in San Diego is unique because it serves families in need who do not qualify for other home visiting services in the county. Specifically, First 5 First Steps families include:

  • Low-income families (below 200% Federal Poverty Level);
  • Pregnant and parenting teens (between ages 13-21);
  • Immigrant and refugee families, particularly from the Middle East and Africa;
  • Military families (active, guard, reserve, and veterans).

Given the diversity of program participants, it was important to develop an evaluation that could accurately capture outcomes for very different cultural populations. Aligning First 5 San Diego’s evaluation priorities while ensuring fidelity to the Healthy Families America Model prompted us to review the implementation science behind many of the selected evaluation tools. Home visitation evaluation tools, as with many standardized assessments, are often validated with populations that are less diverse than First 5 First Steps’ population. Therefore, our first task was to ensure the assessments made sense for the populations served by First 5 First Steps in San Diego, especially the immigrant and refugee communities. This review helped us identify two potential modifications:

Ensure that tools are culturally appropriate.

As we examined some of the questions and observation items on the standardized assessments and talked with home visitors, we realized that some of the questions might not make sense for some families or would be difficult to translate into their native languages. More importantly, we could draw culturally inappropriate conclusions if we relied on the assumptions of some of the questions on the assessments. Two examples illustrate these assumptions:

  • One of the standardized tools used to assess a family’s home environment places a positive value on having a pet in the home. Home visitors indicated that it is not customary to have pets in some cultures, and therefore we would not expect these families have pets.
  • This tool also has a question that asks if the “child has a special place for toys and treasures.” Home visitors pointed out that the word “treasures” does not translate well into some languages, nor is it customary in those cultures for children to have a special place to keep toys or “treasures.”

These important considerations emerged because of the feedback loop between our evaluation team and the home visitors who were providing the services and engaging with the families. Being sensitive to these issues helps home visitation staff provide a more accurate assessment of home environments for these families.

Minimize the burden of data collection.

Through this feedback loop, we also learned that the frequency of existing data collection often felt burdensome to families and home visitors. This data collection burden was compounded by the fact that many of these families might be asked to share personal information through other services they are receiving, such as public assistance. Families may also feel more guarded or protective about the information they are willing to share because they have more at stake. Given this information, our team worked with home visitors to adapt the data collection schedule and to remove questions that were not essential to Healthy Families America or the evaluation.

The value of program staff feedback

At Harder+Company, we pride ourselves on designing and implementing culturally-appropriate and inclusive evaluations. In addition to monthly evaluation meetings with First 5 San Diego, their partners, and the American Academy of Pediatrics-Chapter 3, we solicit feedback from home visiting staff and supervisors who work closely with families across the county. On a quarterly basis, First 5 First Steps staff help us understand how the program’s evaluation tools are working, what challenges home visitors are facing when using the tools, and how the families are responding to the questions they are being asked. Their input has been important to identify those questions that are challenging for some families, and develop ways to adapt the evaluation to be culturally appropriate. The conversations and feedback from home visitors, in particular, are critical to making this evaluation a success.