Have we forgotten about unintended side effects?

By Sarah Mason

Good ideas don’t necessarily make good policy.

Evaluators know this—we’ve talked about the need to measure unintended side effects for years.

PlayPumps, domestic violence and child hunger

Consider PlayPumps: an ingenious idea from South African engineer Ronnie Stuitver.

The idea. Harness an untapped resource (i.e., children playing on a roundabout) to meet a critical community need: clean drinking water. PlayPumps work through a merry-go-round like device that is connected to a water pump. As children play on their roundabout, PlayPumps pump clean drinking water into a storage tank where it can be used by members of the community at their convenience.

Children playing joyously on a roundabout? Clean drinking water? What could possibly go wrong?

Unfortunately, a lot.

The implementation. Turns out the PlayPumps needed to spin all day long to pump enough water for a community. One analysis by the Guardian showed that children would have to “play” continuously for 27 hours a day to meet the target of providing water to 2,500 people per pump. In situations like this, the line between child’s play and forced labor becomes very blurry—particularly when a child going to school interferes with their need to “play” on the PlayPump.

This is one example of a well-intentioned idea that was problematic in practice. There are many more to draw from.

Bamberger and colleagues, for example, have documented examples of projects intended to empower women that ultimately led to violence against those women as male partners became challenged by their empowerment. Other authors have reported on school feeding programs that indirectly incentivize parents to give their children less food.

As evaluators, we know that good intentions are not enough. Human societies are complex systems and it is difficult to predict how complex systems will react to change.

It is part of our job to question—and capture—current, delayed and distal side effects of new policies and programs. But—have we forgotten this during the time of COVID-19?*

Office of Management and Budget

The Office of Management and Budget’s recent efficiency analysis of the White House’s federal vaccine mandate suggests potential unintended consequences are not at the forefront of some evaluators’ minds.

According to the OMB, “the cost of implementing a vaccine mandate is largely limited to administrative costs associated with distributing information about the mandate and tracking employees’ vaccination status. Such costs are likely to be small… However, based on experiences shared by private companies detailed below, we expect few employees to quit because of the vaccine mandate, and side effects lead to significantly less sick leave than COVID-19 infection. And unlike COVID-19 infection, side effects are not contagious to other employees.”

Perhaps. But, thinking like an evaluator:

What could possibly go wrong?

Let’s consider a few possibilities:

• Mental health: What are the mental health consequences (and costs) for employees who experience distress as a result of the mandate? One 2021 study from the National Safety Council and the National Opinion Research Center showed that employees experiencing mental distress cost employers nearly $5,000 per person a year in work days lost—in addition to higher turnover costs ($5,733) and health care costs ($3,000) each year. How many more employees have experienced mental distress since the mandates were announced?

One 2016 experiment, published in the European Journal for Public Health showed that compulsory vaccination significantly increased anger levels among those who were hesitant about vaccines (incidentally it also decreased vaccination rates by 39% in later, voluntary rounds of vaccination). Anger has been consistently linked with increased anxiety, headaches, and high blood pressure. Leaving aside considerations of care for those individuals, these are all factors that can increase health costs for employers.

• Motivation & morale: How will the mandates impact worker motivation and morale? Empirical research consistently shows links between employee morale, motivation and productivity (e.g., “ostracism, incivility, harassment, and bullying have direct negative significant effects on job productivity”). A Gallup Poll from August 2021 reported that 38% of employees were opposed or strongly opposed to vaccine mandates. How many of these employees have been ostracized, harassed or bullied as part of the rollout of the mandate? One report from Seyfarth at Work from September 2021 suggests there have been an increasing number of workplace conflicts related to vaccination in the past months. How will those experiences affect productivity in the long-term?

• Long-term turnover: While the OMB considers immediate effects on turnover, do they factor in long-term turnover possibilities, i.e., individuals who comply in the short-term but seek out longer-term employment opportunities that are not covered by the mandates?

Would considering these potential side effects change the OMB’s ultimate conclusion? Would empirical answers to these questions outweigh the reported benefits?

Perhaps not.

Regardless, as evaluators we aren’t doing our jobs if we’re not asking these questions.

* https://collateralglobal.org is an excellent resource that asks questions about—and captures data on—the impact of COVID-19 responses.