Introducing DISempowerment Evaluation: Lessons from Australia’s Hotel Quarantine System
By Sarah Mason
Two months ago, I stepped out onto a rainy Sydney sidewalk after spending two weeks inside a hotel room. Weighted down with months-worth of luggage, I took a breath of—somewhat—fresh air and reveled in the feeling of freedom.
This was day 14. The day I was released from hotel quarantine.
For those who aren’t familiar with the Australian hotel quarantine program—introduced in March 2020—all travelers arriving on Australian shores are required to quarantine in a government-managed hotel for 14+ days.
Not your mother’s hotel stay
Lest you think this experience is comparable to your most recent holiday jaunt, let me dispel that myth.
On arrival in Australia, travelers are ushered onto buses by law-enforcement and defense force personnel, checked into a hotel by state police and ordered to remain in their hotel rooms—usually with no access to fresh air—until a state health official deems they are not a risk of infection (minimum 14 days; up to 24 days). At my hotel, PPE-clad hotel staff silently delivered food* three times a day while hall monitors patrolled the corridors to ensure neither I or my quarantine compatriots attempted to flee.**
Clear your minds of room service—it does not exist! Forget a daily room clean—your room will not be cleaned at all. Vacuum cleaners are banned and the hotel provides no cleaning products. My hotel also removed all cushions from our couch, most pillows, cutlery—even the coffee machine. Unless you’re a famous tennis player or movie star, you have no chance of slipping into a plush hotel robe or getting cosy in those hotel slippers.
Let me be clear, hotel quarantine is far from the worst thing in the world, but it’s also no walk in the park. Early reports suggest that up to 25% of those who experience quarantine report long-term mental health issues, including symptoms of PTSD.
Hotel detention
Hotel quarantine has been regarded a form of detention by the NSW Ombudsman, the Queensland Human Rights Commission, the United Nations Subcommittee on the Prevention of Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment and the European Committee for the Prevention of Torture and Inhuman or Degrading Treatment or Punishment.
And despite well-established international law (specifically, the Nelson Mandela rule) requiring that any individual held in detention receive at least one hour of fresh air and exercise each day—to prevent the deleterious effects that lack of movement and fresh air can have on mental and physical health—if you are crossing international borders, your next Australian hotel stay will include no such luxury.
How is this relevant to evaluation?!
Feedback loops are an essential first step in any continuous improvement process. Without feedback, leaders and frontline staff are stuck on a metaphorical treadmill: going nowhere, unable to learn about program weaknesses or—perhaps worse—guessing at what might make things better.
This link between feedback and improvement is well understood by evaluators, business folk, sportspeople, educators, manufacturers—almost any field under the sun.
Collecting feedback on hotel quarantine
Unfortunately, this feedback-improvement link has not been embraced by those who manage the New South Wales hotel quarantine system. There are no post-quarantine feedback surveys, no QR codes cheerily encouraging you to “tell us what you like and don’t like!” and there are definitely no follow-up calls to collect in-depth data on longer-term impacts or effects on wellbeing.***
You might think it would be advisable to collect feedback from people who are detained in a room for 14 days with little-to-no control over their surroundings, food, healthcare and movement.
Given the potential for hotel quarantine to induce short- and long-term negative effects, you might also think those responsible for overseeing hotel quarantine would recognize the importance of collecting feedback so they can adapt those parts of the program that lead to the greatest distress.
Yet, based on my experience, Australia’s hotel quarantine system is structured to discourage and prevent efforts to provide feedback on how to improve that system.
Hello, Franz Kafka; welcome to quarantine!
While I began my time in quarantine as a regular human, I soon slipped into evaluator mode.
Each day, I asked the nurses who conduct COVID-19 symptom checks how I could provide feedback (I’m an evaluator, you see, I think it is important to provide feedback). Some of this feedback was minor (e.g., ‘a microwave is a small change that would make a world of difference’); some of it more substantive (e.g., ‘lack of fresh air is detrimental to my mental and physical wellbeing; for the wellbeing of future detainees, please re-think your choice of hotels’).
Each day, I was told there was no one from the NSW Department of Health I could speak to. There was no email address for me to send feedback to, and I definitely could not speak with a supervisor. On day 2, a harried nurse told me they’d find me an email address, but on days 3,4,5 and 6 it failed to materialize. I stopped asking on day 7, worn down by the pointlessness of my efforts.
All I could do, these nurses said, was make a complaint**** to the NSW Ombudsman, an independent legislative body that receives complaints about public-sector agencies.
Trouble is, the NSW Ombudsman can only respond to complaints if the complainant can show they have taken reasonable steps to resolve the issue with the agency they are complaining about. Except—whoops, just by accident—the agency I wished to complain about would not provide a means for me to share my concerns with them.
Even more curiously—as I found out after lodging a complaint—the NSW Ombudsman “does not have the power to investigate the quarantine arrangements” (quote from email communication with NSW Ombudsman). Knowingly or unknowingly, the hotel quarantine workers (not to mention the formal NSW Health documentation) were sending me down a proverbial feedback dead-end that no one could respond to.
For anyone who believes in the value of evaluation—or the importance of improving social programs—this is a major problem.
Empowerment evaluation
Many readers may be familiar with David Fetterman’s unique approach to evaluation. Fetterman and his colleagues (1994) describe empowerment evaluation as the use of evaluation concepts, techniques and findings to foster improvement and self-determination.
Empowerment evaluation seeks to empower program recipients and stakeholders through evaluation so that they have greater control over their own lives—and so there is greater potential for program success.
The dark side of the moon
The antithesis of empowerment evaluation is a system that actively discourages—even prevents—program stakeholders from providing feedback on their experiences. It uses the absence of evaluation mechanisms to undermine potential for improvement and self-determination.
This is the evaluative approach adopted by those overseeing the NSW hotel quarantine: that is, a complete absence of participant feedback mechanisms—combined with processes and personnel that discourage, prevent and frustrate users’ efforts to provide feedback. As a recent participant in a program using this approach, I can confirm it is the opposite of empowering.
Consequently, I am calling this approach to evaluation DISempowerment evaluation.
Whose voice matters?
At its best, evaluation can bring positive, often cathartic effects for those who take part. So many times, program participants have let me know how grateful they are to be able to share their experiences with someone who will listen and document their concerns (often, an evaluator is the only person participants can confidentially share their stories with).
Yet DISempowerment evaluation serves the opposite purpose, causing potential harm to program recipients by conveying the message that their voices don’t matter—and their experiences matter even less.
DISempowerment evaluation also ensures program improvement will not occur, because those on the receiving end of an intervention cannot provide feedback (and there’s likely no one who cares to listen, anyway).
In contrast to our field’s oft-stated goals of advancing social good, DISempowerment evaluation serves to undermine participant wellbeing by removing their voice, their agency and their dignity.
Our obligation as evaluators
As evaluators, we have a moral obligation to ensure there are avenues for feedback about social programs. This remains a moral obligation even—and perhaps especially—during times of crisis when extraordinary measures are put in place.
This is because the absence of evaluation can perpetuate, and even cause harm. When program participants are implicitly—or explicitly—told their experiences do not matter, this can have harmful consequences for mental and physical health. Such a risk is heightened when program participants do not take part in said programs voluntarily (like in cases of hotel quarantine, or other mandatory government programs), but are mandated to do so by a state or federal agency.
Our profession can contribute to advancing the social good—but only when those affected most by a program can share their feedback for the purposes of ongoing improvement and adaptation. Part of our role as evaluators is to ensure those voices are heard, even when the institutions that run those social programs—and perhaps, the public—do not wish to hear them.
*If you would like to imagine this food, imagine cardboard. Blended. Then boiled. Enjoy!
**Generally, not a good idea. Fleeing is punishable with fines of up to $11,000—or imprisonment.
*** Three weeks ago, my local post office sent me an email, asking for feedback about their service. This was after I’d had a five-minute call with an online representative about envelope sizes. The postal service: Interested in users’ experience. The department that was responsible for managing my health and wellbeing for 14+ days? Not so much. And before you say we’re in the midst of an emergency, remember that this system has been in place for over 14 months. That is ample time to create an online survey.
****Note the language: complaint, not feedback. An individual in hotel quarantine cannot provide feedback, only make complaints. This label further serves to discourage feedback (especially in a country with a cultural norm of compliance) because it conveys the view that anyone with feedback is a trouble maker, rather than someone offering suggestions for improvement.
