At an Amazon fulfillment warehouse tucked away in small-town America, an employee whiles away the slow-ticking hours of his daily grind by building castles. His materials aren’t exactly contractor-standard; he stacks cardboard instead of concrete slabs and seals each square with packing tape, rather than cement. When he finally fills and places the last box on his stack, the monitor in front of him dings. He looks up to see a fully-built animated castle blinking on his screen. His “level” of work is, for now, complete.

Welcome to gamification, where even in spaces where we aren’t meant to be having fun, we’re playing for (sometimes questionable) wins.

Today, technology has given us the power to gamify our lives by painting “levels” of challenges and achievements into our daily routine. Rather than take a boring jog around the block, we launch an app that convinces us that if we don’t move faster, ever-more-audible zombies will catch up and eat our brains. We walk towards meeting animated fitness goals, maintain our diets for the social media badges, and compete for achievement points (and the inevitable corporate pizza party) on workplace leaderboards. As with the hypothetical Amazon employee above, a game can sometimes become a metaphor for work itself, with the player as its avatar. Gamification makes the boring parts of our day more interesting; it tricks us into liking tasks we might have otherwise dreaded.

The rationale behind all of this is deceptively simple: If we’re having fun, we’re more likely to be engaged. If we’re engaged, we’ll be more productive. Studies back this line of thinking up: researchers have found that gamification tends to effectively engage users, promote productivity, and boost achievements — although the degree of that efficacy can vary based on the context of the game and the interest of the users.

However, gamification comes with its own set of ethical traps. Today, organizations on the cutting edge of game-inspired innovation are finding new ways to use game mechanics in non-game settings in a way that influences not only how effective we are at a given task, but what we think and feel about completing it. These capabilities present something of a moral quandary to its developers and users. We have to ask: How much of this gamified psychological guidance is good for us — and when does it veer into outright manipulation or exploitation?

Saving lives, in-game and beyond

Let’s consider Embodied Labs’ VR-based game, We Are Alfred. The project, first conceptualized by students at the University of Chicago, places its user in the shoes of Alfred, a 74-year-old man suffering from macular degeneration and hearing loss. In the game, users walk through a series of scenes and gain a better understanding of how it feels to be elderly and experience physical disabilities. But We Are Alfred isn’t meant to be a novel experience. Instead, it was designed to give its target user base — medical students — greater empathy for and understanding of their clients’ experience. As Eric Swirsky, a faculty adviser on the project, said to HuffPost, “It’s not curing, it’s not curative, it’s not even treatment-oriented. It’s about comforting and understanding where the patient is so that you can be with him.”

The gamification behind the project isn’t meant to directly influence doctor productivity — but it is intended to have a long-term influence on mindset, and thus improve a medical student’s ability to connect with and effectively treat patients over time.

This goal — using a virtual environment to influence mindset and behavior even when the experience ends and the player returns to the real world — might sound like a stretch. However, it’s been done before, and with notable success to boot. Over a decade ago, developers at HopeLab launched the 2007 Re-Mission game, which encouraged children with cancer to actively fight against tumor cells. Structurally, it was a basic shooter game; in the real world, it led to a “significantly higher reliability” that the children who played it would take their medicine — and, by extension, have better health outcomes.

The accomplishments of both We Are Alfred and Re-Mission are laudable; they demonstrate that gamification can play a clear and productive role in improving both health care training and patient outcomes. However, their impact is limited to a highly select pool of users within the health industry; what form might psychologically-influential gamification take if it were more widespread across sectors? How would it seek to influence us, and would the result be positive?

As per usual, the product sample we’re looking for can be found on — or rather, at — Amazon.

Have fun … but get to work

While the fulfillment worker at the top of this piece was a hypothetical, the situation he illustrated is far from fictional.

In late May, The Washington Post reported that Amazon had rolled out game-centric updates at five fulfillment warehouses across the US and UK. Workers tasked with repetitive, often boring tasks such as packing boxes and moving items could play games that translated their progress into space, onto racetracks, or, as in the case above, building castles. The faster they work, the more they accomplish in-game. For Amazon, investing in the play screens is a means to both alleviate worker boredom and motivate employees to achieve more.

The effort to lessen boredom and boost morale isn’t exactly unneeded. Over the past few years, Amazon has garnered considerable press for its poor working conditions; in the five years between October 2013 and October 2018, emergency workers were called to address suicide attempts, suicidal thoughts, and dangerous mental breakdowns at least 189 times across 46 fulfillment centers. As one former employee in Florida commented for The Daily Beast, it’s “mentally taxing to do the same task super fast for 10-hour shifts, four or five days a week.”

By adding games to the mix, Amazon hoped to ward off the depression that seemed to haunt its fulfillment centers. It’s a choice that falls in line with the philosophy that underpins the very idea of gamification; to quote gamification evangelist Jane McGonigal in a well-circulated TedTalk, “Games that are done well create a feeling of urgent optimism, which is also the opposite of depression. It makes us feel like we are part of a social fabric, that we’re blissfully productive, and that we have achieved something bigger than ourselves with epic meaning.”

Of course, integrating games is significantly different from integrating games done well — and there’s more than a little doubt that Amazon’s CastleCrafter falls into the latter category. Given the company’s history of prioritizing efficiency to the point of limiting bathroom breaks, some may see the game as a stealthy way to push employees to achieve more and base their working worth on in-game achievement bars that the company can lift at any time. As gamification expert Gabe Zichermann puts the matter: “When [employers] want to generate more output, they can ratchet those levers. It’s like boiling a frog. It may be imperceptible to the user.”

Rather than achieving McGonigal’s “blissful productivity,” employees might find themselves falling behind and losing their self-worth. Their lack of engagement can lead, ironically, to the very problem Amazon fears most: lower efficiency.

But this problem isn’t one confined to Amazon. Scholars have repeatedly found that playfulness at work balances on the edge of a coin; on one side, greater engagement, and on the other, coercion. If an organization’s attempts to gamify tumbles into the latter, it can have a profoundly deleterious impact on the company’s culture. As researchers describe the matter in a literature review for the Journal of Business Research, “Gamification’s ability to invoke intrinsic motivation is an invitation to exploitative behavior […] This may result in turning the gamified aspect of work into a contested space through which power, conflict and resistance is ‘played’ out.”

As tech proponents and innovators, we stand at the tie of a moral high-line when it comes to psychologically-oriented gamification. We know from products like We Are Alfred and Re-Mission that it can have a positive impact — in the latter’s case, it may even help save lives. However, Amazon’s example also demonstrates that when bluntly implemented within an insensitive organizational culture, gamification can become little more than well-veneered manipulation.

Gamification holds too much promise to overlook; one poor execution shouldn’t poison the practice. However, businesses and organizations that expect to implement gamification practices with the ease of flicking the power button on a Nintendo Switch are sorely mistaken. Like any great video game, a truly effective “game” mechanic takes years of focus and work to execute well.

Chris Wang is co-founder and CEO of Thundercore.