How Peloton is using computer vision to strengthen workouts

As you do push-ups, squats or ab work, heft dumbbells, jump or stretch, a device on your TV follows you throughout your workout. 

You are tracked on your form, your completion of an exercise (or lack thereof); you receive recommendations on what cardio, bodyweight, strength training or yoga workout to do next; and you can work toward achievement badges. 

This is the next-level home fitness experience enabled by Peloton Guide, a camera-based, TV-mounted training device and system powered by computer vision, artificial intelligence (AI), advanced algorithms and synthetic data. 

Sanjay Nichani, leader of Peloton’s computer vision group, discussed the technology’s development — and ongoing enhancement — in a livestream this week at Transform 2022.

AI-driven motivation

Peloton Guide’s computer vision capability tracks members and recognizes their activity, giving them credit for completed movements, providing recommendations and real-time feedback. A “self mode” mechanism also allows users to pan and zoom their device to watch themselves on-screen and ensure they are exhibiting proper form. 

Nichani underscored the power of metric-driven accountability when it comes to fitness, saying that “insight and progress are very motivating.” 

Getting to the final Peloton Guide commercial product was an “iterative process,” he said. The initial goal of AI is to “bootstrap quickly” by sourcing small amounts of custom data and combining this with open-source data. 

Once a model is developed and deployed, detailed analysis, evaluation and telemetry are applied to improve the system continuously and make “focused enhancements,” said Nichani. 

The machine learning (ML) flywheel “all starts with data,” he said. Peloton developers used real data complemented by “a heavy dose of synthetic data,” crafting datasets using nomenclature specific to exercises and poses combined with appropriate reference materials. 

Development teams also applied pose estimation and matching, accuracy recognition models and optical flow, what Nichani called a “classic computer vision technique.” 

Diverse attributes affecting computer vision

One of the challenges of computer vision, Nichani said, is the “wide variety of attributes that have to be taken into account.” 

This includes the following: 

• Environmental attributes: background (walls, flooring, furniture, windows); lighting, shadows, reflections; other people or animals in the field of view; equipment being used. 

• Member attributes: gender, skin tone, body type, fitness level and clothing. 

• Geometric attributes: Camera-user placement; camera mounting height and tilt; member orientation and distance from the camera. 

Peloton developers performed extensive field-testing trials to allow for edge cases and incorporated a capability that “nudges” users if the camera can’t make them out due to any number of factors, said Nichani. 

The bias challenge

Fairness and inclusivity are both paramount to the process of developing AI models, said Nichani. 

The first step to mitigating bias in models is ensuring that data is diverse and has enough values across various attributes for training and testing, he said. 

Still, he noted, “a diverse dataset alone does not ensure unbiased systems. Bias tends to creep in, in deep learning models, even when the data is unbiased.” 

Through Peloton’s process, all sourced data is tagged with attributes. This allows models to measure performance over “different slices of attributes,” ensuring that no bias is observed in models before they are released into production, explained Nichani. 

If bias is uncovered, it’s addressed — and ideally corrected — through the flywheel process and deep dive analysis. Nichani said that Peloton developers observe an “equality of odds” fairness metric. 

That is, “for any particular label and attribute, a classifier predicts that label equally for all values of that attribute.” 

For example, in predicting whether a member is doing a crossbody curl, a squat, or a dumbbell swing, models were built to factor in attributes of body type (“underweight,” “average,” “overweight”) and skin tone based on the Fitzpatrick classification — which although is widely accepted for classifying skin tone, notably still has a few limitations

Still, any challenges are far outweighed by significant opportunities, Nichani said. AI has many implications in the home fitness realm — from personalization, to accountability, to convenience (voice-enabled commands, for example), to guidance, to overall engagement.

Providing insights and metrics help improve a user’s performance “and really push them to do more,” said Nichani. Peloton aims to provide personalized gaming experiences “so that you’re not looking at the clock when you’re exercising.”

Watch the full-length conversation from Transform 2022.