AI Yoga Learning Assistant

AI Yoga Learning Assistant

Tech Cat Dream

Computer Systems Technology - Software Development and Network Engineering

12/08/2020

The AI Yoga Learning Assistant is a web platform to learn yoga in a fun way using advanced ML pose detection technology. Made for anyone with a laptop and webcam, your yoga poses can be estimated with feedback provided in real-time. Your webcam video stream is displayed side-by-side with a lesson video.

The platform analyzes your webcam frames and detects 17 body joints in 2D space and compares it to the reference pose in time lesson video. Real-time feedback enables you to improve your posture and take your yoga practice to the next level.

The technologies we have chosen are: -PoseNet on TensorFlow.js. Tensorflow is a JavaScript library developed by Google that allows you to run machine learning models directly in the web browser. Posenet is a pre-trained model that detects the position of key body joints on a 2D graph. -React.js for our front-end interface -Node.js and Express.js for our backend API to process uploaded videos by pre-detect poses and store them -MongoDB for our NoSQL database to store video details.

At present, the features we have developed include the instructor's ability to upload new lesson videos. Our platform pre-detects poses on any custom lesson videos. While students follow the lesson video, their real-time body limb positions are scaled and compared on time with the lesson video. The lesson experience is also enhanced by face filter stickers, ability to add background music, receive performance scores, and review playback of your practice.

Future directions include gesture recognition to replace the in lesson menu controls, voice feedback to cure body adjustments, and multiplayer mode to detect multiple players.

We found this project interesting among all projects that were listed from industry partners. During the project planning phase, our client suggested to use Kinect (a motion sensing input device to detect poses), but came up with the idea to use a Machine Learning library that runs on web browsers (Tensorflow.js).

We divided tasks to each member (managed by the Project Manager) and had meetings every week to discuss all possible solutions for the project. We always tried to find new ideas and discuss on what would be the best solution to meet our client's requirements and ensure low cost (our client is a non-profit organization. Every week, we had meetings with our Capstone professor to show what we had done, then get their feedback and suggestions to improve the project.