Using a camera in smart glasses, it monitors your progress and offers proactive corrections, such as "You're almost done, just a few more slices on the right!".
In edge cases, the system’s visual feedback was partially correct but lacked specificity, such as identifying that some bacon slices were fully cooked without naming which ones.
While initial applications focus on complex tasks like cooking, the multimodal architecture is adaptable to fields such as hardware assembly and vocational training. By converting static video into active, bidirectional, and accessible guidance, Vid2Coach establishes a new benchmark for inclusive technology, as noted by its presentation at venues like ACM UIST . If you want to explore further, vid2coach top
learners. Developed as a research project, it uses smart glasses to monitor a user's progress in real-time and provide proactive, context-aware feedback. Core Technology & Impact
: Outperforms baseline vision-language models (VLMs) by grounding all generated data in temporal frame sequencing. Using a camera in smart glasses, it monitors
Have you ever tried following a complex how-to video while your hands are busy? For many, especially in the blind and low-vision community, traditional video tutorials can be a major hurdle.
Video coaching refers to software and systems that use video analysis, AI, and real-time feedback to help coaches, trainers, and individuals improve performance. Whether you’re a sports coach analyzing an athlete’s technique, a physical education teacher tracking student progress, or someone learning a new skill, video coaching platforms provide powerful tools to accelerate learning and refine technique. By converting static video into active, bidirectional, and
Vid2Coach represents a significant leap forward in AI visual assistance. Rather than simply describing a scene, it , allowing users to leverage their own skills while the AI fills in the visual gaps.
: Users can ask specific questions about the task, and the system responds with answers grounded in both the video knowledge and the user's current progress. Hands-Free Experience : Operates on commercially available smart glasses
Vid2Coach Top Features: Transforming How-To Videos into AI Task Assistants
The system links to commercial smart glasses, such as Meta smart glasses. The glasses’ front-facing camera captures the user's hand movements and object interactions hands-free. The AI monitors progress, tracking whether the current step is irrelevant , in-progress , or complete . Key Features and User Benefits