In early March 2019, DAS Labs worked with a sign language professional associated with a prominent school for the deaf on a prototype that used machine learning models to identify and teach users how to sign specific items appearing in their phone's field of view. The project had real engineering challenges: storing sign language demonstration videos, deciding how large a general object detection model to run on mobile hardware. It was one of our more rewarding early computer vision collaborations.
More recently, Kaitlyn Bonomo from the College of New Jersey reported on a Computer Science professor and student doing something similar, but with the user flow reversed. Instead of teaching a user to sign, their project translates actual sign language into written English.
How MediaPipe Makes It Possible
With Google's MediaPipe, developers can harness sophisticated skeletal tracking that now extracts 3D feature points to estimate the position, pose, and orientation of a person's hand. The system attempts to identify which letter or signed expression is being made. The challenges increase significantly when interpreting a series of different expressions requiring more than a single static pose: non-static, sweeping hand and arm gestures that form complete words or phrases.
At DAS Labs, we see this kind of machine learning work as another example of what AI can offer to increase accessibility and communication between people who speak different languages or have specific disabilities.
Real-World Accessibility Applications
DAS Labs currently works with a venture-backed mapping company, GoodMaps, whose sole focus is helping those with disabilities navigate large indoor spaces like hospitals and shopping centers using AI-powered spatial computing.
In addition, DAS Labs built a proof of concept for visually impaired founder Aziz Zeidieh of RannMax. The system used iPhone's LIDAR capability and object classification to navigate through public places with object detection and obstacle avoidance, similar to how advanced models are used in autonomous driving vehicles.
Michael and his team took our idea for an app and turned it into a functional, accessible, and world-class proof of concept. We could have gone with any one of thousands of iOS app developers, but it was quality we wanted, and it was quality and 100% satisfaction we got when we chose to work with Dynamic Augmented Solutions.
DAS Labs brings augmented reality, computer vision, and machine learning project experience to provide innovators with meaningful solutions that work in real-world conditions, whether deployed on mobile phones, AR headsets, or other devices.
If you have a computer vision or AI accessibility project in mind, Book a 15-min call →