Our Groundbreaking Mixed Reality Facial Tracking Smart Glasses Application Is Poised To Reshape Modern Medicine and Technology

Our Groundbreaking Mixed Reality Facial Tracking Smart Glasses Application Is Poised To Reshape Modern Medicine and Technology

CLIENT

CONFIDENTIAL

DISCIPLINES USED

AR Development
Spatial Design
User Interface Design
User Experience Design

PROJECT

Our groundbreaking mixed reality facial tracking smart glasses application is poised to reshape modern medicine and technology.

The Background

A private client was looking to develop a facial tracking application with uses in medical and healthcare settings. The goal was to increase the accuracy of consultations and procedures, and provide a new standard of training for students and assistants. They tapped our team at CXR.Agency to explore the possibility of a mixed reality smart glasses application – something that had not been done before.

The Background

The Challenges

The Challenges

Mobile headset – This application needed to operate on a mobile processor for obvious reasons – the equipment needed to be readily available and easy to store in an office or classroom setting. Such a complex application had not operated solely on a mobile processor before, so it was unclear if this could be done with the computing power of a portable headset.
Precise facial mapping – The utility of the application depended on the ability to analyze a face in real time and overlay it with data. For this reason, the tracking needed to use the highest standards of facial tracking to analyze multiple points on the face simultaneously. Hololens 2 architecture does not allow for multi-threaded processing as the Hololens has shifted platforms to ARM64, but several of the key APIs – such as Camera frames – still work on ARM32.
Understandable interface – Tracking facial features was a challenge in itself, but the interface also needed to be easily understood and make sense for medical students and professionals who may not necessarily be accustomed to Augmented Reality interfaces. Our design team needed to keep this in mind as they created a user experience that relayed complex information in a straightforward design language.

SOLUTION

Engineered with Unity and C++, this revolutionary application can track up to 17 facial feature points at a time, passing camera frames through algorithms and machine learning to overlay a face with an AR user interface in real time.

SOLUTION

SERVICES / PROCESS

FULL TRAINING SUITE
The application includes a pre-loaded practice face as well, as a cadaver, for training purposes before attempting on an actual patient. This allows the user to become familiar not just with the treatment but the application itself.
CUSTOM TREATMENT PLANS
The application overlays the data in real time, pinpointing the exact location for procedures. Panels of information are also presented to assist the professional in providing all relevant details to the patient.
ONE APP, MANY USES
While created for injection-based procedures, the tracking and application in general can be reconfigured for operations as important as reconstruction and as common as plastic surgery.