Augmented Reality (AR) overlays, Artificial Intelligence (AI) hand tracking and on-device computer vision are not new technologies. Pokemon Go, launched in 2016, caused a social media frenzy and put AR on the radar for millions of users. Technology has improved significantly and we are in an era of new AR devices launching frequently. Research exists and the underlying models are maturing. What has been missing so far is how this technology may look in clinical settings and practice. This is our early contribution to the Hand Therapy AR conversation.
AR-assisted finger goniometry and telehealth
The measurement of joint angles is a core clinical skill for Hand Therapists. Every assessment, outcome measure and discharge will involve taking range of movement measurements.
Conventionally it requires a physical instrument, known as a goniometer and an in-person appointment. The use of telehealth in Hand Therapy has increased significantly since the onset of the pandemic. For those who do not have specialist Hand Therapy in their area, telehealth has made it accessible.
We have been developing a proof-of-concept AR application in telehealth. This includes an overlay of real-time joint angle measurements onto hand joints via a camera. This can be a webcam, phone or tablet camera. The user holds their hand in front of the camera, then the application will automatically detect the finger joints and displays a measured angle on the selected joint in real time.
Note: the application supports three-axis measurement. In this demonstration only X and Y are active.
What is working: joint detection and angle calculation across the finger joints, real-time overlay, consistent and satisfactory performance under specific conditions.
What still needs work: accuracy under challenging conditions and user interface, which offers significant potential for both the clinician and patient.
The clinical implications are substantial. Objective remote range of movement measurements, outcome tracking with quantified progress and data that is accessible at the click of a button.
Current capability: real-time landmark detection in video sessions, frame-by-frame tracking of hand position and finger angles, viable performance on standard consumer hardware.
Development needed: integration with clinical documentation workflows, outcome measure mapping and validation in clinical populations and settings.
Google AI Edge hand landmarks in video Hand Therapy
Google AI Edge hand landmark detection tracks 21 anatomical points across the hand in real time. We have now integrated this into a video Hand Therapy proof-of-concept.
When applied to a live session, the landmark model provides the therapist with objective positional data, which includes joint coordinates, finger angles and movement range. This data is not visible to the naked eye. A visual reference has been applied for users.
For virtual Hand Therapy specifically, this addresses a genuine limitation: the gap between what a therapist can observe in a remote session and what they can assess in-person. It does not close that gap entirely, but it narrows it meaningfully while also offering potential to make sessions more interactive and engaging.
Why Hand Therapy?
Conditions affecting grip, mobility and fine-motor function are precisely the conditions that lend themselves to AR-based assessments. The clinical need and the available technology are well matched. The application layer is largely unbuilt, yet the foundations are accessible.
For developers and researchers
If you have found this through interest in Google AI Edge or AR development and are curious about clinical applications, the Hand Therapy use case is exciting. There are many possibilities for clinicians, researchers, students and patients.
“We are at an inflection point where the hardware, AI models and the clinical need are converging. We are sharing this work to showcase not only what is possible now, but to encourage others to move into this exciting space and help build what comes next.”
— Ms Michelle Razo, CHT - Belfast
Who is Ms Michelle Razo?
An experienced Occupational Therapist and Certified Hand Therapist with a demonstrated history of working internationally at leading trauma centres for the hand, wrist and upper limb. Her current practice is as a specialised Consultant Hand Therapist focusing on the management of complex cases, particularly those who have not been able to find solutions for their ailments.
Ms Michelle Razo additionally played a role in incorporating video Hand Therapy on a regional level in Northern Ireland and uses it in her clinical Hand Therapy practice.
Get in touch to connect.