Resonant Echoes
"Bridging Deaf Children to Quality Education Recovery"
Overview
“Bridging Deaf Children to Quality Education Recovery”
Resonant Echoes is an innovative augmented reality interface that helps teachers understand the intentions of language-impaired kids, facilitating better rehabilitation and social engagement for deaf students.
Deaf children often lack sufficient language skills during rehabilitation, which hinders their socialization due to difficulties in communication. The ‘Resonant Echoes’ helps therapists overcome understanding difficulties with an XR display and embedded AI, focusing on a role-play game scenario.
Role
Solo Project, Graduation Project
Duration
2023 Aug. – 2024 (5 months)
Director
Yingqing Xu, Head of Future Lab, Tsinghua
Category
Accessibility, Deaf education, XR, Headwear
Problem
In 2023, I volunteered at the China Deaf Center. There I encountered congenitally deaf children wearing cochlear implants and receiving speech training, which was a difficult process because of poor communication between deaf kids and therapists
Around 466M people globally suffer from hearing loss. WHO research indicates that a significant portion of these individuals could see notable improvements through hearing aids and proper rehabilitation treatments. Among therapists, 85% of my interviewees ranked ‘difficulty in understanding’ as the primary challenge.
According to WHO’s World Report on Hearing 2023, around 466 million people worldwide currently experience disabling hearing loss, expected to increase to 700 million by 2050. Without intervention, this can negatively impact various life areas.
Technologies like cochlear implants can significantly help children with severe hearing loss, particularly when paired with proper rehabilitation. Post-implantation, extensive rehabilitation and support services are crucial, with cognitive training playing a significant role.
An interview-based study was initiated at the China Deaf Rehabilitation Research Center, involving 14 rehabilitation therapists, conducted through face-to-face and telephone interviews.
Key findings:
1. 85% of interviewed therapists rank ‘difficulty in understanding what the children mean’ as the top challenge in rehabilitation teaching, far surpassing the second-ranked issue of ‘lack of attention.’
2. 90% of interviewees believe that difficulties in understanding significantly reduce the effectiveness of cognitive training. The primary reason is that challenges in sentence comprehension hinder the continuation of conversations.
3. 90% of interviewees expressed concern about the social integration of deaf children, highlighting it as the primary issue raised.
User Research : Shadowing
In order to grasp how therapists and deaf children communicate, their key challenges, and current solutions, I undertook both an expert interview survey and a shadowing investigation at China Deaf.
Among surveyed individuals, over 90% expressed concern about the social engagement of deaf children. To foster more interaction, they frequently resort to rephrasing or clarifying referents. While sometimes effective, these methods often fail due to difficulties in understanding sentences. Hence, difficulties in understanding children’s sentences are considered the biggest obstacle in cognitive training.
At the China Deaf’s affiliated preschool rehabilitation center, I carried out a shadowing research observing classes in both the higher and lower age brackets for a cumulative 30 hours. The main observations included language training, games, creative tasks and daily activities.
Key findings are listed below:
1. Therapists often use paraphrasing to increase responses to deaf children (i.e., restating the child’s intent). While effective, it is sometimes disrupted by a lack of understanding of the spoken content.
2. Another method to enhance responses is to ask the child to identify the object they are referring to and repeat their intent. This approach improves mutual understanding success but is more time-consuming for therapists.
3. Effective paraphrasing or responses can greatly enhance the sense of social participation for deaf children. Otherwise, conversations become too brief, reducing kids’ enthusiasm to communicate.
Prior Arts
Opportunity: Most deaf edu products focus on providing children with linguistic training / solo gaming experiences. Is there a high-tech way that allows more kids and a teacher to participate in a game together, achieving good mutual comprehension?
Inspiration :
Comprehensive Multi-player Gaming
Inspiration: role-play games and storytelling events are common activities at kindergartens, where it provides intense interactions between multiple players.
HMW: How might I create an AR interface that supports mutual understanding in role-play games for deaf kids and therapists?
Concept :
AR Headwear for visualizing deaf kids' intentions in roleplay games
Experiment
In order to create an AR interface that helps with understanding, I conducted 2 tech experiments.
Experiment 1 was about sharing kids’ interested points in sights with teachers(therapists);
Experiment 2 aims to restore incomplete or distorted expressions to their intended meanings.
In the above experiments, computer vision, eye-tracking technology, and LLM were utilized.
If you wish to get access to the source code, please see: Resonant Echoes at Github.
In this experiment, I used a large model named G-VOILA. It’s an enhanced version of the Kosmos model, integrated with the ability to read eye movement data. This model can complement unspoken language information through eye-tracking, aligning well with my concept.
I am also one of the initiators of this technology. If you wish to learn more details about the customed LLM model, please see: G-VOILA: Can AI understand us better if eye data is shared? (research under review.)
Prototype
Product Design
Interface Design
Special Thanks to:
Prof. Yingqing Xu
Head of the Future Lab
Tsinghua University
Prof. Yuntao Wang
Faculty of the PI HCI Lab
Tsinghua University
Yan Qin
Faculty of Rehabilitation Center
China Deaf
Xueqiao Wang
for 3D Printing & Design Critics
