Both students specializing in Information Technology at Le Hong Phong High School for the Gifted (Ho Chi Minh City), Nhat Huy and Minh Man share a passion for technology and programming languages. These two young individuals also have a common desire to apply their knowledge in practical ways, particularly to assist those with special circumstances, especially the hearing impaired.
“Rebuild and Redesign” Many Times
Embarking on the research of smart glasses integrated with artificial intelligence (AI) to help the deaf and mute communicate confidently, Nhat Huy set a goal to create glasses capable of two-way communication. “There are many products that assist the hearing impaired, but they usually only allow for one-way interaction. I wanted to create a compact product that enables two-way interaction to bridge the communication gap and foster a sense of closeness for the hearing impaired,” Nhat Huy said.
During the summer of 2022, instead of spending time relaxing and preparing for their crucial 12th-grade year, Nhat Huy and Minh Man dedicated all their time to learning sign language and communicating with special friends. By August, they began writing their first lines of code.
Nguyễn Minh Nhật Huy and the AI-integrated glasses model supporting communication for the hearing impaired.
While studying at school and conducting scientific research, the team had to “rebuild and redesign” their product multiple times due to dissatisfaction with the results. Ultimately, the smart glasses were completed with a camera in the center, which captures the sign language of the hearing impaired, converts it into sound signals, and then plays back what they want to say through a speaker. Conversely, to help the hearing impaired understand what others are saying, the glasses record the speaker’s voice, convert the audio into text signals, and display it on the glasses’ screen.
During the research process, Nhat Huy’s biggest concern was the reading ability of those who are congenitally deaf and mute, as many of them cannot read text. “When an 11-year-old deaf girl (in first grade) put on the glasses, I was very worried because I didn’t know if the product could be applied practically. Seeing the girl understand the language made me very happy because this was the first stepping stone and motivation for both of us to continue deeper research,” Nhat Huy recounted.
Continuous Improvement
Weighing approximately 300 grams, the glasses are composed of: a 4G SIM card connecting the internet to a server, a Raspberry Pi circuit, battery, camera, microphone, OLED screen… The assembly cost for this type of glasses is around 800,000 VND each.
The research team utilized the Google Assistant virtual assistant tool to record voice and convert it into text. For the sign language reading and voice conversion part, the team had to independently research, writing Deep Learning algorithms to process the data. The glasses have a voice recognition speed of 0.3 seconds per word, with an accuracy rate of over 80%; gesture recognition takes 0.8 – 1.5 seconds per word, with an accuracy rate of over 70%.
The smart glasses have been tested at two schools for the hearing impaired in Ho Chi Minh City and received positive feedback. However, the research group wants more time to upgrade their data repository, as currently, the glasses can only analyze about 100 basic communication sign characters, which accounts for just 1% of the sign language volume for the hearing impaired. “The group is still providing additional data for the glasses to recognize. The camera currently cannot scan many sign language actions, which hinders the communication ability of the deaf and mute,” Nhat Huy added.
According to Ms. Pham Thi Be Hien, Principal of Le Hong Phong High School for the Gifted, the school always creates conditions for students to cultivate their passion for scientific research by providing useful playgrounds and scientific competitions. Through this, the school can identify students with potential and promising models to participate in larger competitions.
|
The AI-integrated smart glasses developed by the research team of Nguyen Minh Nhat Huy and Tram Minh Man won first place in the 2023 Ho Chi Minh City Department of Education and Training’s Science and Technology Competition, and second place in the 2023 Ho Chi Minh City Young Informatics Contest organized by the Youth Science and Technology Development Center – Ho Chi Minh City Youth Union as the permanent organizing body. The team is currently preparing documentation to submit this model for the 29th National Young Informatics Contest in 2023. |
