The creation of hyper-realistic sign language avatars by a start-up in New Zealand might not only open pathways to learning in the Deaf community but could also mean the difference between life and death in an emergency.
When one of the co-founders of Kara Technologies, Arash Tayebi, developed Meniere’s disease, which led to deafness in one ear, he realised there was a lack of educational resources for Deaf people. With friends including Farmehr Farhour – a fellow engineering student at the University of Auckland – he started investigating the issue.
Farmehr said their first step was to look at creating captioning software that catered for the New Zealand accent. “But we soon realised that sign language is the first language of any Deaf person.
“Then when we created the company we said ‘Let’s make digital avatars’ because there is just so much content out there that human interpreters cannot keep up,” Farmehr said. Kara Technologies was born in 2018, the year they finished university.
Complementing, not replacing
Farmehr is quick to point out that they are not aiming to replace human interpreters.
“For instance, in an emergency situation, we can deliver an accurate sign language translation in under a minute, whereas it would normally take a long time to find a human interpreter. However, our technology would not be ideal for long-form contextually heavy information such as press conferences, which is where human interpreters are needed,” he said.
Farmehr said his team’s technology is a world first and is able to generate sign language translations in real-time from a natural language input.
“We have co-designed the technology with the Deaf community from day one and have developed the most advanced sign language avatars in the world. We use the latest Large Language AI models for semantic translation and our own proprietary technology to kinematically drive the avatars based on a dictionary of signs.
“Our process is end-to-end reviewed by Deaf sign language experts for quality assurance. That means that we can ensure our content is 100% accurate while maintaining the speed of content production,” Farmehr said.
Listening to the end-user
Kara Technologies has developed proprietary technology in-house on how to drive the digital avatars for sign language smoothly, and Farmehr said they use the best-in-class digital humans from Epic Games and innovate on top of pre-existing AI models for translation of the content.
“We wanted to differentiate ourselves with the quality of the avatars’ sign language. Every country has its own languages which are very, very focused on minute fingerspelling or specific facial expressions.
“We realised if you were in the context of storytelling, it’s all about that emotion and how you portray it and it doesn’t make a difference between languages.
“However, for systems where it is about getting the information out, such as an emergency situation, we wanted to make sure that the way we automate that generation of avatar sign language content is standardised to an extent where it’s not dialect based,” he said
Farmehr says their Deaf community advisory board also deals with questions of ethnicity and culture. For example, a Maori avatar cannot be derived from motion capture of a non-Maori person.
“There’s quite a lot of discussion in the community and there’s no consensus about this. What we do is make sure that we standardise as much as possible, and we create avatars that are as neutral and diverse as possible. We try to represent all communities across different spectrums.
“We are a Deaf-led organisation, and co-create our technology with guidance from the community and stakeholders. We have conducted extensive surveys in Australia, New Zealand and the US, and the project received a positive reception in all three countries, highlighting the importance of providing equal access to information in sign language for the Deaf community.
To ensure Kara Technologies can expand, Farmehr said their system is agnostic. While many countries have differences in their sign language, there are enough similarities that the team only needs to create a new data set of signs, like a vocabulary. The underlying backend of the systems remains the same.
Emerging from the chrysalis
For such a young team, there have been some steep learning curves.
“Our first challenge was keeping ourselves accountable by having the Deaf community included in the design process. As engineers coming straight out of university all we want to do is build, but you have to step back and ensure that you do it the right way. You include the people who receive the service at the end of the day, and they need to make the decisions on how it looks, not us,” Farmehr says.
Another challenge has been focused around educating organisations in what accessibility actually looks like and the importance of understanding the need for additional accessibility features and inclusivity.
All indications so far are that they are on the right track. Their sign language avatar version of The Very Hungry Caterpillar has close to 90,000 views on YouTube and Farmehr said the feedback, especially from teachers, has been extremely positive. The next challenge is how to scale the approach across large chunks of digital content, such as educational resources.
“We are actually in the middle of working on how we automate that process while having some human oversight,” Farmehr said.
The team is also working with the Australian Red Cross and emergency providers in the US to create avatars for emergency and public announcements.