Dragonfly: Sign Language to Voice System (Session 1E)
Main Topic: Deaf Community
Session Type: Paper
DragonFly is a system we have developed that bridges the communication gap between deaf and hearing individuals. This technology enables deaf and hearing individuals to communicate with each in their own languages, American Sign Language (ASL) and English, using mobile devices, such as smartphones and tablets. The system is designed for use in impromptu situations when interpreters are not available, and can be used either face-to-face or on-line. At past PacRim Conferences, we presented the research and development (R&D) we have been carrying out to create this innovative new capability, and demonstrated the operational prototypes we developed as part of the ongoing work towards the overall end-to-end solution. At last year’s conference, participants were able to sign words and short sentences into Dragonfly based on a restaurant scenario, and the system automatically translated and enunciated their signing into English. The audience was very positive about the capability, and they provided us with excellent feedback, which we incorporated into the new version of the system. Since the last conference, we have developed an avatar which is displayed to the deaf user on their mobile device, which signs what the speaker of English said. We have also developed the ability for continuous sign recognition by developing new techniques for the Automatic Sign Language Recognition (ASLR) and Machine Translation (MT) components. This makes the automated translations more accurate and it also enables the user to sign in a greater number of different types of sentences with more vocabulary. We presented our work at the most recent Association of Machine Translation in the Americas (AMTA) conference in September of this year and the project has generated much interest in the MT community.
At PacRim 2021, we will demonstrate the end-to-end Dragonfly system, discuss our ongoing R&D efforts to develop and improve the system, and our plans for commercializing this technology. Participants will be able to learn about this new, enabling technology and the commercialization plans we are putting in place to commercialize this capability in the near future. They will also learn about how this innovative tool will facilitate inclusion by increasing the number of conversations between the hearing and deaf communities. The audience will come away with information about a new technology that they may be interested in using themselves when it becomes available.
Keywords: Deaf Accessibility, Communications, Sign Language
Mr. Cham LeangUS Government - Program Manager
Cham, of Northern Virginia, part of Washington D.C, is a Senior Program Manager with the federal government who manages multiple engineering projects. He is currently President and leading the “Deaf In Government (DIG)” Organization to serve all local, state, and federal level. He is Deaf and has a passion for enhancing innovation in accessibility space for all.
Dr. Patricia O’Neill-BrownU.S. Government - Advisor
Dr. Patricia O’Neill-Brown holds a Ph.D and M.S. in Computational Linguistics from Georgetown University. She works for the U.S. Government and for over 20 years, she has led the development, implementation, and fielding of integrated Machine Translation (MT) solutions to meet a wide variety of user needs — from leveraging MT to help users understand content in other languages to bringing automated MT systems to bear upon translation processes.