Body language interpretation research connects computer science student with ASL community
Have you ever received a text message in which the sender’s tone was unclear? That’s because crucial pieces of information are missing to fully understand the message.
Gestures and body language are important parts of speech that can’t be conveyed in the text- and speech-based input that dominate today’s human-computer interfaces.
People who rely on text-based communication systems such as those with hearing impairments, but also have fully expressive communication systems like American Sign Language (ASL), could benefit from machine interfaces that can understand gestures as well as spoken or written words.
To address this need, computer science doctoral student Prajwal Paudyal is working on SCEPTRE in the iMPACT Lab. SCEPTRE is a smartphone interface that uses two wireless armband sensors to take in ASL sign gestures to communicate via computer systems.
“The ASL user trains the system by performing three iterations of a gesture, which is captured by a smartphone,” Paudyal says. “The data is then sent to the server to train a computer model. Later, when the user needs to use the system, she puts on the wireless armband sensors and performs the gesture, which is recognized in the smartphone, in a fog server or from the server, depending on communication channels.”
Finding a computing need and solving it
Paudyal became interested in working with technology to assist ASL users when he started his research for human-computer interaction using mobile devices to redefine the way people interact with machines.
“I quickly realized that having a syntactic language to interface with machines is very important,” Paudyal says. “I started looking into sign languages because they are completely defined languages with full expressibility and flexibility. I started taking ASL classes myself and had the opportunity to interact with people who are deaf as well as with numerous users of ASL.”
He decided to continue on with this work for his doctoral research project because of its potential to help people.
“The immediate impact of this type of research on the real world and the potential increase of quality of life it promises to bring brought me a sense of fulfillment, and thus I decided to stick with this line of research for my Ph.D. dissertation,” Paudyal says.
Recognition for research excellence
For his research efforts so far, Paudyal received the Spring 2016 Graduate and Professional Student Association (GPSA) Outstanding Research Award, which recognizes ASU graduate and professional students who exemplify excellence in research and development. Students are judged on the interdisciplinary character of their projects, the positive impacts on the community and the outstanding contribution of the research, among other factors.
Computer Engineering Graduate Program Chair and computer science Professor Sandeep Gupta nominated Paudyal for the award. The iMPACT Lab faculty advisor thinks Paudyal’s research is very promising and pioneers pervasive personalized and extensible assistive technology for sign language users.
“He earned the GPSA award because his research is an interdisciplinary project that combines computer science (the Internet of Things and human-computer interaction), linguistics and speech and hearing science,” Gupta says. “It reflects that he is doing impactful research on a complex problem that involves knowledge from various fields. This kind of use-inspired research can not only solve immediate problems but also spur fundamental research questions.”
Paudyal is thankful for the recognition he has received, and says it inspires further commitment to his work.
“Receiving the Outstanding Research Award has helped me solidify my commitment to research and to the field,” Paudyal says. “The acknowledgment and support I get from peers, other researchers and the graduate community in general at ASU helps me stay focused and inspires me to do more.”
A love of discovery makes a good researcher
Paudyal says he owes his skill as a researcher to his desire for discovery and sense of curiosity.
“I like being on the cutting edge and discovering things for the first time,” Paudyal says. “In my ‘other life’ outside of the research lab, I am a photographer and an adventurer and I like discovering new places. I think this curiosity and sense of adventure helps me a lot as a researcher because research to me is the ultimate intellectual adventure.”
He also enjoys documenting his life by writing blogs and marking trails at the places he explores to guide his future excursions and to learn as much as he can. His meticulous approach also helps him in documenting his research.
“I have a similar attitude about documenting a lot and staying vigilant when I conduct research,” Paudyal says. “This has helped me grow as a researcher because I can constantly reevaluate my approaches and improve my process. I think this vigilance more than anything else is what makes me a good researcher and has put me on an upward spiral of improvement.”
Continuing to help the world through accessibility research
Paudyal has taken his research beyond the lab to apply his work in ways that can help those who need more accessible communication options.
He and his colleagues submitted a grant to the National Institutes of Health (NIH) that aims to do a usability study on emergency healthcare scenarios for people with hearing and speaking difficulties.
He also is actively involved in ASL and deaf clubs around the Phoenix metro area.
Next, Paudyal plans to extend his research in two key areas to improve its practical and theoretical applications.
“First I want to incorporate fingerspelling, which is what ASL users use if there is no word equivalent in the ASL dictionary,” Paudyal says. “I am also focusing on the theoretical background for sign language and gesture processing to see if I can make contributions in the theoretical realm as well as the practical realm.”
After he earns his doctorate he hopes to find a position that allows him to continue his research in accessibility, particularly music therapy and sign language processing.
Media Contact
Monique Clement, [email protected]
480-727-1958
Ira A. Fulton Schools of Engineering
