THE OPPORTUNITY
THE SOLUTION
I was tasked with creating a multimodal device that addresses a problem by incorporating visual, auditory, and haptic components in an inclusive manner. I decided to focus on nonverbal communication, which plays a crucial role in daily interactions by helping convey emotions and intentions. By designing something to appeal to multiple senses, I aimed to ensure that users with different abilities or preferences could access its functionality.
SenseSpeak is a device that uses sensory feedback and gesture recognition to enhance communication. It provides audio and haptic feedback to assist blind individuals in understanding nonverbal cues, and features an online component that can display detected gestures during virtual meetings. Additionally, this online component can be used to translate ASL in real-time, enabling seamless communication between ASL users and non-users.
HOW IT WORKS
My device is worn on the wrist in order to detect the wearer's movements to communicate the gesture. The device is capable of recognizing a library of gestures, such as waving, and will vibrate and say the meaning of the detected gesture, such as "Josh is saying hello". The online component displays the detected gesture on the screen during virtual meetings like Teams or Zoom. This feature makes the device useful not only in face-to-face interactions but also in online settings.
INITIAL CONCEPT
The original design utilized a microbit attached to the wrist and an online application to enhance nonverbal communication. With its affordances of real-time communication and portability, the device allows ease of movement and customization for tailored feedback. When the device detects the wearer's movements, it announces the corresponding action or meaning, such as "Wave: greeting.".
To test my design, I created two prototypes using the microbit. The first prototype uses distinct pitches to represent different gestures made by the user. Each gesture is accompanied by a different pitch. The second prototype, on the other hand, verbally announces the gesture being performed by the user.
CLIENT FEEDBACK
During a client feedback session, valuable suggestions were made to improve the design of the device. One suggestion was to create a library of gestures and incorporate different vibration patterns to enhance the base idea. We also discussed exploring additional gestures by observing how peers interact during Zoom calls. It was also recommended to think about language use to avoid sounding too robotic. We discussed the concept of shared intentionality and the potential to use nonverbal gestures as metaphors. For instance, instead of announcing "Wave: Greeting," the device could say "John is saying hello."
Overall, the feedback session allowed me to refine and develop the design, resulting in the current version of the device that offers improved functionality and usability.
DEVICE DEMO CODE FOR WAVING
ONLINE COMPONENT MOCKUP
For users who prefer not to use the audio component or need additional visual support, the online component can display the detected gesture or action on the screen with a thick border for enhanced visibility. This feature can also benefit users who rely on American Sign Language (ASL) translations. To ensure accessibility, the gesture/action label is displayed in a dark blue color that meets the Web Content Accessibility Guidelines (WCAG). With our online component, users can choose the feedback mode that best suits their communication needs and preferences, making virtual interactions more inclusive and accessible.
AD CAMPAIGN
The "Communication Without Limits" campaign uses bold, accessible colours and hand gestures (including ASL for "help") to show our features for alternative communication methods. The campaign's clear messaging appeals to a wide audience with different abilities and communication preferences, emphasizing the importance of inclusive communication. The campaign's effectiveness lies in its ability to communicate a powerful message that resonates with a diverse audience.
PROTOTYPE DEMO
*Prototype demo with the latest design iteration will be uploaded soon. Stay tuned for updates!*