Abstract

Real-time communication between Deaf or Hard of Hearing users and hearing users remains constrained by fragmented speech recognition, sign language synthesis, and video conferencing technologies. Although advances in automatic speech recognition (ASR), text-to-speech (TTS), and avatar-based sign synthesis have improved individual components, end-to-end systems continue to suffer from high latency, platform restrictions, limited linguistic fidelity, and weak integration across modalities.

This work presents Tandem, an end-to-end communication platform prototype for bidirectional spoken–signed interaction that bridges the gap between component-level progress and deployable real-time accessibility systems. Unlike existing approaches that rely on proprietary conferencing APIs, fragile browser overlays, or offline pipelines, Tandem prioritizes full system control and low-latency streaming.

Tandem is implemented using a WebRTC-based video conferencing stack with custom signaling and STUN/TURN-assisted connectivity. Client-side audio is captured in real time, streamed via Socket.IO to a cloud-based ASR service, and routed through text-to-speech synthesis and avatar-based sign synthesis using the open-source Sign.mt framework. In parallel, a computer vision–based ASL recognition model implemented using PyTorch and skeletal keypoint representations enables signed input to be translated back into spoken output. Together, these components demonstrate that tightly integrated, open, real-time pipelines can substantially improve responsiveness and accessibility in spoken–signed communication, while providing a reproducible system architecture for future research.

Advisor

Guarnera, Heather

Department

Computer Science

Disciplines

Artificial Intelligence and Robotics | Computer Sciences | Graphics and Human Computer Interfaces

Keywords

accessibility, webrtc, sign-language, machine learning, flask, mediapipe

Publication Date

2026

Degree Granted

Bachelor of Arts

Document Type

Senior Independent Study Thesis

Download

Share

COinS
 

© Copyright 2026 Hiruy Worku