Introduction
Motivation
Approximately 11,000 Americans lose their hearing every year, and over 500,000 people in the United States use American Sign Language (ASL) as their primary language [1]. Despite this, the overwhelming majority of hearing people have no ASL knowledge, creating a persistent communication barrier in everyday settings. Sign language recognition (SLR) has been an active area of computer vision research for decades [2], yet most deployed systems remain too expensive or hardware-intensive for everyday consumer use. A lightweight, webcam-based recognition system could make ASL communication significantly more accessible at near-zero cost.
Problem Definition
We focus on isolated gesture recognition: given a short video clip of a single ASL sign, classify it into one of 10 gesture categories. The system must run in real time on a standard laptop webcam without requiring a GPU or specialized hardware. This rules out computationally expensive approaches such as full-frame CNNs and constrains both model complexity and inference latency to fit within a 33 ms/frame budget for 30 fps output.
Project Goals
Our goals were to (1) build a complete end-to-end pipeline from webcam capture to live sentence output, (2) compare a classical ML baseline (KNN) against two recurrent architectures (LSTM, GRU) on self-collected data, (3) analyze where each model succeeds and fails per class, and (4) identify next steps for scaling to real-world deployment.
Demo Video
Webcam inference where ASL gestures are assembled into a sentence in real time.
Demo Video Coming Soon
Place the file at assets/demo.mp4 and the page will switch to the video player.
ML Pipeline
Six stages from raw webcam data to real-time inference.
Collect Data
Capture 30-frame sequences per gesture class from a webcam in a controlled recording session.
src/collect_data.py
Extract Keypoints
Run MediaPipe Hands [3, 4] to extract 21 landmarks per hand (42 total), then normalize relative to the wrist origin.
src/utils.py
Preprocess
Build NumPy arrays from raw sequences and split into 70/15/15 train / val / test sets.
src/preprocess.py
Train Models
Train KNN [7], LSTM [5], and GRU [6] on the processed sequences. Early stopping and gradient clipping applied to recurrent models.
src/train_knn.py + src/train_lstm.py
Evaluate
Generate confusion matrices, per-class accuracy, and timed inference benchmarks for all three models.
src/evaluate.py
Real-Time Inference
Open webcam, buffer 30 frames, run the chosen model, and append high-confidence predictions to an on-screen sentence.
src/realtime_inference.py
Dataset
Self-collected data. No external dataset was used.
Each sample is a 30-frame window with 126 features per frame: 63 (x, y, z) coordinates per hand, both hands, normalized to the wrist origin to remove positional variance. This skeleton-based representation, rather than raw pixels, reduces input dimensionality and makes features lighting- and background-invariant, a strategy supported by prior SLR work [2, 9].
Model Architectures
Three approaches covering the spectrum from classical ML to recurrent deep learning.
K-Nearest Neighbors serves as the classical baseline. The 30-frame sequence is flattened to a 3,780-dimensional vector, reduced via PCA (95% variance retained), and classified by majority vote among the k = 5 nearest neighbors (Euclidean distance). KNN requires no training loop and provides a strong interpretable baseline against which temporal models can be measured [7].
Long Short-Term Memory networks were introduced by Hochreiter & Schmidhuber [5] specifically to capture long-range dependencies in sequential data. Our 2-layer LSTM (hidden size = 128) processes the 30-frame keypoint sequence and classifies from the final hidden state. Gradient clipping (max norm = 1.0) and early stopping (patience = 15) are applied to prevent exploding gradients and overfitting.
Gated Recurrent Units were proposed by Cho et al. [6] as a computationally lighter alternative to LSTM. GRU merges the forget and input gates into a single update gate, reducing parameter count while preserving the ability to model temporal sequences. Empirical work by Chung et al. [8] found GRU competitive with LSTM on many sequence tasks, a finding our results echo at 4× faster inference.
Model Comparison
All three models reached 80% test accuracy; inference speed became the deciding factor.
Since all models tied at 80% accuracy, inference latency separates them in practice. KNN is fastest (2.53 ms) but treats each frame independently with no temporal modeling [7]. LSTM captures motion context [5] but at 28.59 ms exceeds the headroom available at 30 fps on lower-end hardware. GRU [6] captures equivalent temporal structure while running nearly 4× faster than LSTM [8], making it the best choice for real-time CPU deployment.
| Model | Test Accuracy | Inference Speed | |
|---|---|---|---|
| KNN [7] | 80% | 2.53 ms / sample | Fastest |
| LSTM [5] | 80% | 28.59 ms / sample | Slowest |
| GRU [6] | 80% | 7.17 ms / sample | Best Balance |
Confusion Matrices
Click a model to inspect its per-class performance.
All three models classified visually distinct signs (hello, love) perfectly, but consistently confused motion-sensitive pairs like yes/no and me/you. This aligns with findings in the SLR literature [2, 9] that skeleton-only features may be insufficient to disambiguate gestures whose keypoint trajectories overlap spatially, pointing to motion velocity or depth features as a natural extension.
Training Behavior
How LSTM and GRU learned over 50 epochs.
Training accuracy climbed quickly while validation accuracy plateaued, a textbook sign of overfitting. With only 100 total samples across 10 classes, both recurrent models saturated the available training signal within 15 epochs. This is consistent with the data-hunger documented in deep SLR surveys [9]: recurrent architectures require hundreds of samples per class to generalize beyond the training signers. Larger, more diverse datasets remain the primary bottleneck.
Live Inference System
What happens frame-by-frame when the system runs.
At startup, the system opens the default webcam and launches a MediaPipe Hands [4] session. Each incoming frame is processed to extract 21 hand landmarks per hand, normalized relative to the wrist origin, and pushed onto a rolling 30-frame buffer.
Once the buffer is full, the feature vector is passed to the loaded model (KNN, LSTM, or GRU). If the prediction confidence exceeds a configurable threshold, the recognized gesture label is appended to an on-screen sentence. The buffer resets and collection continues.
The system runs in a tight OpenCV display loop with majority-vote smoothing over the last 8 predictions to reduce per-frame noise. No GPU is required; the GRU model runs comfortably in real time on CPU.
First-Person Video Extension
Exploring a glasses-mounted perspective for naturalistic ASL capture.
The live inference system uses a standard desktop webcam as the video source. MediaPipe [3, 4] extracts hand landmarks from each frame in real time, and the model classifies the buffered sequence on-device with no external server required.
Ray-Ban Meta glasses can serve as a wearable, first-person video capture source, removing the constraint of a fixed desktop camera. On-device ML inference via TFLite or CoreML would enable fully embedded deployment [9].
- Run inference on video files recorded from the glasses
- Build a mobile bridge to stream frames to the inference script in near-real-time
- Investigate on-device ML options for fully embedded inference
Key Insights
What we took away from building and evaluating this system.
Data Was the Bottleneck
With 10 samples per class, model complexity was irrelevant; all three architectures hit the same ceiling. The SLR literature consistently shows that data diversity and volume drive accuracy more than architecture choice at this scale [2, 9].
GRU Was the Right Pick
KNN and LSTM represent the extremes of simplicity and complexity. GRU [6] sits between them, modeling temporal sign structure while running 4× faster than LSTM [8], comfortably within the 33 ms real-time budget on CPU.
Similar Signs Exposed the Limits
Gestures like yes/no and me/you share spatial overlap in keypoint space. Koller's survey [2] identifies this as a known challenge in skeleton-based SLR; motion velocity, depth, or signer-relative orientation are the recommended extensions.
Robustness Needs Diversity
All data came from the same signers in controlled conditions. Rastgoo et al. [9] document that signer-independence is the primary open challenge in deep SLR; adding varied signers and lighting is the highest-impact next step.
Conclusion & Future Work
What We Accomplished
We designed, built, and evaluated a complete real-time ASL gesture recognition system from scratch: from webcam data collection and MediaPipe [3, 4] keypoint extraction to model training, evaluation, and live inference with an on-screen sentence builder. All three models (KNN [7], LSTM [5], and GRU [6]) achieved 80% test accuracy on our 10-class dataset. With accuracy equal, GRU emerged as the best deployment choice due to its 4× speed advantage over LSTM [8] while still capturing the temporal structure of signs.
Lessons Learned
The most important lesson was that data quantity matters more than model complexity at small scale, a pattern well-documented in the SLR literature [2, 9]. Temporal modeling (LSTM, GRU) is essential for motion-dependent signs; KNN loses directional information by treating frames independently. Migrating to the MediaPipe Tasks API mid-project reinforced the value of designing for dependency volatility early.
Future Work
With more time and resources, we would expand to the full ASL alphabet, collect data from a wider signer population to address the signer-independence challenge identified by Rastgoo et al. [9], and add a null/rest class to prevent false positives between gestures. On the architecture side, Transformer-based temporal models have shown strong results on sequence tasks and warrant exploration. The most compelling direction is integration with Ray-Ban Meta glasses for first-person, wearable ASL capture, removing the fixed desktop camera constraint entirely.
Team Roles
Each team member contributed technically and academically to this project. Roles include coding, data collection, model development, evaluation, and documentation. All members agree to the responsibilities listed below.
Nissi Otoo
- Team lead & coordination
- Documentation
- LSTM model development
- Evaluation pipeline
Joshua Streat
- Documentation
- Data collection
- GRU model development
- Training curve visualizations
Emmily Castaneda
- Documentation
- Evaluation
- KNN model development
- Model comparison visualization
References
- R. E. Mitchell, T. A. Young, B. Bachleda, and M. A. Karchmer, "How many people use ASL in the United States? Why estimates need updating," Sign Language Studies, vol. 6, no. 3, pp. 306–335, 2006.
- O. Koller, "Quantitative survey of the state of the art in sign language recognition," arXiv preprint arXiv:2008.09918, 2020.
- C. Lugaresi et al., "MediaPipe: A framework for building perception pipelines," Workshop on Perception for AR/VR at ICCV, 2019.
- F. Zhang et al., "MediaPipe Hands: On-device real-time hand tracking," arXiv preprint arXiv:2006.10214, 2020.
- S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
- K. Cho et al., "Learning phrase representations using RNN encoder-decoder for statistical machine translation," in Proc. EMNLP, pp. 1724–1734, 2014.
- T. Cover and P. Hart, "Nearest neighbor pattern classification," IEEE Trans. Information Theory, vol. 13, no. 1, pp. 21–27, 1967.
- J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling," arXiv preprint arXiv:1412.3555, 2014.
- R. Rastgoo, K. Kiani, and S. Escalera, "Sign language recognition: A deep survey," Expert Systems with Applications, vol. 164, p. 113794, 2021.