A gesture is not a photograph
SanketSpeak started with a simple ambition: translate dynamic Indian Sign Language gestures into speech in real time. The word dynamic is what made the project interesting.
A single frame can show where a hand is. It cannot fully explain where the hand came from, how it moved, or what the next frame means. Sign language lives across space and time, so the model needed to learn both.
That is why I combined convolutional features with a BiLSTM and an attention mechanism. The CNN captures spatial information from each frame. The BiLSTM reasons across the sequence. Attention helps the network focus on the moments carrying the strongest signal.
The model was not learning a pose. It was learning the shape of a motion.
The pipeline mattered as much as the network
Before training, I built a preprocessing pipeline to extract and standardize keyframe sequences. Real videos are uneven: clips have different lengths, hands enter the frame at different moments, and camera conditions change.
Each sample had to become a consistent temporal input without erasing the movement that defined it. I selected representative frames, normalized dimensions, and kept the ordering intact.
def prepare_sequence(frames, target_length):
indices = np.linspace(0, len(frames) - 1, target_length)
selected = [frames[round(i)] for i in indices]
return np.stack([normalize(frame) for frame in selected])
MediaPipe and OpenCV helped with capture and feature preparation, while TensorFlow and Keras handled the model. The cleanest training run would still be useless if the live camera produced a different kind of input, so I treated preprocessing as a shared contract rather than a notebook cell.
Why bidirectional context helped
When interpreting a recorded sequence, later frames can clarify earlier ones. A BiLSTM reads the sequence in both directions, letting the representation use context from the complete gesture.
For a live product, that design creates a latency tradeoff: waiting for more frames improves context but delays the spoken result. The prototype favored reliable sequence classification, but a production version would need a deliberate decision about when a gesture has ended.
Attention made that decision-making more visible. Instead of compressing every timestep equally, the model learned to emphasize the frames that contributed most to the prediction.
Ninety-five percent was a milestone, not a conclusion
The model reached 95% validation accuracy on the INCLUDE-50 dataset. I was proud of that number, but it also forced me to ask better questions.
- Does performance hold under different lighting?
- What happens when the signer stands farther from the camera?
- Are similar gestures confused in consistent ways?
- Does the training set represent different signing styles and body shapes?
- How should the interface behave when confidence is low?
Accuracy describes a dataset split. Accessibility describes a person’s experience. They are related, but they are not interchangeable.
Speech should not pretend certainty
The final stage turns a predicted gesture into audio with pyttsx3. That output feels satisfying because it makes the system tangible: movement becomes language that another person can hear.
It also raises an interface question. A wrong spoken result can be more disruptive than a low-confidence visual suggestion. A better version should expose confidence, offer quick correction, and avoid speaking when the signal is ambiguous.
Human-centered ML is full of these moments. The technically impressive action is not always the most helpful one.
What I would build next
I would expand the vocabulary, evaluate signers unseen during training, and measure latency on ordinary hardware. I would also work with people who use Indian Sign Language rather than assuming a benchmark captures the entire problem.
SanketSpeak taught me that assistive technology should begin with humility. A model can recognize patterns, but usefulness comes from the surrounding choices: the data collected, the mistakes anticipated, the feedback offered, and the people invited into the design.
The project began as a computer-vision challenge. It ended up teaching me much more about listening.
