Gesture Desktop — Hand Gesture Control

Gesture Desktop is a GENBOT project that lets you control your browser cursor and page scroll entirely with hand gestures, using a webcam and Google's MediaPipe hand-tracking pipeline. It is not a demo video — the technology is live on this website in the bottom right "Gesture Control" toggle.

How it works

• MediaPipe Hands locates and tracks a single hand in real time.
• Landmark coordinates (index finger and thumb) are mapped to screen coordinates.
• Pinching index and thumb together performs a click on whatever is under the cursor.
• Large vertical hand movement triggers smooth scrolling.

Under the hood

The pipeline runs entirely in the browser: MediaPipe hands.js and camera_utils.js on a hidden webcam feed, with landmarks mapped to the page. Because the model runs client-side, no video leaves your computer — useful for hands-on HCI research with zero privacy overhead.

Technologies

• MediaPipe Hands — real-time hand landmark detection.
• Browser computer vision and camera integration.
• JavaScript interaction mapping — the same pipeline GENBOT uses for human-robot interaction research.

Related work

• NexBot — human interaction vision pipeline built on similar hand-tracking research.
• AI & Machine Learning department