Chen Liang

@chenliangcs.bsky.social

9 Followers 47 Following

PhD @ UMich https://liangchenlc.com

A figure showing a comparison between the existing approach and the proposed HandProxy approach. The existing approach usually relies on a list of discrete, predefined commands such as "Activate the lever," "Select the cube," "Open the box," "Close the box," "Resize the window," and "Resize the object". The proposed HandProxy approach supports more natural, expressive interactions using varied and context-rich phrases like "{punch, push, cut} the cube," "grab the {lever, fruit}," "{move, twist} the {hand, object}," "put apple into the basket," "pinch up the window corner," and general control commands like "stop, continue, undo." These inputs are decomposed into virtual proxy hand movements, which then interact directly within a virtual environment.
a figure showing possible factors causing hand interactions to be unavailable, including situational impairments (e.g., occupied hands, fatigue after long term use), environmental constraints (e.g., confined physical space, virtual object out of reach), and upper-body limitations (spasm or tremor, rapid fatigue, poor coordination, difficulty controlling distance/direction)
A figure showing an overview of the HandProxy system in 3 stages. The user starts by giving a speech command (first stage), which is used and interpreted by the HandProxy system to control a virtual proxy hand (second stage), and the proxy hand will then interact within the virtual environment to perform hand interactions on behalf of the user (third stage), followed by a video demo of a virtual hand puts a virtual apple and peach into the basket, and maximizes the volume by pushing the volume slider up