Jesture

FRAMEWORKS – Jesture is an interactive installation based on the principle of mixed reality, which tries to enhance the meaning of an architectural space. We have tried to achieve it by developing a game named ‘Jesture’, where virtual objects play with the real space, walls and depth according to the hand gestures of the user.

The game is about navigating the character through different levels and directing it back to the initial position. The direction of the player’s hand gestures guides the motion of the character in virtual space. The character moved in left or right direction by simply moving the players’ hand in left or right direction. The character can also jump up by moving the hand in the upward direction.

The level of immersion in a gaming experience can be enhanced by using mixed reality techniques because of the involvement of real time interactions between real and virtual objects. So we tried to use the MR technologies to translate the conventional gaming world into more naturally immersive one.

We initially started our mixed reality experimentation with the traditional game of ‘Pong’, where the virtual ball bounces within the physical boundaries of the wall For this game, we selected the wall which housed the railing of the stairway as the game The profile of the stairway acted as a boundary for the virtual ball to bounce back. Players were provided with a wired glove which controlled the paddle for hitting the ball.

The video (below) showcases our initial explorations involving Projection Mapping techniques, which revealed us the problems regarding lighting, shadow and issues of perspective. We chose Foyer at National Institute of Design, Ahmadabad, India which provided us a challenging setting for the gaming experience.

Players were provided with a wired glove which controlled the paddle for hitting the ball. The wired glove is equipped with triple-axis Accelerometer, single-axis MEMS Gyroscope and Force Sensing Resistor. It is connected to a single board microcontroller prototyping platform – Arduino. The game interaction is achieved by programming in an open source programming language – Processing 1.2 and projected using projection mapping technique.

This is our 1st video prototype of the working of the sensors

Selected Awards Exhibited at the Art Exhibit of SIGGRAPH ASIA 2011 at Hong Kong, China. Runners-up at Yahoo! USID 2010 Academic Showcase, as a part of USID 2010 conference at Hyderbad, India

Nice Cube and Play Pong were other games developed under FRAMEWORKS series.

Created in collaboration with Promila Roychoudhury.