Congratulations

[Pause for laughs]

- for not joining the losers next door

Hello

My name is Mairead.

  • 10+ years in web industry

  • Specialise in Frontend, Responsive & Javascript

  • Interested in interaction paradigms

- growth of responsive and touch screen

- internet of things

- led me to explore 3D input and kinect

Technology stack

- Ask for a show of hands

3D motion sensing

Devices that detect motion in space

  • Wearable Accelerometers

  • Muscle sensing

  • Cameras/Infrared

  • Apple device (coming soon)

- Brief run down of devices available. How do they work

Real world application

Leap Motion device

  • Near field mode & Finger detection

  • Low cost per unit

  • Very low latency

  • Mimics touch events

- Outline why Leap motion chosen. Can I show field of view? Is there a good Hand demo here?

Leap JS library

Javascript API for handling device data

  • Controller, Frame

  • Hand, Pointable

  • Gesture

  • Interaction Box

- Describe basic API methods

- Interaction Box

- Demo finger points on screen

3D in Browser

WebGL canvas context

  • Web GL exposes Javascript API

  • GLSL written in C like syntax

  • Heavily optimised for GPU

  • Vertex and Fragment Shaders

- List main concepts

- Display sample C code

- Explain how to link to it

Enter ThreeJs

Mr.Doob = Legend

  • Camera

  • Core - including Geometry

  • Lights

  • Materials

- Summarise basic API

ThreeJS basics

  • Stage, Grid, Geometry

  • Materials and lighting

  • Cameras

  • Normal = Rainbows!

- Open window in separate port

- Show code samples

Here comes the science

Euclidean geometry

Describing the motion of items in space

- Ask for discussion afterwards

Vectors

Describe direction

From one point in space to another

Rotation

Euler Angles

Describe the angle of rotation around each axis

Gimbal locking

The Problem of angles in calculating rotation

Pitch, Roll and Yaw

Tait-Bryan angles

  • Another form of Euler Angles

  • Commonly used in Aeronautics

3D modelling software

Blender

  • Open source [a.k.a. free!]

  • Not as complicated as it looks

  • Has ThreeJs plugin to export JSON

  • Shares common 3D language

- Fairly shallow learning curve, plenty of resources

- Demonstrate Blender GUI

- highlight similarities in Blender language and ThreeJs

Using Armature

Based on model by Erik90mx

  • Mesh and armature

  • Named bone structure

  • Vertex group mapping

- Demonstrate vector mapping to named bones

Blender to ThreeJs

Exporting JSON for ThreeJs Loader method

  • Materials

  • Vertices

  • Bones = map to LeapJs hands and pointables

- Show Bone array

- Run through other pertinent elements, shaders, vertex groups etc

Position and rotation

Generating hand movement

Based on work from a couple of blog posts

Positioning Hand

var hand = frame.hands[0];

handMesh.position = hand.palmPosition; 

var rotation = {
	z: hand.pitch(),
	y: hand.yaw(),
	x: hand.roll()
};

bones[0].rotation.set(rotation.x, rotation.y, rotation.z);
						

Positioning Fingers

var fingerDir = finger.pointable.direction;

mainBone.rotation.set(0, -fingerDir[0], fingerDir[1]); 

// apply rotation to phalanx bones
for (var i = 0, length = finger.phalanges.length;
 i < length; i++) {
  var phalange = finger.phalanges[i];
  phalange.rotation.set(0, 0, fingerDir[1]);
}
						

The thing

Moving hand and fingers

  • Pitch, Roll and Yaw movement of palm and Gimbal lock

  • Finger detection and impaired movement

  • Finger rotation

- Demonstrate limitation of finger movement

- Demonstrate Pitch, Roll, Yaw

- Demonstrate Gimbal lock

Conclusion

Further work required

  • Still at experimental stage

  • Reduce flexibility of finger rotation

  • Convert Rotation to Quaternions to prevent Gimbal lock

  • Use animations to simulate posed gestures

Fin.

Thanks for coming!

- Hand movement prompts finish animation on screen.