Leap motion and VR

About me

My name is Mairead

  • 15 years in web industry

  • Specialise in Frontend & Javascript

  • Experimenting with interaction paradigms

Talk contents

  • My work

  • 3D in browser

  • Mathematics

  • Input device

  • Output device

  • Architecture

  • Virtual reality

  • Project future

My work

Where it started

The story so far

  • Kinect - JS hack over 4 years ago

  • Leap Motion release

  • Custom built Hand rigging last year

  • Google cardboard to simulate VR

Real world application

Why use 3D motion technologies?

Leap motion and VR

Use of technology

Technology stack

3D in browser

3D in Browser

WebGL canvas context

  • Web GL exposes Javascript API

  • GLSL written in C like syntax

  • Heavily optimised for GPU

  • Vertex and Fragment Shaders

- List main concepts

- Display sample C code

- Explain how to link to it

Enter ThreeJs

Mr.Doob = Legend

  • Camera

  • Core - including Geometry

  • Lights

  • Materials

- Summarise basic API

ThreeJS basics

  • Stage, Grid, Geometry

  • Materials and lighting

  • Cameras

  • Normal = Rainbows!

- Open window in separate port

- Show code samples


Euclidean geometry

Here comes the science

Describing the motion of items in space

- Ask for discussion afterwards


Describe direction

From one point in space to another


Euler Angles

Describe the angle of rotation around each axis

Gimbal locking

The Problem of angles in calculating rotation

Pitch, Roll and Yaw

Tait-Bryan angles

  • Another form of Euler Angles

  • Commonly used in Aeronautics

Input device

Leap Motion

  • Near field mode & Finger detection

  • Low cost per unit

  • Very low latency

  • Mimics touch events

- Outline why Leap motion chosen. Can I show field of view? Is there a good Hand demo here?

Leap JS library

Javascript API for handling device data

  • Controller, Frame

  • Hand, Pointable

  • Gesture

  • Interaction Box

- Describe basic API methods

- Interaction Box

- Demo finger points on screen

Leap JS plugins

Limitations of using third party libs

  • Pros: Much more sophisticated

  • Cons: Tied to plugin architecture/Golden Handcuffs

Output device

Google Cardboard

Not a spoof of Google Glass

  • Pros: Very cheap lightweight VR headset

  • Cons: Uses Mobile device as screen - limitations of mobile

  • Beware magnets! They don't like accelerometers


Leap Websockets

Limitations of connecting USB data to mobile

  • Mobile device won't connect directly to Leap socket

  • Hemorrhaging Node servers

Socket server

Serving data over websockets

Leap Networking plugin

Leap plugin to pipe real time frame data

WebRTC architecture

Using client side webRTC


Demonstrating USB data ported to second screen

  • Connect to browser via WebRTC

Leap RTC in action

Photographic evidence

Virtual reality

VR simulation

Using mobile data from accelerometer

Signal processing

Smoothing accelerometer data

  • Sensor noise causes juddering

  • Low pass filters offer a simple averaging effect

  • Kalman filters are self correcting

  • Implementing smoothing causes latency in motion

VR mobile demo

Project future

Next steps

Remaining platform issues

  • Improve signal processing with Kalman filter

  • Improving latency in hand movement over WebRTC

  • Limitations of mobile: processing, battery, APIs

  • Experiment with VR APIS and Oculus rift