I - Context:

The TrackCap project is a system in development which enables smartphones to be used as 3D interaction devices for Augmented and Virtual Reality applications on mobile head mounted displays (HMD).
The system consists of a planar tracking target, which is attached to the HMD, tracking software which runs on the smartphone and estimates the 6DOF pose of the phone relative to the HMD, and a network interface that transmits the estimated pose of the phone to the HMD.
As the HMD estimates its position and orientation in 3D world space, the relative position and orientation of the smartphone is mapped into the world coordinate system as well. This enables interactions of the smartphone with virtual objects that are displayed in the HMD

Figure 1. Overview of the TrackCap application

Figure 2. System Overview. (a) Our system consists of 10: HMD, 11: detachable marker mount, 12: planar fiducial marker, 13: camera, 14: smartphone. In addition, the system uses a 6DOF pose estimator on the smartphone, a network module to transmit the estimated pose to the HMD (15), and a pose combiner running on the HMD, which maps the pose of the smartphone into the world coordinate frame. (b) We designed a set of 3D printable parts to mount TrackCap to a variety of HMDs. I) Microsoft HoloLens.

Figure 3. Study setup to measure performance during the selection of distant objects. 

II - Tasks to be performed:

• Integration and testing on Microsoft HoloLens glasses
• Development of an AR Application in Unity3D
• Integration and testing on Microsoft HoloLens glasses
• Development of network logic in UNet
• Development of a mobile application for Android
• 3D modeling in Blender and Maya
• Writing of a tutorial
• Design of a HoloLens-compatible LED prototype