Interested in joining the Aero Glass team? Send your solution to the challenge described below to career@glass.aero!

The Setup

You have a smart glass, with two cameras built in, where the relative orientation of the cameras is unknown. One of the cameras is generally looking ‘forward’ (the same direction the glass users is looking), the other is generally looking ‘upwards’ (sees stuff above the users head).challenge

You have markers positioned in front of the user, and also above the users head. The relative orientation of the markers in relation to each other is unknown. In front of the user you have a single marker, above his head, you have multiple markers.

The forward looking camera is recording images of the single marker in front of the user, while the upward looking camera is recording images of the markers above the users head. There is a small time difference between the recordings. You’ve run marker detection on all of these images, and you have saved the rotation matrix & the translation vector for each of the markers in each image.

You group the measurements into pairs: one measurement from the forward looking camera (seeing a single marker), and one measurement from the upward looking camera (seeing multiple markers), with the time difference between these measurements being lower than some preset value. Such a pair will be referred to as ‘a measurement’, and contains, for each marker visible, the rotation matrix, the translation vector, time timestamp, and a reference to what camera is this coming from.

The Problem

Given measurements as described above, determine the relative rotation of the marker in front of the user and the markers above the users head.

The Solution

After having found the relative rotation, take marker data from the upward facing camera only, and calculate the orientation and location of the marker in front of the user. Verify the correctness of your solution by either or both:

  • calculating the error of the calculated marker in front of the user and the real marker measurements for the same marker
  • render the marker, or the axes of the marker, visually unto the images taken by the forward facing camera, and visually see if they match the marker as recorded in the image

The Data

Find test data for the above problem here. If you’ve downloaded our dataset earlier, an update generated with real-world marker sizes (correct translation vectors) can be found here.

The directory structure of the data is as follows:

  • camera0 – raw images and PNGs for the forward facing camera, each filename represents the timestamp of the image taken. images with the suffix ‘undistort’ represent images that have been ‘undistorted’ from the original image.
    • the file info.txt provides resolution and other info
    • the file x7_0_internal_camera.xml is the OpenCV camera intrinsic parameter file
  • camera1 – raw images and PNGs for the forward facing camera, each filename represents the timestamp of the image taken. images with the suffix ‘undistort’ represent images that have been ‘undistorted’ from the original image.
    • the file info.txt provides resolution and other info
    • the file x7_1_internal_camera.xml is the OpenCV camera intrinsic parameter file
  • tracking_values_XXXms._timeout.txt – measurement file, where XXXms is the maximum time difference between two images to form a usable ‘measurement’, a CSV with the following columns.
    • measurement number (1, 2, 3, …)
    • camera id:
      • 0 for the forward facing camera
      • 1 for the upward facing camera
    • marker number, a unique id for the marker that is described in this line. note: this is unique for the particular camera only
    • 3×3 rotation matrix, as 9 floating point values, matrix representation, row major order
    • translation vector, as 3 floating point values
    • timestamp, in milliseconds – the corresponding image file will have the same name
  • markerconfig – various images of the actual physical setup of the data sampling environment

Additional details:

  • the forward facing camera image is rotated by 90 degrees clockwise. it’s a ‘portrait’ camera in terms of physical orientation, but the image returned is rendered as landscape. look at the images in the markerconfig directory to understand the scenario
  • image files are presented both in ‘raw’ format and as PNG files. the command used to convert from raw to PNG was:
    • for camera 0: mia-raw2image -s 2048,1080 -i $FILE -o $FILE.png
    • for camera 1: mia-raw2image -s 640,480 -i $FILE -o $FILE.png
  • matrices are represented in row major order, that is, a matrix representation of 1, 2, 3, 4, 5, 6, 7, 8, 9 describes a matrix with the following rows:
    • 1, 2, 3
    • 4, 5, 6
    • 7, 8, 9
  • real world marker sizes were as follows:
    • camera0: 185mm on each side
    • camera1: 70mm on each side
  • K and Kf matrices that facilitate re-projection of the markers for camera0:
    • K:
    • -1.9329 -0.0077 1.0250
      0.0003 1.9536 0.5405
      0 0 0.0010
    • Kf:
    • 1.3366 0 1.0265
      0 1.3389 0.5684
      0 0 0.0010
  • K and Kf matrices that facilitate re-projection of the markers for camera1:
    • K:
    • -611.6784   -3.4124  322.1377
      1.0869  616.6464  239.9408
      0 0 1.0000
    • Kf:
    • 228.7900 0 310.1800
      0 227.5100 244.4900
      0 0 1.0000