Editing Projects:Improve tracking of individual markers and marker patterns

== Overview == 

The raw output of the Vicon system is a set of 3D point detections in every frame. On the individuals, the markers are arranged to form unique patterns on a 2D planar surface (think of a bunch of markers glued in a particular way onto a billboard).

The Vicon Nexus software detects these patterns within the point cloud on a per frame basis, while an additional tracker software assembles the per-frame detections into tracks, correcting some of the mistakes in per-frame detection. (TODO: is this indeed how it works, do the different Vicon modules something different? please explain, ideally also in the documentation on the Vicon data format). However, this process is not free of errors, and we thus would like to improve it.

== Contact ==

* Mathias Günther, mathias.guenther@uni-konstanz.de
* Mate Nagy, mnagy@orn.mpg.de
* Hemal Naik, hnaik@orn.mpg.de
* Bastian Goldluecke, bastian.goldluecke@uni-konstanz.de

== Aims ==

The key goal is to take the [[Vicon:Data format documentation|output of the Vicon system]] and further process it to improve track generation for the different marker patterns (individuals and their pose). Desired outcomes are:

* fill in gaps where only some but not all makers of the full pattern is detected
* smoothing of tracks
* filtering of certain artifacts (e.g. random pattern flips in the output of Nexus).

Desireable are both frameworks for high-quality offline data processing as well as an online/quasi-real time solution working directly on the data stream of 3D points from the system.

== Estimated level of difficulty ==

There are different subtasks to be solved with increasing degree of sophistication/difficulty.
* One option could be to first work on track assembly/completion for individual markers. This could already improve raw data quality enough that Nexus and the Vicon tracker produce better results.
* One could work on improving different subsystems first, such as per frame pattern detection hypotheses.
* Next step would be embedding per-frame pattern hypotheses into a state-of-the-art tracker to build tracks/pose of individuals.
* The most interesting approach would be to adapt a state-of-the-art tracker for individual object detections to pattern tracking, where the marker detections belong to certain rigid patterns, and some of a pattern's points can be missing on a per-frame basis (i.e. a per-frame pattern detection is not a required first step). The goal is then doing the assignment from detections to tracks of patterns.

== Provided data ==

See [[Vicon:Data format documentation]].



== Suggested approaches ==

See above. Further notes:

# Backpack points can be transferred to coordinate system of VICON using output of tracker (6DOF pose transformation between VICON and Backpack coordinate system). Tracker has better optimization techniques than nexus therefore trajectories are more complete. The result is compared with 3D points given by Nexus and a combined dataset can be created. 
# Results from step 1 can be compared with .csv of unlabeled points (Tracker output of 3D all detections) to label all detected 3D points and remove ghost points.
# The combined result from 1 and 1 can be used then to identify random flips in angles, positions and IDs.