Status: ongoing

Runtime: 2017 -

Participants: Andreas Schmid, Raphael Wimmer, Dennis Schüsselbauer, Laurin Muth

Keywords: Position, Tracking, Pattern, Mouse


The absolute position of a device on a two dimensional surface like a table is to be tracked autonomously by scanning a dot pattern with the PMW3360 mouse sensor.


There is a working prototype that can track its relative movements and can discern between a few distinct patterns which allows for limited use of absolute tracking. For now the patterns that can be discerned by the sensor are recognized by the average amount of white and black as well as the automatic adjusting shutter speed which in turn is affected by the brightness.

Future/Current work

We are working on analysing the raw frame of the mouse sensor to recognizing a actual dot pattern. The pattern will most likely use some kind of two-dimensional De Bruijn matrix to be able to recognize the absolute position.


The prototype is a M5Stack Fire - a stackable microcontroller (ESP32) with different peripheries. It is powered by a battery module and the custom module at the bottom holds the mouse sensor and provides the tracking facilities.


There are two demos which show the current capabilities of the projekt.

The first one is a showcase of the interaction possibilities that the tracking methods provide. The three applications are activated through the recognition of one of three corresponding patterns. There is a selection application where you can change two characters by moving the device vertically or horizontally. Another is a magic lens application that lets you search a picture within a narrow window which can be controlled with the movement of the device. The last application shows a stream of the raw frames captured by the sensor and basically make for magnifying glass.

The second demo involves two devices which track their relative movement and send it to each other (over WiFi). Based on the exchanged data you can calculate the bearing and distance of each other. This is visualized as a googly eye which stares towards the paired device and a proximity indicator to alert of a nearby device. The three patterns work as known points in space and allow to readjust the devices to an absolute position.

Video for our MuC demo paper. The device shown in this video can distinguish between several surfaces and start different applications depending on them. By moving the device on the surface, those applications can be controlled.


The sensor of a digital mouse (PMW3360) can be used to capture a small raw image (Fig. 01) of the surface underneath it. Combined with a dot pattern on the surface, it might be possible to use this image to track the absolute position of the sensor.

Figure 01: Example for a raw image captured by the sensor.


Andreas Schmid, Raphael Wimmer, Dennis Schüsselbauer, Laurin Muth

Workshopband Mensch und Computer 2018

Development of a system for tracking the 2D position of tangible blocks and their spatial relations (Tweet this with link)


News / Blog