Incorporating Hand-Writing Detection and Recognition into Physical-Digital Workflows and Workspaces

Status: ongoing

Runtime: 2018 -

Participants: Jürgen Hahn, Nicolas Symeou, Raphael Wimmer

Keywords: Interaction Techniques, Computer Vision, Machine Learning, Deep Learning


Enable the digitisation of hand-written annotations and signatures on physical paper or etc., in order to incorporate them into the digital twin for machine processing in regard of physical-digital affordances.

In order to optimally use the required technologies, the status quo of Machine Learning / Deep Learning in regard of Hand-Writing Detection and Recognition has to be familiarized with. Then, appropriate techniques are customized for the physical-digital use case. This requires custom collected data sets and a custom designed and trained neural network. Stroke coordinates, stroke progression and individual point coordinates of the strokes of letters, words, sentences serve as input for said trained NN. These are extracted via image processing and computer vision techniques. Ultimately, interaction techniques, familiar to users, are retained in the new digitised context of their workflow.

Used Technologies:

Relevant Literature:

Future Extensions: