About the research group

(this description is taken from the initial project proposal and will be updated over time)

The term 'affordances', originally coined by Gibson (1977) and later appropriated by Norman (1988, 2004), describes the properties of an object which determine how it can be used. For example, a physical book 'affords' sequential reading, fast random access, annotations, and various other uses. It does not afford several other uses, such as full-text search, concurrent multi-user access, or fast copy and paste. With the ongoing digitisation of physical media, books, files, folders, music records, and many other specialised storage media are being replaced by digital counterparts. This impacts how users interact with information and media, how organizations handle document workflows, and how people notice and discover information.

1. Interaction with Digital Media

Digitisation of artifacts, documents, and workflows frees them of physical limitations but also strips away the inherent affordances of physical media. While physical objects inherently have certain affordances, digital documents (respectively, their user interfaces) do not automatically possess affordances. Instead, developers have to design and implement all intended affordances and provide 'signifiers' that indicate to the user which options are available for interacting with the document. Therefore, digital tools and media usually have a different and much more limited set of affordances than their physical counterparts, This way, digitalisation limits human expressiveness by default.

2. Document Workflows

Digitalisation uniquely affects public administration. For centuries, administrations have optimized how they handle information stored on paper, developing a wealth of practices for efficient annotations and workflow management. Digitalisation rapidly replaces these cultural techniques with off-the-shelf GUI widgets. When digitising workflows, many institutions focus on efficiently storing and retrieving documents - the most obvious cost factors. However, in many cases creating and reading documents becomes less efficient.

3. Serendipity and Awareness

Often we come in contact with information by accident or peripherally, Physical media inherently affords serendipity and peripheral awareness. A person's book shelf gives visitors an opportunity to accidentally discover new authors and learn about the owner's taste. On a desk, one might stumble upon old notes which are surprisingly relevant for the task at hand. As the media that shape our view of the world are moved from shelves into clouds, they no longer afford serendipity. A similar effect is taking place with social networks and the Internet of Things. As digitalisation rips out physical limitations of electromechanical and social relations, old mental models fail. We no longer feel in control.

Despite the obvious advantages of digitalisation, we might be losing something by thoughtlessly discarding physical media - and should try to retain some of their affordances in our brave new digital world.

The underlying core question I want to investigate within the next five years is: “How can we keep or introduce useful affordances of physical media in purely digital representations - and how can we enhance physical paper in order to better support digital workflows?”

More specifically, I want to answer three sets of questions:

  1. How do people utilize affordances of physical media - especially paper - in various contexts; which affordances get lost when moving from traditional physical media to digital representations; and which affordances of digital and physical media are so useful that they should be added to their counterparts?
  2. How can we integrate physical paper in digital document workflows; how can we leverage physical affordances for interaction with digital information; and how can we make sure that our solutions actually get used?
  3. How can physical representations of digital information facilitate serendipity and give users a sense of control over their data and the Internet of Things surrounding them?

Since computers entered offices and living rooms, inventors and scientists have researched ways of replacing paper documents and workflows with digital solutions. As Sellen and Harper (2001) discuss in their seminal book “The myth of the paperless office”, physical paper offers many affordances that are hard to replicate digitally. Therefore, they argue that the the role of paper for storing information will shrink but that paper will still remain an important tool for interacting with information. In the late 1990's and early 2000's, researchers started looking at tablet computers and pen computing as ways for replacing or augmenting paper. Steimle (2009) gives a good overview of research in this area and identifies five key affordances of paper which would need to be supported by digital alternatives: ease of navigation, intuitive annotation, flexible spatial organization, collaboration and mutual awareness. Many studies on paperless or hybrid physical-digital document workflows have been conducted over the years, focusing e.g., on air traffic controllers (MacKay 1999), scientists' lab notebooks (Tabard et al. 2008; Roubert and Perry, 2015), libraries (Gebhardt. et al. 2014) or medical records (Houben et al. 2015).

The conclusion by Houben et al. (2015) is representative of findings in many other publications on digitisation of paper-based workflows: “[…] a paper-based medical record is key in the subtle coordination inside the ward as its physical form helps to both achieve local coordination and awareness, as well as facilitates micro- and macro-mobility. These results echo previous findings, but our study highlighted that these paper-based affordances are not transferred to the electronic medical record used in the hospital.”.

An important challenge for researchers and practitioners who want to design usable paperless or hybrid workflows is a lack of hard- and software that offers and extends the affordances of paper (Yeh 2007; Bouck-Standen 2016; personal communication). This is also suggested by Steimle (2009) who identifies several directions for future research which include e-ink, replicating further affordances of paper, and better support for complex tasks and collaboration. Nevertheless, commercial solutions for document workflows - which are currently being rolled out in Bavarian public administration - do not offer any of the affordances described by Steimle.

However, in recent years, several technical and methodological advances have occurred, sparking new interest in physical-digital paper-based and paperless workflows. In his dissertation, Brade (2015) shows that and how digital tools can replace paper in certain cases when optimised properly. Embedded, wireless platforms (such as the ESP8266) have become cheaper and more powerful, making the Internet of Things significantly more accessible. Electronic ink technologies have matured, enabling thin, flexible, high-resolution displays. Steimle (2015) argues that printable electronics have become cheap enough and can be manufactured with little effort, enabling many novel affordances of printable media. Abdelrahman et al. (2015) present examples of interactive forms enabled by printable electronics. Sweeney et al. (2016) demonstrate new approaches for making interactive electronic paper.

In summary, over the last decades researchers have identified advantages and limitations of physical and digital documents in specific contexts. Practical implementations of digital paper were hindered by a lack of sufficiently paper-like hard- and software. However, in recent years, fabrication techniques and display technologies have evolved which might allow developing truly digital paper.

We will address the three research questions following the 'design science' approach commonly used in HCI research (Peffers et al., 2007; MacKay and Fayard, 1997). In multiple iterations we will investigate real-world usage and challenges of physical and digital workflows, develop prototypical solutions, conduct appropriate evaluations, and refine our understanding of the topic While the first steps are very clear, we will adjust focus and methods of our research based on new insights or challenges. Initially we will start with the following approaches for answering the research questions:

1. Understanding Affordances

In order to create a comprehensive list of existing and lost affordances of physical and digital documents, we will conduct observational field studies, review literature, and organize focus groups. We will especially focus on public administration as an important and under-investigated user group. Later on, we will also explore how our partners and other users work with the improved tools which we developed.

2. Creating Affordances Through Novel Hard- and Software

We will explore different approaches for digitally enhancing paper while retaining its unique affordances. Initially, we will design and build custom thermochromic and electrophoretic displays with integrated sensing and communication, explore their properties, and implement interaction techniques such as copy&paste, undo, or digital signatures for them. In parallel, we will design digitally-augmented furniture that supports organizing paper documents and collaboration as well as traditional computing.

3. Serendipity and Awareness

We will conduct studies investigating how users discover interesting information in physical media collections, such as bookshelves, CD collections, or paper piles - and how discovery happens in digital collections. Furthermore, we will explore users' mental models of the Internet of Things, e.g., how switches, sensors, displays, and actuators are connected in their opinion. Based on our findings, we will explore physical and digital solutions for supporting serendipity in digital collections and awareness of digital connections.

We plan to make the following academic and practical contributions:

  • Our partners have working solutions and a better understanding of how to address the challenges of digitalisation
  • Our guidelines and heuristics help practitioners to implement usable workflows for digital documents
  • Our hardware and software designs push the state of the art and serve as basis for commercial and open-source products that are of use for researchers and practitioners.
  • Our publications raise new awareness for the role of affordances in interacting with digital media and show generic ways to think about these and implement them..

We will take the following six important measures to ensure timely and enduring practical and academic impact of our work:

  • We will aid our project partners in maintaining and further developing their prototypes.
  • We will investigate experimental systems that could be deployed in real settings within five to ten years.
  • We will conduct our research as 'open science', continuously publishing our ideas, experiments, and findings online. This will reduce duplication and speed up dissemination of our findings.
  • We will leverage existing 'open-source' software and hardware, contribute to it, and release own software and hardware under permissive licenses
  • We will strive to publish all work as 'open access' articles at top conferences (CHI, UIST, UbiComp, TEI).
  • We will work with experienced hardware and software developers to make selected parts of our systems commercially available
  • Abdelrahman, Y., Kubitza, T., Wolf, K., Pohl, N., & Schmidt, A. (2015). Tick that Box: Interactive Paper Documents. In Human-Computer Interaction (pp. 376-383). Springer International Publishing.
  • Bouck-Standen, D., Schwandt, M., Winkler, T., & Herczeg, M. (2016). ELBlocks – An Interactive Semantic Learning Platform for Tangibles. In Procedings of the Workshop 'Be-greifbare Interaktion' in Conjunction with MuC '16.
  • Brade, M. (2015). Visualization Methods for Interactive Gathering and Structuring of Information in the Context of Knowledge Modeling. Dissertation. TU Dresden
  • Gebhardt, C., Rädle, R., & Reiterer, H. (2014). Employing Blended Interaction to Blend the Qualities of Digital and Physical Books. i-com, 13(3), 36-42.De Gruyter
  • Gibson, J. J. (1977). The Theory of Affordances. In Perceiving, Acting, and Knowing, edited by Robert Shaw and John Bransford. Lawrence Erlbaum Associates
  • Houben, S., Frost, M., & Bardram, J. E. (2015). Collaborative affordances of hybrid patient record technologies in medical work. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 785-797). ACM.
  • Mackay, W. E., & Fayard, A. L. (1997). HCI, natural science and design: a framework for triangulation across disciplines. In Proceedings of the 2nd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 223-234). ACM.
  • MacKay, W. E. (1999). Is paper safer? The role of paper flight strips in air traffic control. ACM Transactions on Computer-Human Interaction (TOCHI), 6(4), 311-340.
  • Norman, D.A. (1988). The Psychology of Everyday Things. Basic books.
  • Norman, D. A. (2004). “Affordances and Design.” http://www.jnd.org/dn.mss/affordances_and.html
  • Roubert, F., & Perry, M. (2013). Putting the lab in the lab book: supporting coordination in large, multi-site research. In Proceedings of the 27th International BCS Human Computer Interaction Conference (p. 12). British Computer Society.
  • Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of management information systems, 24(3), 45-77.
  • Steimle, J. (2009). Integrating printed and digital documents : interaction models and techniques for collaborative knowledge work. Dissertation. TU Darmstadt
  • Steimle, J. (2015). Printed electronics for human-computer interaction. interactions 22, 3 (April 2015), 72-75. DOI=http://dx.doi.org/10.1145/2754304
  • Sweeney, D., Chen, N., Hodges, S., & Grosse-Puppendahl, T. (2016). Displays as a Material: A Route to Making Displays More Pervasive, in IEEE Pervasive Computing, vol. 15, no. 3, pp. 77-82, July-Sept. 2016. doi: 10.1109/MPRV.2016.56
  • Tabard, A., Mackay, W. E., & Eastmond, E. (2008, November). From individual to collaborative: the evolution of prism, a hybrid laboratory notebook. In Proceedings of the 2008 ACM conference on Computer supported cooperative work (pp. 569-578). ACM.
  • Yeh, R. B., Klemmer, S. R., & Paepcke, A. (2007). Design and Evaluation of an Event Architecture for Paper UIs: Developers Create by Copying and Combining. Stanford University Computer Science Department Technical Report.