→ Slide 1

Copy & Paste, Drag & Drop

Interaction Techniques and Technologies (ITT), SS 2025

Raphael Wimmer

→ Slide 2

Overview

These are slides/notes for the lecture, automatically generated from the slide set. Please extend this outline with your own notes.

→ Slide 3

Common Interaction Techniques and Implementations

↓ Slide 4

Shneiderman's Eight Golden Rules of Interface Design

  1. Strive for consistency.
  2. Enable frequent users to use shortcuts.
  3. Offer informative feedback.
  4. Design dialog to yield closure.
  5. Offer simple error handling.
  6. Permit easy reversal of actions.
  7. Support internal locus of control.
  8. Reduce short-term memory load.

<small>Shneiderman, B. and Plaisant, C., Designing the User Interface: Strategies for Effective Human-Computer Interaction: Fifth Edition, Addison-Wesley Publ. Co., Reading, MA (2010), 606 pages. online </small>

↓ Slide 5

Overview

→ Slide 6

Clipboard

↓ Slide 7

General Concept

Questions:

↓ Slide 8

Buzz Group

How does a clipboard behave? What do you expect from a typical clipboard?

↓ Slide 9

Clipboard Usage

[^stolee]: Stolee, Elbaum, & Rothermel (2009) Revealing the Copy and Paste Habits of End Users. [^retzer]: David Retzer (2017) “Implementierung eines Copy&Paste-Mechanismus mit Quellenattribuierung” (Master's Thesis). [^kim]: Kim, Bergman, Lau, & Notkin (2004) An Ethnographic Study of Copy and Paste Programming Practices in OOPL.

↓ Slide 10

Clipboard Usage Patterns

Basic patterns[^stolee]


Complex patterns[^stolee]

[^stolee]: Stolee, Elbaum, & Rothermel (2009) Revealing the Copy and Paste Habits of End Users

↓ Slide 11

Buzz Group

How does a clipboard work?

↓ Slide 12

Qt Clipboard inspector

~~~~

!/usr/bin/python3

from PyQt6.QtWidgets import QApplication

app = QApplication([“”,“”]) clipboard = app.clipboard() mimeData = clipboard.mimeData() print(“Current clipboard offers formats: ” + str(mimeData.formats())) for f in mimeData.formats():

  print("---- %s ----" % f)
  data = str(mimeData.data(f))
  if len(data) > 100:
      print(data[:100] + " [...]")
  else:
      print(data)
  print("")

~~~~

↓ Slide 13

Microsoft Windows

↓ Slide 14

Mac OS X / macos

  • “Pasteboard” for historical reasons
  • manager process pbs accessed via APIs
  • Standard classes for clipboard content (NSString, NSAttributedString, NSImage, NSURL, NSColor, NSSound), general class NSPasteboardItem or custom classes adopting NSPasteBoardReading protocol
  • additional find buffer for text searches
↓ Slide 15

iOS

  • General Pasteboard and Find Pasteboard similar to macos
  • apps can create additional pasteboards
  • Uniform Type Identifier (UTI) describes type of data (e.g., public.jpeg or com.myCompany.myApp.myType)
  • Pasteboard class provides convenience methods for copying/pasting images, URLs, …
↓ Slide 16

X11 (Linux/BSD)

  • Main difference to OS X and Windows: no data is actually stored in the clipboard
  • Instead:
    • On “copying” data to the clipboard, the application tells the X Server that it now owns the CLIPBOARD selection.
    • The previous owner of the CLIPBOARD selection gets notified.
    • When pasting data from the clipboard, the application directly requests the data from the application that owns the CLIPBOARD selection.
    • Content negotiation: pasting applications asks for data types the application can provide and can also try to request data in arbitrary format.
  • furthermore: PRIMARY buffer (holds selected text, pasted via middle click) and SECONDARY buffer (mostly unused), similar mode of operation as clipboard.
  • data types identified via MIME types

Further reading: explanation by Jamie Zawinski, Freedesktop.org clipboard "specification", ICCCM: Peer-to-Peer Communication by Means of Selections, X11: How does “the” clipboard work?

↓ Slide 17

Wayland (Linux)

  • Wayland: successor to X11 protocol
  • Similar to X11: source application creates a wl_data_source, receiving application sees it as wl_data_offer object.
  • Source application creates offer for each MIME type that is offered.
  • Receiving application sends selected MIME type and file descriptor to source application.
  • Source application writes data into file descriptor.

Further reading: Wayland protocol: data sharing, Wayland clipboard and drag & drop

↓ Slide 18

Android

  • Clipboard holds one ClipData object at a time, consisting of:
    • ClipDescription object containing list of MIME types
    • one or more ClipData.Items, all having the same type: text, URI, or Intent
  • Content providers allow for retrieving data with a specific MIME type via a URI
  • Usage: [ClipboardManager](http://developer.android.com/reference/android/content/ClipboardManager.html).setPrimaryClip(ClipData)
↓ Slide 19

Qt Implementation

↓ Slide 20

Clipboard Managers

  • listen to changes in clipboard state (X11: needs to grab ownership of CLIPBOARD selection)
  • add additional features
    • persist history
    • select format to paste
    • modify data
    • add formats (e.g., OCR)
    • shortcuts for managing clipboard contents
↓ Slide 21

Academic research on clipboards

Example: Citrine (Stylos et al. 2004)

↓ Slide 22

Further Reading

  • "Copy-paste conclusions: put the metadata on the clipboard"
    • “Step by step we’ve been teasing apart the clipboard to see how metadata could survive a copy-paste between applications. If the metadata survives, then the destination can be used to automatically credit the original source and the creator, without the user having to do it manually.” "On Clipboard Formats – 2006-09-15"
    • “The Carbon version of Gecko doesn’t interoperate with anything but other Carbon Gecko processes. I figured I should try to do better with the Cocoa nsClipboard.

    This stuff is so underdocumented that it isn’t even funny. This document is written so that others might find something when they search the Web.”*

↓ Slide 23

Drag and Drop

  • Windows: Object Linking and Embedding (OLE)
  • OS X: Cocoa Drag Manager
  • generally similar to clipboard operation
  • source application may offer multiple formats
  • on drag start: change cursor to indicate dragged object
  • when dragging into another window: receiving window can accept or reject drag
  • data is only copied on release of mouse button
↓ Slide 24

DnD Qt Implementation

↓ Slide 25

Your Tasks

  • Try out the clipboard and drag-and-drop examples
  • Find out what kind of data different applications offer on different operating systems
  • Write a small tool that intelligently modifies the clipboard contents (e.g., removing all passwords from plain text)
↓ Slide 26

Recap

  • Different clipboard implementations across operating systems
  • Data often available in different formats
  • X11, Wayland have an asynchronous clipboard
  • format negotiation in X11 is quite a mess
→ Slide 27

END