→ Slide 1

Copy & Paste, Drag & Drop

Interaction Techniques and Technologies (ITT), SS 2025

Raphael Wimmer

→ Slide 2

These are slides/notes for the lecture, automatically generated from the slide set. Please extend this outline with your own notes.

→ Slide 3
↓ Slide 4
  1. Strive for consistency.
  2. Enable frequent users to use shortcuts.
  3. Offer informative feedback.
  4. Design dialog to yield closure.
  5. Offer simple error handling.
  6. Permit easy reversal of actions.
  7. Support internal locus of control.
  8. Reduce short-term memory load.

<small>Shneiderman, B. and Plaisant, C., Designing the User Interface: Strategies for Effective Human-Computer Interaction: Fifth Edition, Addison-Wesley Publ. Co., Reading, MA (2010), 606 pages. online </small>

↓ Slide 5
  • Clipboard, Drag and Drop
  • User Interfaces
  • Interactive exploration
  • Implementations in different operating systems
  • Clipboard managers
→ Slide 6
↓ Slide 7

Questions:

  • What is a clipboard?
  • How does the user interface for clipboards look like?
  • How can clipboard functionality be implemented?
↓ Slide 8

How does a clipboard behave? What do you expect from a typical clipboard?

↓ Slide 9
  • 3-4 interactions with the clipboard per hour (maximum: 20)[^stolee]
  • programmers: 16 interactions / hour[^kim]
  • copy: 45%, paste: 52%, cut: 3%[^stolee]
  • 80% of copied data is pasted later[^stolee]
  • majority of copy/paste operations restricted to ca. 5 applications per user[^stolee]
  • most copied from: browser[^stolee]
  • most pasted to: text editor[^stolee]
  • clipboard mostly accessed via shortcuts[^retzer]
  • context menu rarely used (exception: copying images)[^retzer]

[^stolee]: Stolee, Elbaum, & Rothermel (2009) Revealing the Copy and Paste Habits of End Users. [^retzer]: David Retzer (2017) “Implementierung eines Copy&Paste-Mechanismus mit Quellenattribuierung” (Master's Thesis). [^kim]: Kim, Bergman, Lau, & Notkin (2004) An Ethnographic Study of Copy and Paste Programming Practices in OOPL.

↓ Slide 10

Basic patterns[^stolee]


Complex patterns[^stolee]

[^stolee]: Stolee, Elbaum, & Rothermel (2009) Revealing the Copy and Paste Habits of End Users

↓ Slide 11

How does a clipboard work?

↓ Slide 12

~~~~

!/usr/bin/python3

from PyQt6.QtWidgets import QApplication

app = QApplication([“”,“”]) clipboard = app.clipboard() mimeData = clipboard.mimeData() print(“Current clipboard offers formats: ” + str(mimeData.formats())) for f in mimeData.formats():

  print("---- %s ----" % f)
  data = str(mimeData.data(f))
  if len(data) > 100:
      print(data[:100] + " [...]")
  else:
      print(data)
  print("")

~~~~

↓ Slide 13
↓ Slide 14
  • “Pasteboard” for historical reasons
  • manager process pbs accessed via APIs
  • Standard classes for clipboard content (NSString, NSAttributedString, NSImage, NSURL, NSColor, NSSound), general class NSPasteboardItem or custom classes adopting NSPasteBoardReading protocol
  • additional find buffer for text searches
↓ Slide 15
  • General Pasteboard and Find Pasteboard similar to macos
  • apps can create additional pasteboards
  • Uniform Type Identifier (UTI) describes type of data (e.g., public.jpeg or com.myCompany.myApp.myType)
  • Pasteboard class provides convenience methods for copying/pasting images, URLs, …
↓ Slide 16
  • Main difference to OS X and Windows: no data is actually stored in the clipboard
  • Instead:
    • On “copying” data to the clipboard, the application tells the X Server that it now owns the CLIPBOARD selection.
    • The previous owner of the CLIPBOARD selection gets notified.
    • When pasting data from the clipboard, the application directly requests the data from the application that owns the CLIPBOARD selection.
    • Content negotiation: pasting applications asks for data types the application can provide and can also try to request data in arbitrary format.
  • furthermore: PRIMARY buffer (holds selected text, pasted via middle click) and SECONDARY buffer (mostly unused), similar mode of operation as clipboard.
  • data types identified via MIME types

Further reading: explanation by Jamie Zawinski, Freedesktop.org clipboard "specification", ICCCM: Peer-to-Peer Communication by Means of Selections, X11: How does “the” clipboard work?

↓ Slide 17
  • Wayland: successor to X11 protocol
  • Similar to X11: source application creates a wl_data_source, receiving application sees it as wl_data_offer object.
  • Source application creates offer for each MIME type that is offered.
  • Receiving application sends selected MIME type and file descriptor to source application.
  • Source application writes data into file descriptor.

Further reading: Wayland protocol: data sharing, Wayland clipboard and drag & drop

↓ Slide 18
  • Clipboard holds one ClipData object at a time, consisting of:
    • ClipDescription object containing list of MIME types
    • one or more ClipData.Items, all having the same type: text, URI, or Intent
  • Content providers allow for retrieving data with a specific MIME type via a URI
  • Usage: [ClipboardManager](http://developer.android.com/reference/android/content/ClipboardManager.html).setPrimaryClip(ClipData)
↓ Slide 19
↓ Slide 20
  • listen to changes in clipboard state (X11: needs to grab ownership of CLIPBOARD selection)
  • add additional features
    • persist history
    • select format to paste
    • modify data
    • add formats (e.g., OCR)
    • shortcuts for managing clipboard contents
↓ Slide 21

Example: Citrine (Stylos et al. 2004)

↓ Slide 22
  • "Copy-paste conclusions: put the metadata on the clipboard"
    • “Step by step we’ve been teasing apart the clipboard to see how metadata could survive a copy-paste between applications. If the metadata survives, then the destination can be used to automatically credit the original source and the creator, without the user having to do it manually.” "On Clipboard Formats – 2006-09-15"
    • “The Carbon version of Gecko doesn’t interoperate with anything but other Carbon Gecko processes. I figured I should try to do better with the Cocoa nsClipboard.

    This stuff is so underdocumented that it isn’t even funny. This document is written so that others might find something when they search the Web.”*

↓ Slide 23
  • Windows: Object Linking and Embedding (OLE)
  • OS X: Cocoa Drag Manager
  • generally similar to clipboard operation
  • source application may offer multiple formats
  • on drag start: change cursor to indicate dragged object
  • when dragging into another window: receiving window can accept or reject drag
  • data is only copied on release of mouse button
↓ Slide 24
↓ Slide 25
  • Try out the clipboard and drag-and-drop examples
  • Find out what kind of data different applications offer on different operating systems
  • Write a small tool that intelligently modifies the clipboard contents (e.g., removing all passwords from plain text)
↓ Slide 26
  • Different clipboard implementations across operating systems
  • Data often available in different formats
  • X11, Wayland have an asynchronous clipboard
  • format negotiation in X11 is quite a mess
→ Slide 27