~~REVEAL~~

====== Copy & Paste, Drag & Drop ======

Interaction Techniques and Technologies (ITT), SS 2025

Raphael Wimmer

===== Overview =====

<div class="notes">
These are slides/notes for the lecture, automatically generated from the slide set. 
Please extend this outline with your own notes.
</div>

===== Common Interaction Techniques and Implementations =====

==== Shneiderman's Eight Golden Rules of Interface Design ====

1. Strive for consistency.
2. Enable frequent users to use shortcuts. 
3. Offer informative feedback. 
4. Design dialog to yield closure.
5. Offer simple error handling. 
6. Permit easy reversal of actions.
7. Support internal locus of control. 
8. Reduce short-term memory load.

<small>Shneiderman, B. and Plaisant, C., Designing the User Interface: Strategies for Effective Human-Computer Interaction: Fifth Edition, Addison-Wesley Publ. Co., Reading, MA (2010), 606 pages. [online](http://www.cs.umd.edu/~ben/goldenrules.html) </small>

==== Overview ====

* Clipboard, Drag and Drop
* User Interfaces
* Interactive exploration
* Implementations in different operating systems
* Clipboard managers


===== Clipboard =====

==== General Concept ====

Questions:

* What is a *clipboard*?
* How does the user interface for clipboards look like?
* How can clipboard functionality be implemented?


==== Buzz Group ====

How does a clipboard *behave*? What do you expect from a typical clipboard?

==== Clipboard Usage ====

* 3-4 interactions with the clipboard per hour (maximum: 20)[^stolee]
* programmers: 16 interactions / hour[^kim]
* copy: 45%, paste: 52%, cut: 3%[^stolee]
* 80% of copied data is pasted later[^stolee]
* majority of copy/paste operations restricted to ca. 5 applications per user[^stolee]
* most copied from: browser[^stolee]
* most pasted to: text editor[^stolee]
* clipboard mostly accessed via shortcuts[^retzer]
* context menu rarely used (exception: copying images)[^retzer]

[^stolee]: Stolee, Elbaum, & Rothermel (2009) Revealing the Copy and Paste Habits of End Users.
[^retzer]: David Retzer (2017) "Implementierung eines Copy&Paste-Mechanismus mit Quellenattribuierung" (Master's Thesis).
[^kim]: Kim, Bergman, Lau, & Notkin (2004) An Ethnographic Study of Copy and Paste Programming Practices in OOPL.


==== Clipboard Usage Patterns ====

![Basic patterns[^stolee]](img/clipboard_patterns_1.jpg)

----

![Complex patterns[^stolee]](img/clipboard_patterns_2.jpg)

[^stolee]: Stolee, Elbaum, & Rothermel (2009) Revealing the Copy and Paste Habits of End Users

==== Buzz Group ====

How does a clipboard *work*? 

 
==== Qt Clipboard inspector ====

~~~~ 
#!/usr/bin/python3
from PyQt6.QtWidgets import QApplication

app = QApplication(["",""])
clipboard = app.clipboard()
mimeData = clipboard.mimeData()
print("Current clipboard offers formats: " + str(mimeData.formats()))
for f in mimeData.formats():
    print("---- %s ----" % f)
    data = str(mimeData.data(f))
    if len(data) > 100:
        print(data[:100] + " [...]")
    else:
        print(data)
    print("")
~~~~


==== Microsoft Windows ====

* [Clipboard formats](https://msdn.microsoft.com/en-us/library/windows/desktop/ms649013(v=vs.85).aspx) describe the type of data in the clipboard
* Clipboard may contain the same data in different formats (e.g., as plain text, HTML, RTF)
* [Standard formats](https://msdn.microsoft.com/en-us/library/windows/desktop/ff729168(v=vs.85).aspx) 
  (CF_BITMAP, CF_UNICODETEXT, ...)
* Registered formats (custom, for data exchange between applications, e.g., [CF_HTML](https://msdn.microsoft.com/en-us/library/aa767917(VS.85).aspx)
* Private formats (for internal use)
* Access via [SetClipBoardData(format, data)](http://msdn.microsoft.com/en-us/library/windows/desktop/ms649051%28v=vs.85%29.aspx), [IsClipBoardFormatAvailable(format)](http://msdn.microsoft.com/en-us/library/windows/desktop/ms649047(v=vs.85).aspx), [EnumClipboardFormats](http://msdn.microsoft.com/en-us/library/windows/desktop/ms649038(v=vs.85).aspx), [GetClipBoardData()](http://msdn.microsoft.com/en-us/library/windows/desktop/ms649039(v=vs.85).aspx)
* Clipboard needs to be explicitly cleared before copying data to it.
* Further Reading: "How the clipboard works" ([Part 1](http://blogs.msdn.com/b/ntdebugging/archive/2012/03/16/how-the-clipboard-works-part-1.aspx), [Part 2](http://blogs.msdn.com/b/ntdebugging/archive/2012/03/29/how-the-clipboard-works-part-2.aspx))


==== Mac OS X / macos ==== 

* "Pasteboard" for historical reasons
* manager process `pbs` accessed via APIs
* Standard classes for clipboard content (`NSString, NSAttributedString, NSImage, NSURL, NSColor, NSSound`), general class `NSPasteboardItem` or custom classes adopting `NSPasteBoardReading` protocol
* additional *find buffer* for text searches
* see also: [pasteboard programming guide](https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/PasteboardGuide106/Introduction/Introduction.html)

==== iOS ====

* *General Pasteboard* and *Find  Pasteboard* similar to macos
* apps can create additional pasteboards
* `Uniform Type Identifier` (UTI) describes type of data (e.g., `public.jpeg` or `com.myCompany.myApp.myType`)
* `Pasteboard` class provides convenience methods for copying/pasting images, URLs, ...
* ([documentation](https://developer.apple.com/library/prerelease/ios/documentation/UIKit/Reference/UIPasteboard_Class/index.html))


==== X11 (Linux/BSD) ==== 

* Main difference to OS X and Windows: no data is actually stored in the clipboard
* Instead:
    * On "copying" data to the clipboard, the application tells the X Server that it now owns the `CLIPBOARD` selection.
    * The previous owner of the `CLIPBOARD` selection gets notified.
    * When pasting data from the clipboard, the application directly requests the data from the application that owns the `CLIPBOARD` selection.
    * Content negotiation: pasting applications asks for data types the application can provide and can also try to request data in arbitrary format.
* furthermore: `PRIMARY` buffer (holds selected text, pasted via middle click) and `SECONDARY` buffer (mostly unused), similar mode of operation as clipboard.
* data types identified via [MIME types](https://en.wikipedia.org/wiki/MIME#Content-Type)

Further reading: [explanation by Jamie Zawinski](http://www.jwz.org/doc/x-cut-and-paste.html), [Freedesktop.org clipboard "specification"](http://standards.freedesktop.org/clipboards-spec/clipboards-0.1.txt), [ICCCM: Peer-to-Peer Communication by Means of Selections](http://tronche.com/gui/x/icccm/sec-2.html#s-2), [X11: How does “the” clipboard work?](https://www.uninformativ.de/blog/postings/2017-04-02/0/POSTING-en.html)


==== Wayland (Linux) ====

* Wayland: successor to X11 protocol
* Similar to X11: source application creates a `wl_data_source`,  receiving application sees it as `wl_data_offer` object.
* Source application creates `offer` for each MIME type that is offered.
* Receiving application sends selected MIME type and file descriptor to source application.
* Source application writes data into file descriptor.

Further reading: [Wayland protocol: data sharing](https://wayland.freedesktop.org/docs/html/ch04.html#sect-Protocol-data-sharing), [Wayland clipboard and drag & drop
](https://emersion.fr/blog/2020/wayland-clipboard-drag-and-drop/)


==== Android ====

* Clipboard holds one `ClipData` object at a time, consisting of:
    * `ClipDescription` object containing list of MIME types
    * one or more `ClipData.Item`s, all having the same type: text, URI, or Intent
* *Content providers* allow for retrieving data with a specific MIME type via a URI
* Usage: `[ClipboardManager](http://developer.android.com/reference/android/content/ClipboardManager.html).setPrimaryClip(ClipData)`
* [facilities for copying data structures and streams](http://developer.android.com/guide/topics/text/copy-paste.html)


==== Qt Implementation ====

* Wrappers for every [platform](http://code.qt.io/cgit/qt/qtbase.git/tree/src/plugins/platforms) (e.g., [X11](http://code.qt.io/cgit/qt/qtbase.git/tree/src/plugins/platforms/xcb/qxcbclipboard.cpp), [Windows](http://code.qt.io/cgit/qt/qtbase.git/tree/src/plugins/platforms/windows/qwindowsclipboard.cpp), [OS X](http://code.qt.io/cgit/qt/qtbase.git/tree/src/plugins/platforms/cocoa/qmacclipboard.mm))
* [QClipboard](http://doc.qt.io/qt-6/qclipboard.html)
* data type indicated via MIME


==== Clipboard Managers ==== 

* listen to changes in clipboard state (X11: needs to grab ownership of `CLIPBOARD` selection)
* add additional features
  * persist history
  * select format to paste
  * modify data
  * add formats (e.g., OCR)
  * shortcuts for managing clipboard contents
  * ...
* See also: [CopyQ](http://hluk.github.io/CopyQ/), [Wikipedia: Clipboard Manager](https://en.wikipedia.org/wiki/Clipboard_manager)


==== Academic research on clipboards ====


![Example: Citrine (Stylos et al. 2004)](img/citrine.jpg)



==== Further Reading ====

* ["Copy-paste conclusions: put the metadata on the clipboard"](http://commonsmachinery.se/2013/10/copy-paste-conclusions-put-the-metadata-on-the-clipboard/)
  *"Step by step we’ve been teasing apart the clipboard to see how metadata could survive a copy-paste between applications. If the metadata survives, then the destination can be used to automatically credit the original source and the creator, without the user having to do it manually."*
* ["On Clipboard Formats – 2006-09-15"](https://hsivonen.fi/kesakoodi/clipboard/)
  *"The Carbon version of Gecko doesn’t interoperate with anything but other Carbon Gecko processes. I figured I should try to do better with the Cocoa nsClipboard.
  This stuff is so underdocumented that it isn’t even funny. This document is written so that others might find something when they search the Web."*


==== Drag and Drop ====

* X11: [DND extension](http://jjlindal.net/jafl/xdnd/)
* Windows: Object Linking and Embedding (OLE)
* OS X: Cocoa Drag Manager
* generally similar to clipboard operation
* source application may offer multiple formats
* on drag start: change cursor to indicate dragged object
* when dragging into another window: receiving window can accept or reject drag
* data is only copied on release of mouse button

==== DnD Qt Implementation ==== 

* [QDrag, QDrag{Enter,Leave,Move}Event, QDropEvent](http://doc.qt.io/qt-6/dnd.html)
* again: wrappers with OS-specific code
* MIME-encoded content (like X11), gets converted for Windows and OS X.
* Code example: [tutorial at zetcode.com](http://zetcode.com/gui/pyqt5/dragdrop/) and `dragdrop_example.py` in GRIPS.

==== Your Tasks ====

* Try out the clipboard and drag-and-drop examples
* Find out what kind of data different applications offer on different operating systems
* Write a small tool that intelligently modifies the clipboard contents (e.g., removing all passwords from plain text)

==== Recap ====

* Different clipboard implementations across operating systems
* Data often available in different formats
* X11, Wayland have an asynchronous clipboard
* format negotiation in X11 is quite a mess 

===== END =====