Creative professionals and the two-step process for developing ideas
Leading up to this project, our team interviewed dozens of creative professionals such as professors, authors, film-makers, and web designers. Although the details of how these pros work are different, a pattern emerged in this research: they all live or die by the strength of their ideas.
But ideas aren’t summoned from nowhere: they come from raw material, other ideas or observations about the world. Hence a two-step creative process: collect raw material, then think about it. From this process comes pattern recognition and eventually the insights that form the basis of novel ideas.
The collection step is gathering raw material in the form of sketches, photos, websites, text excerpts, and manuscripts in a single location. This can happen all at once in a binge-research session or it can be a slow curation over months or years.
In the thinking step, the creative pro sifts and sorts their collected raw material. They look at it from different angles, rearrange, try to get new perspectives. They are looking for patterns that can lead to fresh insights.
Next, we investigate the tools used by creative professionals in this collect-then-think process.
Paper notebook
The notebook and pen or pencil is a classic and beloved tool for creative people. Notebooks are sketchy and freeform. Turning over a new page in a notebook to capture a fresh thought is fast and easy. And notebooks are portable but also offer a comfortable posture for thinking.
Notebooks can be a blank canvas for recording observations and ideas, but they can also be a collection space: for lecture or meeting notes, excerpts from other works such as in a commonplace book, or print media clippings as in a scrap book.
There is some evidence that handwritten notes are good for memory. See The Pen Is Mightier Than the Keyboard
Post-its and war rooms
Post-it notes on a wall, index cards on a desk or pinboard, and war rooms offer some of the same value as notebooks (sketchy and freeform) while having the additional value of tapping into human spatial memory.
From migratory birds to open-sea navigation by pre-civilization humans, life on earth has a strong innate ability to remember and navigate space. Our brains have a dedicated region in the hippocampus for spatial navigation. It seems to be possible to activate this region in a digital/virtual environment with the right orientation cues.
Pinboards, chalkboards, and war rooms tap into our innate capabilities for spatial reasoning — and have the additional benefit of being a way for small groups to collaborate on collect-then-think.
Note-taking apps
Almost all creative professionals use a smartphone-based note-taking app for quick capture of thoughts on the go. Some examples include Evernote, iOS Notes, Google Keep, Drafts, Bear, iA Writer.
These tools are great for mobility and sketches in pure text (such as an outline or first draft of an email, paper, or article). But their use is narrow as they are poor at images, web, and other media types.
Virtual pinboards
Virtual pinboards like Milanote, RealtimeBoard, and Jamboard offer a promising “cards on a canvas” model that taps the user’s spatial reasoning and encourage a messy, playful approach.
Other pinboard-like digital tools include Pinterest, which is popular for mood boarding; and Prezi, which encourages hierarchical arrangement of information.
Although most of these remain niche products that have not found mainstream usage with creative professionals, we think they show promise as a way to bring the best parts of the analog post-its/war-room approach into the digital realm.
Ink & Switch has previous experimented with virtual pinboards via the Pushpin desktop app and the Dossier iPad app.
Filesystems
A venerable approach to colllecting raw material is simply to save it to a hard drive and browse with the operating system-provided file browser.
Related and adjascent tools include Dropbox and Network Attached Storage devices. File formats like .txt and Markdown are private, trustworthy, and reliable over time.
But files also tend to be clumsy and old-school, poorly adapted for a multimedia-, web-, and mobile-oriented world.
Digital sketchbooks
Digital sketchbooks and annotation tools with first-class stylus support include Goodnotes, PDFExpert, and the iPad’s built-in markup tools.
These have found some traction with the pros we’ve interviewed. The form factor suggests the analog equivalent of notebook and pen, and freeform scribbling is conducive to the creative process. But these apps usually optimize for drawing on a blank canvas or annotating a single image or PDF.
Project vision
We started with the observation that creative professionals develop their ideas in a two-step process. We saw that they get great value from tools that support that process. In surveying those tools, we we were inspired by capabilities like freeform sketching and spatial navigation, but also saw many gaps such as lack of mixed media.
Putting all of these ideas together, we set out to build Capstone: a digital tool for the collect-then-think process of developing ideas.
Capstone’s vision comes in three three main pillars:
- A digital-information collection tool. Pros need a place to collect many types of digital media: the raw material for their thinking process. Our focus for this project is collection done on a desktop computer’s web browser.
- A place to sift, sort, find patterns and insights. Once the raw material is collected, the user should be able to sift and sort through it in a freeform way. All media needs to be represented in the same spatially-organized environment.
- Use a tablet and stylus to its full potential. Pen-and-paper solutions, especially the A4 notebook, seem to be more conducive to thinking and having better ideas. Our hypothesis is that the thinking step should happen on a tablet with freeform inking via the stylus.
Demo video
Here is a full walkthrough of the resulting application, showing a personal use-case from one of our team members making a life decision about a move to a new city.
In the example, the user wants to record the details of his experience into an archive. That archive will be (1) a place for him to sift, think, and find insights that might guide his decision about whether Austin is a good place for him to live and work longer term, and (2) a visual aid for him to use when explaining to colleagues, friends, or family about his time in the city.
Other examples of topics our test users tried with Capstone: planning a trip; researching equipment for a new hobby; dog training; learning about a new technology stack; organizing a team sprint; and preparing writing and speaking topics for upcoming professional conferences.
While the use case as shown in the video is simple, we hope you can extrapolate to what a more mature version of this tool might enable.
With the project vision established, this next section describes what we built and why.
Briefly, the components are:
- Capstone Clipper browser extension to collect text, images, and websites on a desktop computer
- The shelf as a portal to transmit collected material to the Capstone tablet
- Tablet thinking space with mixed media cards and inking
- Spatial navigation with zoomable boards
- No titles on items
- The shelf as a re-imagined copy-paste buffer
- Card duplication with mirror
- Command paradigm of hands are safe, stylus edits
- Use the edges of the tablet to create and delete cards
- The unsolved problem of stylus tool selection
Let’s go deeper on each.
The collection step
Capstone Clipper is a browser extension for the collection step.
Clip images, text, websites
The Capstone clipper can capture media from a desktop computer by dragging in files, pasting from the copy-paste buffer, or capturing directly from the browser.
Clipped items are saved into the user’s shelf.
Shelf as a portal
The shelf is a freeform space that appears on the user’s desktop and on their tablet. The user can mentally model this as a portal that is always the same between the two devices.
On the desktop, the shelf is a regular window. On the tablet, the shelf is an area in the lower part of the screen which can be summoned by dragging up from the bottom of the screen, and hidden by dragging down.
The thinking step
Now we get to the heart of Capstone: the freeform sift/sort/arrange interface running on the tablet.
Media cards + inking
Capstone’s interface brings together freeform arrangement of media cards (images, text, web clippings, PDFs) and inking (sketches, diagrams, handwritten text, other annotations) in the same space.
Media cards can be arranged freely, like index cards on a table or post-its on a wall. We sought to avoid the precision typically required by digital tools (for example, Illustrator or Keynote) which require the user to work with selections, rotate and resize handles, or front/back arrangement. Instead we wanted immediate interaction moving or resizing of cards in a single gesture.
Inking can happen anywhere. The user can draw and erase on a board and over the top of text, image, and web cards.
Spatial navigation with zoomable boards
In Capstone, the user organizes their clippings and sketches into nested boards which they can zoom in and out of. Movement is continuous and fluid using pinch in and out gestures.
Our hypothesis was that this would tap into spatial memory and provide a sort of digital memory palace.
Nested folders on desktop computers are the venerable model for place-based navigation. Despite attempts at innovation such as tags and search-first interfaces, nested folders remain both popular and effective. See The Science of Managing Our Digital Stuff, chapters 4 and 5.
Pinch gestures for zooming in and out are familiar from many touchscreen apps, such as Google Maps. We like the association with real-world navigation, and see it as a good example of innovation from the mobile world that can be brought to bear on creative tools.
No titles
Computer applications typically list items by a title, such as a filename. This abstract representation doesn’t match how the document looks (although sometimes it is accompanied by a preview).
Our idea for Capstone is that representations should look like the document. Some media types, like images and short texts, can display the whole document. Web clips, multi-page PDFs, and longer text excerpts cannot be shown in full, but we we can show a reasonable summary.
The shelf as reimagined copy-paste
In desktop operating systems, cut-and-paste is a crucial capability. Pro users have ctrl-C/ctrl-V hotkeys in their muscle memory.
But the traditional copy-paste buffer is due for a refresh. It can only hold one thing at a time, it can’t be edited, and it is invisible. On touch devices, copy-paste is typically hidden behind a set of clumsy submenus.
Invisible copy-paste buffers can surprise users, such as accidentally pasting a password into a chat window.
Capstone’s solution is called the shelf. Already described is how the shelf is a portal between the desktop computer and tablet, a sort of private inbox. Once within the sifting/sorting/organizing step, it becomes more: a scratch space and a cozy peripheral space that has the same spatial capabilities of boards but always travels with the user.
The user can move or duplicate cards from any board and put them on the shelf, navigate to another board, and then drop them there. Like traditional copy-and-paste, the shelf is a buffer that stays with the user as they navigate around the app; unlike traditional copy-and-paste, it is manifest on the screen, can hold multiple items at the same time, and can be edited.
Item duplication with mirror
Another important function in power tools is the ability to duplicate an item. Desktop productivity tools often have a “duplicate” or “clone” context menu item, and the user can also copy-paste.
Our approach in Capstone is the mirror command. The user can mirror a card by “pinning” down the card with one finger and dragging out with the stylus:
The finger+stylus hybrid action is an example of what can be done if a tablet app seeks to reach beyond the limitations imposed by the one-handed, small-screen assumptions typically used when designing for touch devices.
Hybrid gestures between finger and pen are challenging on Chrome OS and other tablet platforms. OS gesture recognizer libraries are not built for this kind of work, with complicating factors like palm rejection.
Using the tablet and stylus
That covers Capstone’s features supporting the collection and thinking steps of the creative process. Now we’ll address form factor.
Our user research suggests that the A4 notebook shape, especially when paired with a comfortable reading chair or writing desk, is an ideal posture for thinking and developing ideas. That suggests the digital form factor of tablet and stylus.
Existing tablet apps on mobile platforms don’t tend to be very powerful. We speculate this is because they are built as scaled-up smartphone apps, hence they don’t take full advantage of the expressiveness of two hands and a stylus.
Another design approach to tablet apps is to treat them as a port of desktop applications. Attempts to map mouse and keyboard command vocabularies to a touchscreen environment are often clumsy and confusing.
Capstone’s vision is to avoid both of these problems by treating the tablet as distinct from smartphones or desktop computers. Instead, the tablet is a third class of computing device with its own capabilities and place in the user’s life.
Here are three UI approaches we tried in our efforts to make the tablet a device with its own identity.
Hands are safe, stylus edits
The command model for most tablet and phone apps is onscreen chrome (buttons, toolbars, etc) that the user can tap to activate the app’s functions. The stylus is either undifferentiated (treated identically to a finger) or sometimes offers a limited activity such as a separate markup mode.
For Capstone, we tried a different command model: hands are for navigation, the stylus is for editing.
Users are conditioned by ten years of smartphone apps to be able to drag things onscreen with their finger. But in our tests we found that users quickly acclimated to using the stylus for moving and resizing.
We also found that this is an anatomically correct answer: fingers are coarse / non-precise, and the stylus is fine / precise. The stylus provides a higher degree of spatial freedom by expanding our touch zone with smaller arm moves which causes less fatigue, and higher accuracy with its thin tip.
Edges to create and delete cards
An important operation in any thinking tool is creating a new blank space for work. The user wants the equivalent of turning over a new page in a notebook, or pulling a fresh sheet of paper off of a stack.
A common approach to this is a plus button or menu option to create a new item. These are often paired with decisions the user needs to make — is it a text note or a drawing? Is it landscape or portrait orientation? All this ceremony slows down the user in getting straight to recording their ideas.
The Drafts note-taking app deserves special praise for its ease of creating a new blank canvas: each time the user invokes the app, they get a new blank page by default.
Deleting or archiving of items is the symmetrical operation: it must be fast and frictionless to dispose of whatever items the user has decided to cull. Ceremony (such as “Are you sure?” confirmation dialogs) and delay (such as long-press to activate a context menu) are antithetical to a speed-of-thought tool.
For Capstone, our approach is to “pull in from edge” to create a new board and “throw off the edge of the world” to delete.
In our testing we found it natural and even fun to both create and delete items. In creating a board, the motion of the user’s hand can be fluidly continued to put the board where they want it. The two operations (create board, place board) the two flow together into one seamless motion.
It was also fun to “flick” the card away as it approaches the screen edge, including deleting several items in a row (flick, flick, flick).
Stylus tool selection
Tool- or mode-switching seems to be necessary for any sufficiently-powerful interface. To avoid the user needing to constantly reposition their hands, input capabilities will need to serve multiple functions. On the tablet, we have an even more restricted input space, so tool-switching is essential.
Desktop programs have a variety of tool-switching approaches, from the tool palettes of Photoshop, to the “dead man’s switch” metakey approach of copy/paste and other shortcut keys, to the highly modal approach of the vim programmer’s editor.
On Capstone, it was one of our design goals to keep as much of the screen space free of administrative debris like toolbars and menus. Despite attempting a wide range of approaches, we failed to solve the tool-selection problem and had to fall back on the unsatisfying solution of an onscreen tool selection palette.
Here are some of the other tool-switching approaches we experimented with:
- Command glyphs
- A modifier key activated by the non-dominant hand
- Using a second entry tool, like a second stylus or a dial
- Pressing the stylus barrel button
- Using physical buttons on the tablet such as the volume button
- Using stylus pressure sensitivity to switch between modes, as described in Building interactive systems: principles for human-computer interaction
- Obscuring the front-facing camera with the non-dominant thumb, as tried by Astropad
- Knuckle gestures, as seen in Experimental analysis of mode switching techniques in pen-based user interfaces (video)
We don’t like the onscreen tool palette and consider this a significant unsolved design problem.
Shortly after the Capstone project wrapped, Apple released their attempt at solving this problem: a stylus which includes a barrel double-tap.
That concludes the description of Capstone’s major design decisions. If you have a Chrome OS tablet, we encourage you to try it for yourself and tell us what you think.
To bring this manuscript to a close, we’ll share what we observed from testing on our team and with external users.
What worked
Mixed media + sketching
Freely-arrangeable multimedia cards and freeform sketching on the same canvas was a clear winner. Every user immediately understood it and found it enjoyable to use.
Advanced touch UI interactions
The command model of “hands are free, stylus edits” was mostly a success. And hybrid touch-finger interaction like mirror and pulling in cards from the edge of the screen created a fluid and natural environment.
Thumb + Pen Interaction on Tablets provides a good look at simultaneous finger and stylus input, including using the non-dominant hand to activate mode-switching.
Zoom navigation and “peek”
Nested boards with pinch zoom in/out worked as expected. But an unexpected benefit that emerged in user tests is the idea of a “peek.”
A user can begin a pinch-out gesture to zoom up a board to almost their full tablet screen and read in detail what is on the board. If it’s not what they’re looking for, they can reverse their pinch direction without lifting fingers from the screen to cancel the operation.
Mobile-style motion design for productivity apps
In desktop productivity apps, results from user actions would snap into place onscreen with no motion or transitions. The addition of mobile-style motion design and transitions to Capstone seemed to make the app come alive.
Our observations suggest that these are more than just visual flare: they help create a fluid, environment where the user can move fast and with confidence, because the app visually reinforces every action they take.
Viability of no titles
We used representations of images, text, websites, and boards which look as much as possible like the original content. Boards in particular give you a feel for the colors and shapes they contain, and the ability to read any handwritten large titles. The “peek” capability reinforced this.
Our observation was that a “no titles” approach is indeed viable if there is enough summary information and it is fast and cheap to quickly look more closely. But given the small data sets used in our test cases, further research would be needed to find out if this will work at scale.
Fractal arrangement of ideas
The boards-within-boards navigation metaphor of Capstone encouraged users to organize their thoughts in a hierarchy.
The Pyramid Principle is a managerial technique for presenting ideas in a telescoping arrangement.
For example, one of our test users developing content for a talk about their charitable organization’s mission created a series of nearly-empty boards with titles. Combined with board previews, this creates an informal table of contents for the ideas the user wanted to explore and could start to fill in.
Chalk-talk use case
The cards plus inking plus zoomable boards lent itself naturally to telling a multimedia story, ala chalk talks or Saul Kahn’s videos.
We speculate that a product-quality version of Capstone could be useful for:
- distributed teams (screenshare your tablet while talking over video chat)
- live or prerecorded educational content as found in MOOCs such as those on Coursera
Unlike slide decks created with software like PowerPoint or Keynote, these “cards and sketching” presentations felt more lively and human.
What didn’t work
Stylus mode-switching
As documented above, our solution to stylus modes (card editing vs inking) was an onscreen tool palette, which users found clumsy and error-prone.
Complex motion and transitions on web stack
The web stack we were building on made complex transitions challenging. Smooth transitions such as moving in and out so that it felt like one continuous space was a significant technical challenge. Our engineers concluded that the web stack (which is optimized for discrete page navigation) is not ideal for this. A better approach would be more like a video game: continuous rendering of an archive as a single “world.”
See Appendix A for more.
Left- vs right-handed use
Most of our tablet UI paradigms depend on knowing which hand is dominant for the user. For example, onscreen mode-switching buttons should be located wherever the user’s non-dominant hand is likely to be located (including gripping the tablet). But that is a different place depending on the users’ handedness.
“Southpaws” comprise about 10% of humans, but there is some evidence that there is a higher proportion among creative professionals.
A left- vs right-handed toggle would probably be the solution for a more complete prototype.
Problems with lost, malfunctioning, or out-of-battery stylus
Making the stylus a required part of a power-tool interface had many benefits, but it also exposed problems we faced throughout the project:
- A stylus that inexplicably ran out of batteries after a week, with no indication as to why it was not being recognized by the tablet
- Varying pressure sensitivity, with some users finding they had to press so hard as to distort the screen
- Frequent lost styluses (“wait, let me find my stylus…” was a constant refrain)
We speculate that, to be viable, a tablet+stylus platform would need to address these. One solution is that if stylus hardware became cheap enough and worked universally, then workplaces and individuals could keep pen-holders full of styluses around, just as today we keep spare pens around.
The Universal Stylus Initiative seeks to standardize stylus-on-touchscreen interfaces to make interchangeable stylus use possible.
Screenshot clipping
While the Capstone clipper allows users to capture entire web pages, images, and fragments of text, often users wanted an “excerpt” of a website or PDF with all formatting intact. The user behavior here was to take a screenshot using the “capture region” screenshot capability of their desktop OS.
Screenshots as a way to capture and archive content has many downsides: it’s not searchable, the text is not selectable, it loses quality when sized up, and most of the metadata (e.g. what website it came from) is lost.
See Appendix C for expanded discussion of web page archival.
And yet, screenshots are easy, reliable, and universal on all computing platforms. Users are naturally drawn to take screenshots of text: what could an application or platform do to lean into this tendency, such as automatic OCR? We think this warrants more investigation.
Conclusion
At Ink & Switch we believe in tools that help humans think, create, and grow in our personal and professional lives.
We feel that computers have significant untapped potential in this domain. But as the technology industry puts ever-more effort into consumer products, we wish there was a similar emphasis on productivity and creativity applications.
Capstone is one experiment with innovation in digital thinking tools. It suggests some promising directions such as mixed media cards and inking on a freeform canvas, spatial navigation, and the tablet as a device with a purpose distinct from phones or desktop computers. That is: a quiet space for ruminating on raw material and finding insights to guide career or life decisions.
If you’re a developer of iPad or Surface applications, desktop productivity software, or a human-computer interaction researcher we hope you find some inspiration from Capstone. We’d love to hear your thoughts: @inkanswitch or [email protected]
