“Methods and Processes of making are ways of investigating possibilities for what the Thing might be…”

—Denise Gonzales Crisp

Design through experimentation has never been my strong suit. I prefer to be analytical, to figure things out with some façade of certainty, to know the how and the why… and then to pursue making. This tendency may be the product of practice in industry. Moving into the academic realm has undoubtedly required a stretch on my part.

When designing for a new information environment, all things become experimental. Research uncovers limited designed precedents. A blank slate appears. I move forward through methods and processes of making, investigating possibilities for a speculative interface that responds to eye movement as an input method.

Temporarily skipping Phase I, I began my work during Phase II, the Brad Tober workshop. With a great interest in the functionality and usability of Wikipedia, I contemplated a research tool as a potential solution to a personal dilemma. I affectionately call this dilemma the “how on earth did I get here” problem. Upon browsing an academic paper, I make a list of unfamiliar terms. Wikipedia is often the source for this top-level definition search. Each Wikipedia search results in additional curiosity-inducing terminology. As I click, and read, and click, and read… the perpetual rabbit hole has taught me much, but I am lost without a way to navigate back through the information. My resources are lost. My curiosity and pondering produces a plethora of untraceable knowledge.

I chose to create a gaze-based navigational interface for Wikipedia that allows a user to track pathways of information (A). Upon reading an academic article, a user records a short list of unknown terms, followed by a Wikipedia search. When reading the search results, the web-based interface detects curiosity or confusion when the user gazes upon additional unknown terminology. The user is then prompted to either “record” the word for future reference or “delve” into searching for the term immediately. Each interaction is recorded in a Path Tracker panel (B). This panel displays the navigational pathways explored by the user. Given prompts for starting a new path or revisiting a previously searched term, the user now has a visual answer to the question, “how did I get here?”

The discoveries of the Brad Tober workshop informed my work for Phase I gaze-based studies. I became curious about the unique affordances gaze offered as an input method. What might be accomplished through eye movement that is superior to the click of a mouse? I began to explore movement, organization, and layering of information using simple, greyscale shapes (C). I investigated possibilities through making. My studies remained without the context of a speculative interface.

I discovered beauty in the simple movements created through the Phase I studies. However, I remained curious about what the eyes could do that the hand could not. I began to create a reimagined interface for the music streaming app Spotify. Through research I found that sound is processed in the visual cortex of the brain (1). Applying this information to justify that an audio-based interface would be better accessed via gaze was a stretch (or a well formulated element of BS).

I started wondering if people have tendencies to look in certain directions for certain things. This curiosity brought me to a series of academic articles regarding spatial metaphors (2–4). Research in the field of cognitive psychology has shown universal cognitive properties of time and space. UP is associated with a positive experience and DOWN with negative. FRONT/RIGHT is associated with the future and BACK/LEFT with the past. These directionally-ordered terms uncovered the affordance of gaze I had been seeking. With this discovery, I developed a simple spatially-based interface that allows a user to interact with and navigate through a music streaming interface (D–F).

If, indeed, spatial metaphors are an instinctual, universal understanding across cultures, perhaps they can be employed to design gaze-based interfaces for future use. The basic system developed — the four quadrants based on spatial metaphors — are not confined to an audio-based interface. In additional to top, bottom, left, right, how might future interfaces offer narrow and deep fields of view to expand spatial affordances? The exploratory work of Phases I, II, and III have inspired me to consider future research and design exploration of gaze-based technology.

References

1. Sound and vision: Visual cortex processes auditory information, too
2. Spatial Metaphors in Temporal Reasoning (Gentner)
3. Music and Identity (Frith)
4. Why we use spatial metaphors, like high and low, for emotions — and how they can help you feel better

My interest with gaze based interfaces started with analyzing approaches to reading academic articles. As students and researchers, we often skim and scan with a goal of finding information related to a specific topic. We look for keywords, theories, diagrams, and conclusions drawn to decide if the article is useful to us. While this is important, I started to question if there was more a gazed-based interface could offer to impact synthesis within reading material.

During the workshop, led by Brad Tober, I explored the possibility of drawing connections between specific keywords within a Wikipedia article. My approach focused on utilizing varying dwell times from the viewer’s gaze to trigger three main levels of connections; Direct, Indirect, and All. Each option was color coded to show a hierarchy between levels and a sense of wayfinding (A). Prototyping the visual display of connections offered space for considering how the UI could develop to track and display keywords and toggle between levels.

Continuing with the idea of relational levels and how they can provide room for synthesis within an article, I focused on representing gaze tracking. I explored how a gaze based interface might interpret a viewer’s reading pattern regarding skimming and/or scanning. Considering the infancy and maturity conditions of the interface, I explored how tracking reading patterns could keep the viewer alert and allow evidence of progress and hierarchy during the reading (B).

Exploring various ways of alerting the viewer led to developing ideas for an interface that could sense when a viewer lost focus while reading. Sensing unfocused gaze in the infancy stage, the interface adjusts incoming text to change in width and spacing to gain the viewer’s attention. In the maturity stage, the interface senses lack of focus more quickly and restructures oncoming content by varying the location or shape of its appearance to grab the viewers’ attention (C).

Final studies focused on developing a system that emphasizes the need for a gaze based interface. Within these studies, I explored the behavior change of content in response to multiple instances of gaze. For instance, if the viewer is skimming a portion of the article, the words or phrases skimmed over most often become larger and shift the article structure. However, in the final system studies I focused on selecting and locating keywords while toggling between a regular and bird’s eye view of the article (D). It was challenging to consider the affordances of a gaze-based interface and speculate what could be successful. Having the opportunity to prototype some concepts and gain feedback from local psychology students offered great insight into why this technology is significant for graphic designers to consider.

For most of us, our gaze is the primary source of input. While interesting to consider how gaze can serve as output, throughout this project I focused on the input side of things, and the different ways eye tracking can enhance small moments.

It was helpful for me to explore small moments within a specific space. I chose an art museum. It took only a few searches to discover that there is plenty academic research on museum labels, visitor engagement, and behavior patterns. I was curious about the research on museum visitors, and used it to guide my studies.

My first study explored the museum label (A). I wondered how the distance of the eye location could dictate interaction. The infancy condition consists of a label that generates information based on the visitor’s distance from the label itself. The maturity condition offers just a slight shift in deliverables. Instead, all the information is readily visible, in a “collage.” Different snippets reveal themselves depending on how close the eye is to the label.

The second study considers a unlabeled reality (B). Rather than the tiny, boring, jargon-heavy labels seen now, imagine a world where the entire wall delivers information. Again, the information changes based on the distance of the visitor’s gaze, but this time it also considers their location (i.e. the closest painting to the visitor). In the maturity condition I explored how augmented reality could fit into the museum space. A visitor could glance into a room and information could “pop” into view without them having to enter the space.

Throughout my first two studies, I began to wonder how the moments work when there are multiple people — and gazes — in the space. For my last study, I wanted to explore the multiple-gaze problem area, rather than ignoring its existence (C). The infancy condition shows how a museum label could track multiple gazes, and display information democratically. The maturity condition imagines digital artworks that zoom in and out based on multiple gazes. Again, based on a democratic system, the maturity condition determines what part of the artwork is featured. With both conditions, visitors are “battling” for what information will be displayed. Visitors who may be interested in less popular things will not have instant gratification.

The last part of this project was crafting a system. Each of my studies explored a different possibility for gaze in a museum space, so I crafted a system that married many of my studies together (D). Before I started, I wanted to choose a user. Considering the content of my studies, I approached the system from a curator’s perspective. I designed a system that would enrich a curator’s understanding of visitor engagement, and help make decisions that improve the visitor experience. First a museum wall would track visitor gazes across multiple works. The tracking would be shown to the curator via a report. The curator would then use the report to curate responsive digital walls (like Study Two’s infancy condition). Again, the viewer gaze would be tracked, and the curator would improve the responsive wall design. The system is quite literally a system, or a cycle, of constant input from both curator and visitor.

I thoroughly enjoyed simultaneously exploring gaze and the museum space during this project. Initially I struggled to be open to mind-blowing possibilities, but soon realized how futuristic thinking can help me consider the present.

While working on this project, I kept being reminded of how easy it is to take for granted the things that we perceive people to be naturally born with, in this case, the use of our hands. It was an eye-opening experience designing for such a unique situation and having to put myself in the “eyes” of the potential user. At first, I was not truly understanding of these types of situations and how revolutionary a gaze-based interface could be in our world. It was not until the Brad Tober workshop (Phase II) that I began to have a better understanding of what a gaze-based interface could offer and how different features of this type of interface could empower the user.

For my studies, I focused on three different aspects of an interface: search, sound and buttons. The search aspect of an interface was the focus of my first set of mini-studies (A). This is where I explored how gaze could work to show the preview of a web page without completely opening it. I also tested in this study how related web pages could be shown before, during and after a search. I tried several different options including a grid system, holographic categorization, a minimal search engine that leads to maximum information, and the idea of page leafing (A).

My second study was about how sound could be incorporated into a gaze system to enhance the usability for those who have degenerative eye disorder or are partially or legally blind. One of my mini-studies questioned how the interface would indicate where text is located on a web page when the gaze was on it, by giving off a certain level of tone. For example, if your eye gaze was over one word on a web page, the pitch would be higher, as opposed to a whole paragraph, where a low pitch would sound to indicate a greater density of text. After text was found there then could be two potentially different options for the text being read. One, a programmed voice could read the text to the user, or two, there could be a sound vocabulary where each letter or the alphabet could be recognized by its own level of pitch. This of course, would be a maturity level of interface where the user would already know the sound vocabulary, much like knowledge of braille.

Study three focused on how buttons could open and close information (B). There are so many potential opportunities of how a web page could be designed when it comes to a gaze interface. Web pages and search engines could give very minimal information and our gaze could then lead us to more in-depth information. Or it could be the opposite; the design could be maximal and our gaze could highlight or focus on one aspect of the page being viewed.

My final system utilized many interface aspects that I explored throughout my studies, but I tried to push the ideas further. I really enjoyed exploring how search could be changed to accommodate a gaze-based interface. Therefore, for my final system I created a web browser that loaded your search content into three dimensional cubes (C, D). As your gaze is over a cube, the cube turns and gives you a preview of the content that is on the related web page. The further the cube turns (related to the user’s dwell time), the more of the image preview it shows, as an indication that the web page is going to open soon (C). Another aspect of my gazed-based browser is the history panel. This is much like the history review page now implemented into major browsers, but the one in my system highlights other previously viewed web pages with related content to the one your eye is currently gazing over. The preview feature is also carried out through the history panel as well.

Throughout this process, I have learned how to explore enhancing currently designed interfaces to benefit all people. It is important as technology continues to advance to take into consideration how design can change the way all types of people move, think and feel.

This project was a challenge for me. My hands are how I interact with the world, so removing them from the navigation of an interface felt bizarre and unnatural. The Brad Tober workshop provided an interesting glimpse into the affordances and limitations of a gaze-based interface, while informing my future studies. To combat the awkward feelings a gaze-based interface gave me, I chose to focus on how a user would read text in a gaze-based interface. Reading in itself is a gaze-based activity, so a reading interface that relies on gaze feels a bit more natural than an interaction that has traditionally been point-and-click or gesture-based.

For my studies, I played with different ways a user could control reading patterns on a gaze-based interface. I explored different text presentations, including paragraph-based and something more “Spritz”-like (RSVP technology), or one word at a time. In a Spritz-style presentation, I see the user as creating a record of the experience by moving his or her focus around the page, arranging the words like concrete poetry. This would create a unique record of how the reader interacted with the text (A). However, the reader would be forced to read every word, making skimming difficult and limiting the user’s control over the experience. I found that a paragraph style presentation, while more traditional, allows the reader to decide how to move through a text. In a maturity condition, the interface could sense a user’s reading pattern and adapt to meet those patterns, dropping out words or lines based on a user’s behavior (B).

I was also fascinated by the idea that an eye-tracking system can pick up on a user’s emotional reactions to a text. In some of my studies I played with how the interface could display those inputs for a user, creating reflective moments during a reading experience. I used different visualization techniques, including a gaze indicator that changes color to reflect emotions or the user’s gaze, “painting” the text with different colors that indicate emotions, thereby creating an emotional record of the reading experience (C).

For my final system, I combined several of these ideas to create a speculative, gaze-based magazine app that uses the user’s emotional inputs, as recorded by the gaze, and artificial intelligence to create a totally customized reading experience (D). The interface is styled to look like inkblots, with the gaze causing blots to expand and contract when the user gazes at them to indicate transitions. The text generates as the user moves through it and reacts to the user’s reading patterns. Strong emotional reactions activate annotations and additional information in the margins. These moments can be as long or short as the user wants and are presented in a Spritz style that continues until the user focuses on a different section of the interface. The interface includes a review mode that allows the user to see their reading and emotional behaviors.

Overall, I believe the possibilities for gaze are rather interesting, especially the emotion tracking possibilities. As machine learning and artificial intelligence become more powerful, the implications of gaze input will only grow.

In one of my initial studies of the gaze interface capabilities, I explored how users could see their emotions while experiencing an app, interface, or technology. For example, while users read, they could see through some type of visual system how the sentences or individual words on a screen can influence how they feel.

In my initial explorations I chose to study how a television screen could reflect an individual’s emotions while watching something controversial, such as a tragic event on a news channel. The screen could change colors reflecting a feeling, or taking the study a step further, the television could decrease or increase the volume depending on user emotion. Then I explored how a television could reflect emotions through gaze in a public setting, with more than one person gazing at the screen. The television could reflect how everyone looking feels, which I discovered could become invasive because people’s emotions would be exposed and some people would like their emotions to remain private.

For my system, I chose to explore gaze and emotion recognition through an individual and private experience. I focused on how a user’s gaze could influence emotional awareness and control, specifically in the context of news media on a social media platform. News in social media, such as Twitter or Instagram, is a powerful tool that has a particularly strong influence today due to political and social conflicts. In my system, the user is experiencing a Facebook argument about politics. The system reads the user’s gaze and recognizes the user’s emotions while reading the argument. As the user’s anger grows,the content of the argument will change, attempting to resolve the anger. When the user tries to comment on the argument, every letter typed will reflect the user’s anger level. The goal is to increase emotional awareness in a user. There are two short-term parts to this system that occur in the moment of the user interacting with the system: first, reading the post and comments, and second, typing a comment before posting it.

Here is what could happen as a user reads through a Facebook article post and the comment sections below the post: as the user reads the first comment, the user becomes angrier; then, the system recognizes that the user is reading, while also interpreting the user’s emotions based on gaze; next, as the system recognizes the user’s gaze and emotions, the text changes from a darker red to a brighter red; after the text becomes mid-tone red, the system attempts to resolve the user’s anger by changing the content as the user continues to read (A).

Here is what could happen as a user types a comment in response to another comment on the post: the user begins to type words into the “Reply” box available below the original comment; then, as the user types, each word enlarges and becomes the shade of red that reflects the user’s anger level; and if or when the user types particularly harsh words into the comment, the words are highlighted even more than the rest of the message by becoming all caps and a brighter shade of red (B).

I explored how the system can store information over time from other users’ gazes through the same news post (C). I was thinking about how the “Like” system works now, and how that system could change if Facebook was completely controlled through gaze. I created a number next to each comment within the news post that shows the user how many other people felt the same way as the user when reading the comment. There is no way to see who felt similarly, just a number that builds over time justifying, or not justifying, the user’s emotions as recognized from gaze. This number is completely dependent on other users’ gaze patterns and how the system interprets their emotions. Depending on the shared emotion, the number will change colors. For now, the number remains red because the user felt angry while reading most of the comments, and other users felt angry, too, while reading the comments.

I learned to explore design through innovative technology with this project and how to strip away that with which we are so familiar. Going further, this type of speculative thinking opens doors for brainstorming how to use technology with other body parts, like our tongue or ears, or taste and sound. Maybe we could use an interface with another sense other than sight or touch.

Eye tracking and gaze navigation have larger implications than simply the visual interaction with our screens, and expand to many fields of research and expertise. My research on the literature regarding eye tracking has revealed the potential for enhancing people’s interaction with the world, where their gaze can give them a new ease of access and navigation.

For the first phase of this project, I developed a set of studies using wireframing and storyboarding to explore the potential of such technology in our daily usage. For each of these studies, I used a system where I placed both infancy and maturity alongside each other, so that we can identify the changes and how the technology could impact our daily interactions.

For my first study, I focused on the use of eye tracking and gaze navigation for the specialization of research content results (A). What if the gaze allowed you to reorganize as you view the hierarchy of your content, based on your own interests and preferences? I liked the idea of an interface being able to reorganize itself based on your gaze behaviors, your emotional response to content, and your research prompts, to enhance and maximize the time we usually spend on the research of content. This intersection exists at the line between machine learning and eye tracking, where the separation is very thin and where both fields could complement each other.

For my second study, I considered the use of eye tracking and gaze navigation in the context of physical environments such as museums. I investigated the idea of how a user’s gaze could be used as a personal docent while instantly providing the user with information regarding the painting or the symbols on a painting. Furthermore, the user’s gaze could become a magnifier allowing him or her to see the complex details in a painting. I considered how a multi-gaze detection interface would constantly reframe the content of digital screens to allow users to have their unique content presented to them in ideal conditions. I also considered the use of gaze navigation in a changing room where the client could interact with the mirror. Finally, I considered how gaze-controlled interfaces would be beneficial in places where hygiene is critical, such as hospitals.

For my last study, I researched the use of eye tracking and gaze navigation in an online learning environment (B) where both the data of the user’s gaze behavior and the gaze itself could serve in enhancing the student learning experience. Data from eye tracking could provide the student with feedback based on his or her efficiency and attention while studying the content. Finally, I considered how being able to see the professor’s gaze on the interface could help the student to maintain his or her attention and keep track of the course.

For the second phase and the development of my system, I decided to stay in the context of my former study to represent the key moments of the journey of a student going through an online interactive art history class. The student can enable the professor’s gaze (C) while accessing new layers of content, and receive immediate and personalized feedback (D) when losing focus or finishing a course.