Augmented Reality

Let It Out: A Self-help Mobile AR Spatial Journaling Experience

A cathartic spatial journaling experience in AR that allows me to see my thoughts and feelings in a new way. It can be used to get past my blue and dysfunctional phases by fully immersing me in the augmented externalization of my thoughts in space with the least amount of energy required while able to track my emotional progress without being reminded of the dark place I was in.

Role

Concept Development
AR Interaction Design
AR Foundation Development
Programmer

Tools

AR Foundation
Unity
IBM Watson

Timeline

March 2020 - May 2020

Abstract

Overview.

Let It Out is a cathartic spatial journaling experience in AR exploring the possibilities of connecting one with their own thoughts and emotions in an unprecedented way which, different from traditional journaling experience, takes space and ephemerality into consideration.
・・・・・・・・・・・・
It allows me to have a conversation with myself, saying my thoughts out loud just like talking with my friends, to explore and process my thoughts and feelings to get through my blue and low energy phase in between the states of being clinically depressed and feeling completely fine. Instead of dividing me from my thoughts with a piece of paper, it fully immerses me in the augmented externalization of my thoughts in space around me with the least amount of energy required using speech to text technology in augmented reality.
・・・・・・・・・・・・
At the end of the spatial journaling experience, those thoughts in the form of text become a cloud that is color-coded based on its sentiment tone, so I would not be reminded both in the physical and the virtual space of the bad places I was in when I am in a better mindset, but still able to track my emotional progress in the virtual space. For example, the clouds are dark blue for depressed thoughts, indigo for frustrated feelings, and etc. Yet, if I want to review and analyze further about what exactly happened, I could also tap on the blurred clouds, and my original thoughts in texts would be revealed. All the blurred colored clouds would then disappear after a season (3 months) as a way to let go of the dark memories and have a brand new start along with our four seasons.

INITIAL CONCEPT X Research

Self-help Tool for Emotional Wellbeing.

I have a huge passion for exploring the needs and motivations behind human behaviors and emotions. Starting from really wanting to salvage myself back to my normal productive state from an almost dysfunctional state, I initiated my research by looking into emotional wellbeing. I realized that there is an area in between the state of having clinical depression or post-traumatic stress disorder (PTSD), and the complete opposite state of thinking there is nothing mentally wrong with oneself, that lacks the attention and care it needs. Psychologist Susan David, Ph.D. introduced the concept of Emotional Agility, which is the ability to be with your emotions with curiosity, compassion, and especially the courage to take steps that connect to what aligns with your values. She stated, “Courage is not an absence of fear; courage is fear walking”, and our emotions are data which inform us whether we are on the path that aligns with our values, and should not be directives that drive our behaviors and actions.

In that light, it makes sense to me why journaling is one of the most effective coping mechanisms for managing psychological stress as it externalizes our intangible thoughts on paper which allows us to literally see our emotions and thoughts as data and makes it easier for us to arrange and process them and identify the root or cause of them.

However, the journaling I mention here is different from maintaining a journaling habit which also benefits us greatly in the long run for our mental wellbeing and helps connect us to our true self for better accomplishing our goals and tracking our progress for future reference. The journaling as a coping mechanism in the context of my project is to be used to manage our stress and take care of our psychological wounds caused by any events in life as it goes through its ups and downs.

Problem Statement

How to Actually Deal with Emotional Difficult Challenges?

There is the state of being clinically depressed or having serious traumatic reactions, which has specific medications and treatments for; there is also the state of feeling that there is nothing psychologically wrong about oneself, but most of us actually experience the state that is in between both extremes. Studies show when we are in that in-between state where we are depressed but do not have depression which severely affects our functionality, we become very inactive, and our energy level is very low resulting in us being in a semi-paralyzed state. This is part of the Spoon Theory, which is a neologism used to explain the reduced amount of mental and physical energy available for activities of living and productive tasks that may be caused by psychological wounds or chronic illness. It acts as an emic descriptor for the planning that people have to do to conserve and ration their limited energy reserves to accomplish the tasks of daily living. Usually, healthy people do not consider much about the energy spent during ordinary tasks, such as eating and bathing, so this theory can help others empathize with the consequence of chronic physical pain or psychological injuries, which research shows it activates the same part of our brain as physical pain, on a daily routine.

However, how to actually deal with emotionally difficult challenges are often left vague. While there is professional help that we could seek, there are limitations, such as time, space, money, and stigma. Being psychologically injured to an extent distorts our perception and scrambles our thinking. It is a manipulator with the ease that allows our mind to trick us into the exact opposite of what we need to do to recover, thinking those around us do not care much for us, wondering why we would set ourselves up for more rejection and heartache, preventing us from feeling emotionally or socially connected. With that mindset, we inevitably choose to be alone, and not doing what we know would actually help - reaching out to friends and talking or hanging out with them. Without both the professional help and social support that we need, one can only help oneself to have just enough strength to move forward and get through the blue phase until we are able to reach out and get helped.

Pain Points

Low Energy X Don't Want to Be Reminded of the Bad Places I Was in.

One of the most effective self-help coping mechanisms to manage our psychological stress is journaling. However, from my own experience, when I am in that inactive and low state, I could not get up to grab food (the longest record is 35 hours without food and almost fainted), let alone sitting up, grabbing pen and paper to write. I usually end up hopelessly and helplessly abandoning myself in my rumination in bed for days without food and water. I also do not want to leave any records to be accidentally reminded later on of the bad place I was in if I journaled. Staring off into the ceiling of my room in my bed with my head on an endless loop of my thoughts and feelings, I thought I could design a new form of journaling that utilizes the ceiling and my room as a spacious canvas to journal in augmented reality on a smartphone which has become a device that never leaves our reach. The form of AR allows me to not be reminded later on of the bad place I was in previously by what I journaled down virtually when I am in a better place back in the physical world. Afterall, despite not wanting to be dragged back into the old dark place and mindset I was in again by what I journaled, the act of journaling is still extremely helpful for processing my thoughts and feelings.

When I am in that inactive and low state, I usually end up abandoning myself in my rumination in bed for days without food and water.
I also do not want to leave any records to be accidentally reminded later on of the bad place I was in if I journaled.
When I am in that inactive and low state, I usually end up abandoning myself in my rumination in bed for days without food and water.
I also do not want to leave any records to be accidentally reminded later on of the bad place I was in if I journaled.

Design For Self

The Best Way to Make It Universal Is to Make It Personal.

Autoethnography is the most common approach in most of my projects, and this thesis project is not an exception. Autoethnography is a form of qualitative research in which an author uses self-reflection and writing to explore anecdotal and personal experiences and connect this autobiographical story to wider cultural, political, and social meanings and understandings. Therefore, in the belief that the best way to make it universal is to make it personal and that, by designing tools to help oneself, you often make something of value to others as well, I started my project Let It Out: a self-help cathartic AR spatial journaling experience, hoping to give helpless people like me some strength to get through lonely and hard times.

Mobile: To solve that low energy problem. Though I was unlikely to get up to grab a pen and paper to write, most of the time, I would be holding my phone scrolling in bed already.
Speech-to-text: Since I’m addressing on the isolated low energy state. I thought why don’t I talk with myself instead and still saying my thoughts and feelings out loud just like talking with my friends.
Mobile: To solve that low energy problem. Though I was unlikely to get up to grab a pen and paper to write, most of the time, I would be holding my phone scrolling in bed already.
Speech-to-text: Since I’m addressing on the isolated low energy state. I thought why don’t I talk with myself instead and still saying my thoughts and feelings out loud just like talking with my friends.

Solution

Spatial Journaling - Visualize in Space, Gone in Physical World.

Let It Out explores the possibilities of connecting one with their own thoughts and feelings in an unprecedented way that allows me to have a conversation with myself, saying my thoughts out loud just like talking with my friends, to explore and process my thoughts and emotions to get past my blue and low energy phases, which is in between the states of being clinically depressed and feeling completely fine. Instead of dividing me from my thoughts with a piece of paper, it fully immerses me in the externalization of my thoughts in a 3D virtual space around me with the least amount of energy required. At the end of the spatial journaling experience, those thoughts, in the form of text, become a cloud that is color-coded based on its sentiment tone, so I will not be reminded of the bad places I was in when I spatial journaled that time, but will still able to track my emotional progress. For example, the clouds are dark blue for depressed thoughts, indigo for frustrated feelings, etc. Yet, if I want to review and analyze further about what exactly happened, I could also tap on the clouds, and my original thoughts in texts would be revealed. All the blurred colored clouds would then disappear after a season (3 months) as a way to let go of the dark memories and have a brand new start along with our four seasons.

Let It Out fully immerses me in the externalization of my thoughts in a 3D virtual space around me with the least amount of energy needed, rather than dividing me from my thoughts with a piece of paper.
At the end of the experience, those thoughts become a cloud that is color-coded based on its sentiment tone, so I would not be reminded of the bad places I was in but still able to track my emotional progress.

Technical Details

Speech to Text X Tone Analysis X AR Foundation.

As to the input, originally I thought of just writing on the screen using my finger, but after play testing some existed AR graffiti apps, I realized our muscle memory for writing doesn’t allow it to be natural. And then I thought ok what if I keep the muscle memory for writing and use gesture recognition for mobile AR, but I realized it is such a new technology it’s other school’s group thesis and, of course, not open-source. That leads to where I landed, voice command.As another very effective way to cope with our emotions is to talk with our friends.By saying our thoughts and emotions out loud, it also helps us process just like journaling.Since I’m addressing on the isolated low energy state. I thought why don’t I talk with myself instead and still saying my thoughts and feelings out loud just like talking with my friends. So, I’m using Speech to Text as an input for my AR spatial journaling experience, and the energy required for this experience is as low as talking.

I use IBM Watson Speech to Text service for the input of Let It Out and feed the transcript to a 3D generator for the output. I also am trying to use Watson Tone Analysis for color-coded clouds. The app is built in the form of augmented reality using AR Foundation in Unity.

Design Element Theme

Dreamy.

To make Let It Out as dreamy and as healing as possible, I incorporated these three elements - acrylic 3D fonts, the yellow-orange, or blue-gray-ish fog appearing and disappearing, and the clouds that the text transformed in the end.

Acrylic 3D fonts
Foggy Atmosphere
Cloud
Acrylic 3D fonts
Foggy Atmosphere
Cloud