<p><img src="/ace-step-1.png" alt="The entrypoint for the App Creation Experience"></p>
<h2>Context</h2>
<h3>Why we did this research</h3>
<ul>
<li>Airtable's core team knew from previous research and enablement that it was just too difficult to get started building useful apps in Airtable, especially in the era of building interfaces. We had defined an Airtable app as data sources, automated workflows, and a user-facing interface layer working together to orchestrate meaningful business processes.</li>
<li>Over the course of two quarters prior, Core product design had been working on a vision for an app-centric Airtable--a way to make both the base and the interface first as a means of getting people started on Airtable more quickly. In Q2-Q3 2023, two key pods in App Building pillar were sprinting to build a portion of the experience that would be known as the App Creation Experience (ACE). This was a big risk because:
<ul>
<li>Building interface-first was an unproven mental model</li>
<li>We weren't sure it was going to result in a useful app</li>
<li>We weren't sure it was going to be an easy-to-use experience</li>
<li>We weren't sure if it was going to appeal to our key unlock audience--lower sophistication builders who historically have had a harder time getting started</li>
</ul>
</li>
<li>Teams were planning to release the experience in beta with the help of customer enablement but didn't have a plan for how they would assess the impact of these uncertainties systematically</li>
<li>My research partner and I roadmapped how UXR would lead the beta over the course of the quarter and planned a four-phase iterative research program up until launch</li>
</ul>
<p>The goals of the research overall were to assess to what extent ACE was:</p>
<ul>
<li>Easy to use</li>
<li>Resulting in useful apps</li>
<li>Valuable to less sophisticated builders</li>
</ul>
<p><img src="/ace-overview.png" alt="The vision for the App Creation Experience"></p>
<h2>Methodology</h2>
<h3>Initial plan: A 3-month research program with iterative sprints</h3>
<p>This was the initial plan that I co-developed with my research partner as part of our 2023 Q2 planning cycle. Each sprint: 1 week recruit, 1 week moderation (2 UXRs), 1 week analysis and synthesis while design iterated on our results.</p>
<p><img src="/ace-sprint-1.png" alt="Initial plan"></p>
<h3>Executed plan: A 4-month research program with iterative sprints</h3>
<p>As the program evolved, we flexed on our plan given eng constraints, recruiting difficulties, and the needs of our cross-functional team.</p>
<p><img src="/ace-sprint-2.png" alt="Our final plan"></p>
<h3>How and why we made these choices:</h3>
<p>We chose a flexible research beta because we knew that the final experience that we shipped was going to be a combination of features, not all of which were ready at the same time.</p>
<ul>
<li><strong>Why sprints?</strong> We chose these to allow design to make changes and then have subsequent phases to test those changes</li>
<li><strong>Why two UXRs?</strong> My research partner and I both worked on the same pillar (App Building), supporting multiple pods contributing to ACE</li>
<li><strong>Why multiple eng environments?</strong> We flexed on staging, production, or prototype testing depending on eng readiness. For testing done in eng environments, we included product features (such as workflows and App Builder improvements) as they became available. For fourth phase, we ran additional ad hoc sessions due to design needs that we prioritized with our design team</li>
<li><strong>Why different recruitment criteria for each phase?</strong> We adjusted recruitment methods based on difficulty to get enterprise FTE 1K+ customers with the use cases that were built. We also took into account how ready and interested users were to build by screening for build intent with a screener questionnaire.</li>
<li><strong>How did we include other team members?</strong> We solicited feedback from research manager and our cross-functional partners for planning, recruitment, and discussion guides, primarily via Slack and Zoom calls as needed</li>
</ul>
<h2>Execution</h2>
<h3>Research sprint ways of working</h3>
<ul>
<li>Each of us contributed to writing up the research plan and discussion guide for that phase</li>
<li>Shared out findings with stakeholders via Slack, discussed as needed in existing weekly ACE meetings, in 1:1s, or in weekly design-research app building pillar syncs</li>
<li>Alternated leading the interviews and taking notes</li>
<li>Invited stakeholders as observers to sessions</li>
<li>Note-taking UXR would lead the Slack hype thread 🧵 for observers</li>
<li>Alternated writing up end of session summaries in Slack for cross-functional partners</li>
</ul>
<h2>Analysis and synthesis</h2>
<h3>How we analyzed and synthesized</h3>
<ul>
<li>Prioritized speed so as to give design time to iterate on our findings</li>
<li>Preserved Friday as our analysis and synthesis day</li>
<li>Divided up sessions and translated session notes into stickies in collaborative Figjam* async</li>
<li>Took turns grouping into themes and writing up our takes on synthesis</li>
<li>Met via Zoom or IRL to talk through themes and write up an EOW summary on Friday</li>
</ul>
<p><img src="/analysis-ace.png" alt="A recreated sample of how we went about our analysis and synthesis together in Figjam"></p>
<h3>What we learned</h3>
<h4>Sprint 1 - Inventory management use cases</h4>
<p>In Sprint 1, we learned that people found the experience usable but experienced confusion with the linked relationships step. Design iterated on this step for Sprint 2 and addressed various usability issues.</p>
<p><img src="/insights-inventory.png" alt="Key insights from Sprint 1"></p>
<h4>Sprint 2 - Marketing use cases</h4>
<p>In sprint 2, we learned that people had trouble recalling the workflow choices they had made. This was a risk because some workflows notified other users, which could happen without the builder realizing. We decided to include a warning step before publishing the app to make sure builders were aware that workflows that could notify others were about to be deployed.</p>
<p><img src="/insights-marketing.png" alt="Key insights from Sprint 2"></p>
<h4>Sprints 3 and 4 - July private beta</h4>
<p>After completing sprints 3 and 4, we had a more holistic picture of the extent to which our new app building experience was helping people build differently with Airtable.</p>
<ul>
<li><strong>Has ACE been successful at helping builders make useful apps quickly?</strong>
<ul>
<li>ACE helps builders generate a starter app quickly because it helps them get to a draft state faster, gets basic elements right, and shows them a trusted POV for how to start. However, apps aren't always close enough to what builders need to be useful without more customization.</li>
</ul>
</li>
<li><strong>Who is ACE best for?</strong>
<ul>
<li>ACE as it will exist is best for builders who need to build something new that may or may not reference, rather than build on, existing data in airtable. Advanced and less advanced builders alike saw value in making apps this way because of the time saved and how it helped them visualize the best way to present data.</li>
</ul>
</li>
<li><strong>What did more successful builders do or know that less successful builders did not?</strong>
<ul>
<li>Builders who got further with ACE knew more about Airtable Interfaces. They were unafraid to try things out and were further along in their understanding of what the app-first vision is.</li>
<li>Clicking to learn is a pattern we observed and throughout base-building history. Those participants whose clicks were cheaper explored more. Most beta builders expected to need to "prepare" data in the base before making an interface.</li>
</ul>
</li>
</ul>
<h2>Decisions informed by research</h2>
<h3>Key decisions</h3>
<ul>
<li><strong>Redesign core page selector UI to emphasize multi-select</strong>. We observed that people had trouble understanding that they could select more than one core page for their app even though some people wanted to. Design worked to revise this selector so that it was more obvious that multiple pages could be selected.
<img src="/core-pages-checkbox.png" alt="Redesigned core page selector"></li>
<li><strong>Add more context to core pages to help with comprehension</strong>. We observed that people were getting to the second workflow selection screen without really understanding what they chose on the core page selector screen. Design worked to add more context to the core page selector screen to reduce the cognitive load folks were experiencing on the workflows screen.</li>
<li><strong>Change workflows UI to eliminate repetition of workflows across multiple core pages</strong>. We observed that people were getting confused that they could add the same workflow to different core pages (e.g., adding a "weekly digest" for sprints but also wanting a "weekly digest" for tasks).</li>
<li><strong>Add visual preview of workflows in wizard</strong>. We observed that people were having a hard time understanding what the workflows did. Design included workflow illustrations to help with this.
<img src="/workflows-preview.png" alt="Redesign of visual preview of workflows"></li>
<li><strong>Add explanation of sync impacts and limitations so people can get their data in faster</strong>. We observed in the third and fourth sprints of this project that once people tried to bring in existing Airtable data from other bases via sync, there were some significant usability issues blocking them from successfully setting up the sync. We met with the sync team to discuss possible fixes to people not understanding where to go to set up sync and also what the effects of sync would be (e.g. is it a one-way sync, two-way sync, etc). Design worked to come up with a fix that eng could implement quickly as a fast follow.</li>
<li><strong>App builder education improvements: tooltips for linked record discovery</strong>. We learned early in this project that people did not understand how Airtabler was creating linked relationships between core pagers as a part of the setup process. As a result, we recommended that we show rather than tell people linked relationships in App Builder once these connections had already been made.</li>
<li><strong>Have builders land in preview mode so they can play to understand their app</strong>. We observed that once people landed in App Builder (the edit state of their built app), they clicked around to learn how their app worked, but the editing tools were getting in their way. We recommended implementing a preview step before allowign folks to edit their apps so that they might learn how their app works and decide if the generated app is what they need.
<img src="/preview-mode.png" alt="Preview mode"></li>
</ul>
<h2>Socialization and next steps</h2>
<ul>
<li>We shared out end of week summaries as needed throughout the project rather than waiting to deliver insights in a final report (to get insights and findings to design quicker)</li>
<li>We created a Sprint 2 report and presented findings in a share out (I led the share out presentation), with Sprint 1 findings included as a fast follow</li>
<li>We also created a Sprints 3 and 4 report (My research partner and I created the deck together), with clips, insights, decisions we made with design examples when applicable, and additional UXR recommendations</li>
<li>My research partner and I were preparing for a read out that we’d have done together prior to a reduction in force that eliminated my position</li>
</ul>
<h3>Other next steps</h3>
<p>Q3 September and Q4 October</p>
<ul>
<li>Wrote up a research plan for a scaled approach to feedback for ACE private and open beta (multiple surveys for different cohorts)</li>
<li>Worked with lifecycle marketing to do multiple sends to cohorts over several week period to gauge app usefulness and value</li>
<li>Incentivization plan (first XXX responses get $XX)</li>
</ul>
<p>ACE is available to select users as of November 1, 2023 (it can be accessed by clicking "Create a base" on the Airtable homepage). Please feel free to contact me if you have additional questions about ACE and its impact.</p>

Case Studies | Airtable App Building Research Beta

A four-month research program for Airtable

App Building Beta (2023)

<h2>Context</h2>
<h3>Why we did this research</h3>
<p>In 2021, people building bases in Airtable were having a hard time making them useful. Activation team within the Growth pillar was working on improving the leaky activation funnel (22%* drop from 'try' to 'build'). Creators were trying out Airtable but not reaching a stage we had defined with DS as more active building.</p>
<p><img src="/builder-funnel.png" alt="The builder journey as defined by data science. *Consider the %s as gestures towards the magnitudes of the dropoff we faced"></p>
<p>Although Airtable was powerful in knowledgeable hands, it had some steep learning curves. An application’s tailorability curve shows the relationship between the skill an app demands of its users and the power they get from it.</p>
<p><img src="/steep-slope.png" alt="From Ludwig, Thomas &#x26; Dax, Julian &#x26; Pipek, Volkmar &#x26; Wulf, Volker. (2017). A Practice-Oriented Paradigm for End-User Development. 10.1007/978-3-319-60291-2_2."></p>
<p>Previous persona research indicated that “database-savvy” builders had less trouble taking advantage of Airtable’s powerful features, but they were not the majority of our users.</p>
<p>Adding features that make an application easier to use “smoothes” the curve and lets more people take advantage of the app’s power.</p>
<p><img src="/gentle-slope.png" alt="From Ludwig, Thomas &#x26; Dax, Julian &#x26; Pipek, Volkmar &#x26; Wulf, Volker. (2017). A Practice-Oriented Paradigm for End-User Development. 10.1007/978-3-319-60291-2_2."></p>
<p>We also knew that creators had three ways of getting started on Airtable--scratch, template, or import--and knew that import was the most successful, but it was less often used.</p>
<p><img src="/three-building-methods.png" alt="At this point in Airtable history, builders could use any of three ways to get started building with Airtable."></p>
<h2>Research Goal</h2>
<p>We wanted to understand how creators used these three ways of getting started and how that affected them over time. We also wanted to understand qualitatively what the blockers were to creators getting started with Airtable and what we might do to be able to smooth Airtable's own difficult tailorability curve.</p>
<h2>Methodology</h2>
<ul>
<li>We knew that it took some time for builders to get to something more "finished" in Airtable. Because of this, we chose to examine how creatbuildersors built in Airtable over time</li>
<li>The research design consisted of three key phases:
<ul>
<li>Initial session: 30-min session getting builders up to speed on the study objectives, what we expected of them, and what they were hoping to build</li>
<li>Diary study portion: Airtable form "diaries" that we asked builders to submit any time they did something in Airtable. We also asked participants to record a weekly video Loom of their progress so far. We compensated them if they submitted at least one diary entry and the video. We kept them in the study for three weeks.</li>
<li>Final session: 30-min session where builders reviewed their progress with us, how far they got, and problems they had</li>
</ul>
</li>
<li>We recruited:
<ul>
<li>Self-serve brand new signups (daily sends coordinated with lifecycle marketing), active users, no gov/edu (did get one edu who we kept in the study)</li>
<li>Screened and selected for corporate use cases</li>
<li>Screened for a range of database-savviness, which included familiarity with Excel, databases, and programming experience</li>
<li>Aimed for approximately 20 people due to expected dropoff and ended up with around 8-10 people who did final sessions with us</li>
</ul>
</li>
</ul>
<p><img src="/diary-study-method.png" alt="Timeline and methods chosen"></p>
<h2>Execution notes</h2>
<p>Over the course of 5-6 weeks I did all of the initial and follow-up sessions, much of them remotely in Florida.</p>
<p>Head of Research Andy Warr and I built an Airtable app to run the diary study in. I collected participants and their information via the screener form that I used to recruit via daily SQL pulls that I coordinated with lifecycle marketing.</p>
<p>I then selected participants for the study based on criteria I aligned on with my cross-functional team.</p>
<p><img src="/participants.png" alt="A recreation of the participants table"></p>
<p>After initial calls, participants were to submit a diary log whenever they used Airtable.</p>
<p><img src="/session-log.png" alt="A recreation of the diary log participants used"></p>
<p>Once participants submitted their logs, I could review them in Airtable.</p>
<p><img src="/example-log.png" alt="A recreation of a session log. I used Interfaces now but did this all in the base back in 2021."></p>
<p>Participants received weekly automated reminders to submit their Loom video logs, customized to their time zone.</p>
<p><img src="/email-automation.png" alt="Automation email example"></p>
<p>When final sessions were complete, I created timelines for each participant in Miro with stickies for each action over the course of three weeks and synthesized themes that I observed.</p>
<h2>Synthesis and socialization</h2>
<p>After analyzing and synthesizing the results, I created a Slides report and a shorter presentation that I presented to the Product Development team, including our CPO.</p>
<p>The diary study was one of the only longitudinal studies done at Airtable, especially on builders, so it was used as an artifact for understanding builders over time for quarters after I completed the work.</p>
<p>Here's a summary of what we learned and what we did about it as a result of this research.</p>
<h2>What we learned</h2>
<h3>Findings</h3>
<p><strong>Participants were building with a combination of templates and "from scratch" building methods</strong></p>
<p>This was because templates got them to an "end state" where they could envision what they wanted. However, templates were too difficult to customize, so participants felt like they had to build from scratch.</p>
<p><strong>Participants who were more database-savvy got more out of Airtable faster because they didn't have to work so hard to figure out how it worked - schema design was hard</strong></p>
<p>Database-savvy participants are used to thinking in systems and understand that data sets in a tool like Airtable are not like tabs in a spreadsheet. They also have some concept of relationality. Participants that didn't have this struggled to understand key high ceiling features like linked records, which we knew from a DS analysis had high precision and recall for a way we defined "success" with a base.</p>
<p>Core product ended up tackling the schema design challenge with the App Creation Experience in early 2023 (see that case study for more information about ACE).</p>
<p><strong>Participants built in short sessions over time, and mistakes were necessary for learning</strong></p>
<p>We observed a key pattern that repeated up through the ACE beta, which was that it took participants making mistakes and charting their way through Airtable to figure it out. However, Airtable is not a graceful tool when it comes to making mistakes and playing around.</p>
<p>I asked participants to estimate how long they spent in Airtable with each diary entry and included a chart of this in the final report. I saw that participants tended to spend 15-20 minutes in Airtable, with some participants spending up to 60 minutes in it.</p>
<h3>Recommendations</h3>
<p><strong>Participants were building with a combination of templates and "from scratch" building methods</strong></p>
<p>We were already pursuing a new NUX experience, but this insight dovetailed into Core Product's strategy of pushing for a more opinionated app-building experience that came several quarters later.</p>
<p><strong>Participants built in short sessions over time, and mistakes were necessary for learning</strong></p>
<p>We introduced a second milestone of our new NUX experience, which was a checklist people could refer back to over time. This experience significantly increased one of our key success metrics.</p>
<h2>What I'd do differently</h2>
<p>This research landed a little too late to make it into the PRD for the first iteration of our new NUX experience, even though the results supported much of what we were doing. This was because the diary study analysis took a long time to complete. Next time I would budget more time for the analysis or do some more concurrently over the course of the three weeks.</p>
<p>If you have any more questions about this research and its impact, including specific metrics, please feel free to contact me.</p>

Case Studies | Airtable Builder Diary Study

Studying building methods over time

Builder Diary Study (2021)

<p><img src="/maximo-banner.jpg" alt="Máximo the Titanosaur’s reveal image."></p>
<h2>Context &#x26; Overview</h2>
<p>After redesigning their website with us in 2018, Field Museum approached Purple Rock Scissors with a new brief—to research, design, and develop a conversational AI for their new titanosaur, Máximo.</p>
<p><img src="/maximo-main.png" alt="Máximo’s chat window on Field’s website"></p>
<p>Working closely with the Field’s digital engagement team, my team developed a flexible, conversational personality for Máximo that went live on the website as well as over SMS. You can talk to Máximo [here] (<a href="https://www.fieldmuseum.org/exhibitions/maximo-titanosaur?chat=open">https://www.fieldmuseum.org/exhibitions/maximo-titanosaur?chat=open</a>).</p>
<p>As of October 2019, Message Máximo averaged around 72% accuracy for user queries and has had over 7,000 conversations.</p>
<p>Message Máximo has since been honored with a 2020 Webby Award Honorable Mention for Machine Learning and Bots. Message Máximo was also honored with a 2020 Silver MUSE award for Best Mobile App from the American Alliance of Museums.</p>
<p><img src="/maximo-webby.png" alt="Message Máximo’s Webby Award nomination"></p>
<h2>My Role</h2>
<p>I was the senior researcher and conversation designer for Message Máximo.</p>
<p>I was responsible for defining the scope of the agent’s knowledge, crafting conversation logic, conducting user research, and driving continuous improvement for the agent’s future content releases.</p>
<h2>My Team</h2>
<ul>
<li><strong>Senior Researcher &#x26; Conversation Designer:</strong> Caitlin Pequignot (PRPL)</li>
<li><strong>Developer:</strong> Brandon Chang (PRPL)</li>
<li><strong>Senior Project Manager:</strong> Francesca Parker (PRPL)</li>
<li><strong>Director of Technology:</strong> Jim Powell (PRPL)</li>
<li><strong>Digital Content &#x26; Engagement Manager:</strong> Caitlin Kearney (Field Museum)</li>
<li><strong>Social Media Manager:</strong> Katharine Uhrich (Field Museum)</li>
<li><strong>Digital Product Specialist:</strong> Andrea Ledesma (Field Museum)</li>
<li><strong>Web &#x26; Digital Engagement Director:</strong> Brad Dunn (Field Museum)</li>
</ul>
<h2>Discovery &#x26; Planning</h2>
<h3>Define Máximo the conversational AI from his established identity.</h3>
<p>Field Museum and design agency Leo Burnett had done extensive work creating Máximo’s personality and identity.</p>
<p>I audited his existing brand identity and worked closely with members of the web team responsible for Máximo’s existing personality on social media. We found three key things that we needed to do to help define what Máximo the conversational agent would do.</p>
<h4>1. Define Máximo’s scope of knowledge - what can he talk about?</h4>
<p>Interviews with my web team stakeholders revealed the topics that users were most interested in were what Máximo liked to eat, how big he was, and how old he was. These use cases helped to clarify the types of intents that key users would be most likely to ask him.</p>
<h4>2. Define Máximo’s use cases - how can users talk to him?</h4>
<p>Through sentiment analysis and interviews, I found that it was important for users to interact with Máximo in person and on the web. I chose to use SMS so that users could talk to him in person and an in-browser window so that users could talk to him remotely.</p>
<h4>3. Define Máximo’s audience - who are the people most likely to talk to him, and what do they want to know?</h4>
<p>Due to budget and time constraints, I couldn’t speak with actual museum visitors during this phase. However, I was able to take advantage of the Field’s extensive work in this area already and used it to draft key personas for Máximo.</p>
<p><img src="/curious-expert.png" alt="Persona A: The Curious Expert"></p>
<h2>Agent Design</h2>
<p>With key topic areas in mind from my work with the Field’s team, I set out to write and organize the core intents that would form the foundation of Máximo’s scope of knowledge.</p>
<h3>Intent Creation &#x26; Prioritization</h3>
<p>Using Airtable and the knowledge categories we had decided upon, I brainstormed all of the different intents we might want Máximo to focus on.</p>
<p><img src="/airtable.png" alt="Snippet of our Airtable database we used to organize intents"></p>
<p>I grouped these by topic, flagging each for priority. Here are examples of some of our primary intent categories:</p>
<p><img src="/intent-priority.png" alt="Primary intent categories for Máximo’s MVP launch"></p>
<p>The Field team and I prioritized groups of intents for copywriting and launch. I worked to write the training phrases and guide copy for the type of response needed to facilitate a happy interaction. The fantastic Katharine and Caitlin from the Field team worked together to craft on-brand responses for Máximo.</p>
<h3>Intent Journey Mapping &#x26; Testing</h3>
<p>As I began journey mapping Máximo’s intents, I began to see how folks would likely encounter his primary intents in an overwhelmingly exploratory conversation structure.</p>
<p>After diagramming the first round of conversational flows, it became clear that  the questions users asked the agent did not necessarily follow a particular order. </p>
<p><img src="/conversationablarge.gif" alt="One small conversational difference in our initial journey mapping can diverge the conversation significantly, even though the conversational “goal” is unaffected."></p>
<p>I concluded that conversations with this agent were better organized by topic in a database structure, rather than trying to account for every possible linear flow. We committed as a team to managing intents with categorical labels in Airtable and using that as our source of truth before migrating content releases into DialogFlow.</p>
<h2>Agent Beta Launch &#x26; In-Person User Testing</h2>
<p>I conducted my first round of beta testing in-person in the museum lobby with the Field team. Using demographic criteria that mapped back to our personas, we approached visitors with a laptop or tablet in order to answer some primary questions:</p>
<ol>
<li>How easy is it for people to initiate conversations with Máximo?</li>
<li>What questions do they tend to ask him?</li>
<li>What areas of conversation does he tend to fail at/excel at the most?</li>
</ol>
<p><img src="/maximo-real.jpg" alt="We conducted the tests around Máximo himself in Stanley Field Hall. Users could approach our booth with our laptop, while other team members approached willing groups of folks with tablets around this area."></p>
<h3>Users expected the agent to know more than it did.</h3>
<p>I scripted and tested the agent prototype in DialogFlow. Lobby visitors were generally enthusiastic about talking to Máximo, especially one particular group (redacted for privacy), exclaiming that they “[thought] this [was] going to be so cool when it’s done!”</p>
<p>However, the NLP engine failed in matching even basic responses. I found that we had many duplicate intents that made this problem worse.</p>
<h3>Users were frustrated by the agent’s fallback responses.</h3>
<p>By observing many users talking to Máximo, I began to see patterns in the types of constructions that triggered the fallback response (the standard “failure” response for when Máximo can’t match a query).</p>
<p><img src="/Dialog+Conversation+1.png" alt="An example of the type of query format that Máximo couldn’t initially answer."></p>
<h2>Beta Launch Content Improvements</h2>
<h3>Creative fallbacks improved user satisfaction.</h3>
<p>I discovered that a lot of folks’ questions had grammatical patterns in them that could fuel conversational opportunity, especially since Máximo had the freedom of an educational and conversational interaction.</p>
<p>I implemented a variety of intents powered by grammatical listeners to fix the fallback problem. I designed these listeners to pick up the “gray area” of clauses that were commonly used.</p>
<p>Here are just a few examples.</p>
<h4>“Why” clauses</h4>
<p>The “why” clause was designed to detect training phrases such as: “why is the sky blue”. It’s not intended to detect a phrase like “why are you so big?” since I designed specific response for that.</p>
<h4>“I think” clauses</h4>
<p>The “I think” clause was designed to detect phrases like: “I think that is a stupid idea”, but not “I think you are an awesome friend”.</p>
<p><img src="/Dialog+Conversation+2.png" alt="An example of the type of the “do you think” intent that catches all instances of “do you think” clauses that don’t have a specific intent to match."></p>
<h2>Pre-Launch Testing Results</h2>
<h3>Our additional testing led to improvements in agent training and overall mobile experience.</h3>
<p>Our second round of user testing revealed several opportunities to improve the mobile web experience. </p>
<p>For this testing, we wanted to test the following objectives:</p>
<ul>
<li>Identify training improvements to the NLP engine to see if responses were better aligned with user intent</li>
<li>Assess how usable the mobile web chat window was for users of various demographics</li>
</ul>
<p>We ended up testing 85 distinct user groups split across different demographics.</p>
<p><img src="/users-surveyed-maximo.png" alt="The proportions of folks we were able to test in our second round of usability and content testing."></p>
<ol>
<li>
<p>Users had some difficulty understanding how to pull up the iOS keyboard on iPhone. This led to us incorporating <strong>mobile-specific help text</strong> in the input placeholder to better frame expected user behavior.</p>
</li>
<li>
<p>Families with <strong>elementary school-aged children had the longest engagement time</strong> and the most enthusiastic questioning. They were also the largest group we surveyed.</p>
</li>
<li>
<p>Users still had some trouble posing initial questions. This frustration could be reduced with <strong>better framing of the welcome intent</strong>, on which we are continuing to iterate.</p>
</li>
<li>
<p>Overall user attitude toward the agent was overwhelmingly <strong>positive</strong>. Most negative interactions were due to the agent not matching questions properly, users not being sure what to ask, or connectivity problems.</p>
</li>
</ol>
<p><img src="/user-attitude.png" alt="Folks’ attitudes towards Máximo as we tested him"></p>
<h2>Post-Launch Analytics</h2>
<p>One of the most important drivers of a satisfying chatbot experience is how accurately the agent responds to users. Sampling user sessions from a live chatbot and tracking bot performance over time is essential.</p>
<p>For Máximo, the team and I used Dialogflow analytics to study how he has responded to users since his soft launch. Dialogflow’s conversation flow was vital in exploring how users move through an ecosystem of agent intents.</p>
<p>To get a true sense of how well Máximo was doing, I took a sample of 50 user sessions at soft launch and 50 sessions at public launch, then compared the results after training Máximo. By organizing user session accuracy into Airtable, we were able to identify some valuable insights on his performance.</p>
<p>During the week of his soft launch, Máximo was consistently accurate across short, medium, and long conversations. We were surprised by this, hypothesizing that he might be more accurate for shorter conversations.</p>
<p><img src="/maximo-results.png" alt="Máximo’s accuracy across short, medium, and long conversations, as defined by # of turns"></p>
<p>After two weeks of training, we found that Máximo’s accuracy for long conversations had actually increased to 79%. It’s possible that over time and with larger sample size, Máximo would be more accurate with short conversations, since users don’t present him with as many questions. However, it’s also possible that users engaging with Máximo for short conversations may tend to ask more conventional questions, which are easier for him to answer.</p>
<h2>Results</h2>
<p>After months of being live, Máximo averaged around 72% accuracy in October 2019 and had had over 7,000 conversations up to that date.</p>
<p>I was able to present the work I had done with the Field team at Museum Computer Network’s 2019 conference in San Diego, California.</p>
<p>Message Máximo has since been awarded a 2020 Webby Award Honorable Mention for Machine Learning and Bots. Message Máximo was also honored with a 2020 Silver MUSE award for Best Mobile App from the American Alliance of Museums. You can also read Live Science’s review of his chatbot feature [here] (<a href="https://www.livescience.com/65837-titanosaur-maximo-chats-with-you.html">https://www.livescience.com/65837-titanosaur-maximo-chats-with-you.html</a>).</p>
<p><img src="/maximo-press.png" alt="Máximo’s Live Science feature."></p>
<p>If you have specific questions about this work, please feel free to email me at <a href="mailto:cmpequignot@gmail.com">cmpequignot@gmail.com</a>.</p>

Case Studies | Message Maximo

Researching a conversational agent for Field Museum

Message Máximo (2018)

<p><img src="/session.jpg" alt="Some members of our ideation group thinking through early stages of the CAPEX decision-making prototype."></p>
<h2>Context &#x26; Overview</h2>
<p>In the wake of Hurricane Irma, I flew to Cleveland to partner with Siemens, a major power plant company, and two members of my Purple Rock Scissors team.</p>
<p>Siemens had engaged us to help them rapid prototype a solution to one of their clients’ biggest financial challenges: helping them visualize and make decisions around their $5 billion coal plant mortgage—all in just three days.</p>
<h2>My Role</h2>
<p>I was a supporting product strategist in the workshop and the lead strategist on the requirements collection and writing for the MVP.</p>
<h2>Purple Rock Scissors Agency Team</h2>
<p>-<strong>Product Strategist</strong> Caitlin Pequignot
-<strong>Vice President</strong> Michael Parler
-<strong>UX Designer</strong> Joshua Hurtado</p>
<h2>The Problem</h2>
<p>Upon conducting initial discovery sessions with the Siemens and coal plant company team, we learned that the power plant company needed to decommission key coal plants in their fleet to comply with government regulations by year 2035.</p>
<h3>The costs of running a coal plant are passed onto the customer.</h3>
<p>Through a series of interviews with coal plant managers and executives, I got a 30,000-ft view of how energy is generated from coal plants, how it is sold, and how that money is leveraged against the debt that power plant corporations hold to fund their facilities. That financial burden is then passed on to customers in the form of more expensive energy.</p>
<h3>Energy costs to customers could be reduced by making smart investment decisions.</h3>
<p>I led discovery sessions that revealed that coal plant companies can reduce the cost of energy by slowly decommissioning the coal plant over time. This can be done by extending the life of certain parts of the coal plant.</p>
<h3>The coal plant team lacked a clear way to risk model these investment decisions.</h3>
<p>Not only was it important for the team to forecast effects of investment decisions, but they had no way to visualize that risk modeled outside of conventional Excel spreadsheets. </p>
<p><img src="/paper-prototyping.png" alt="My agency team and I doing some initial paper prototyping."></p>
<h2>The Solution</h2>
<p>Together with the Siemens and coal plant team, we hypothesized that visualizing the effects of possible investments would help managers and executives make better decisions.</p>
<h3>The company had two distinct user groups with different needs.</h3>
<p>In order to make good investment decisions, coal plant managers needed to see how unit and part-level investments affected plant productivity, while executives needed to see fleet-level consequences. I led open card sorting workshops to surface these needs.</p>
<h3>Plant managers and executives wanted a holistic view of their investment decisions.</h3>
<p>Despite their different needs, both users groups wanted the ability to create simulation models in one seamless interface. We also needed to consider how managers and executives would be able to forecast effects of CAPEX investment decisions using common UI elements.</p>
<p>By the end of the second day, my UX designer and I converged on three main views for the power plant visualization:</p>
<ul>
<li><strong>Fleet view</strong> would help company executives visualize that 30,000-ft view of their decision-making.</li>
<li><strong>Plant view</strong> would help plant managers see how component-level investment decisions might affect the productivity and efficiency of their overall plant operations.</li>
<li><strong>Component view</strong> would help plant managers assess how investment decisions affect the day-to-day, and eventually minute-to-minute efficiency of plant components, such as turbines and boilers.</li>
</ul>
<p><img src="/prototype-1.jpg" alt="An early executive view of a sample power plant. Macro controls for unit-wide operation are included just under the default simulation dropdown, as managers needed to be able to adjust these quickly based on daily energy demand. Available projects for simulation can be selected using the accordions shown. Effects on overall coal plant mortgage (CPM) are shown in the data visualizations to the right."></p>
<h3>Both groups needed to compare and contrast simulations.</h3>
<p>I led lo-fi prototyping sessions that helped surface the plant managers’ need to compare and contrast different investment simulations. We decided that each view would be governed by an overall simulation engine. This allowed users to load, save, and view simulations that pulled from a database of potential investment projects.</p>
<p><img src="/prototype-2.jpg" alt="An early manager view of an individual plant. Here, macro controls are still visible for each unit within the plant, and projects can be still applied using the accordion in the sidebar. Effects on expected life of individual pieces of equipment can be explored using the tab system on the right."></p>
<h2>The Result</h2>
<h3>Our prototype became a web app with more phases to come.</h3>
<p>The desktop web app empowered our users to make better decisions.</p>
<p>By the end of the third day, we had produced a clickable prototype of a potential web app for the power plant company and presented it to the president of the company. </p>
<p>Over the course of the next five months, I worked with Siemens data scientists, project managers, and developers from my team to bring the prototype phase of the project into reality. As our visual designer made additional feature and visual design iterations, I wrote detailed functional requirements for each app screen.</p>
<p>A phase 1 of the internal web app tool finished development in mid-2018.</p>
<h2>Next Steps</h2>
<h3>Planned upgrades will help users make even better decisions.</h3>
<p>More complex risk analysis was planned for phase 2.
Plans to incorporate more complex risk analysis, as well as including more plants from the company’s fleet, were planned to be included in phase 2 of this build.</p>

Case Studies | Siemens

Creating a Simulation Visualizer with Siemens

CAPEX Simulation (2017)

Case Studies | Snap Focus

Discovery and evaluative research for Snap’s learning platform

Snap Focus (2019)

<h1>driving across nebraska</h1>
<p>driving across nebraska, i could count hay bales,
but instead
i am counting feelings,
and failing,
because feelings cannot be numbered, only
misunderstood.</p>
<p>i am watching the fields of our food go by,
the earth we turn to make our lives go on,
block by block in bodegas
large and small.</p>
<p>and i am struck by how much i do not understand,
the silver irrigating arches, the geometry of grasses,
the spaces between people and the ideas rooting deep and alone
in the houses far from me.</p>
<p>and i feel the echoes chasing me down highways and through time,
i am driving west but i am making revolutions around lake eola still,
where the helicopters are chopping up the sky into jagged pieces
that have been falling for two years, tearing me down
to the coldness of a root cause
that no verse has helped me name.</p>
<p>and i cannot close it cleanly,
i am not much further along —
in the flatlands are we not always the center of our skies?
except i am more west than i was forty minutes ago,
watching nebraska’s clouds patching up the pieces
of our atmosphere.</p>

driving across nebraska | Poems by Caitlin Pequignot

June 2018

driving across nebraska

<h1>good UX</h1>
<p>one day i did write about UX, and it was<br>
the eve of the end of a thing, for things either end<br>
or evolve, and we all knew it was getting<br>
harder and harder to breathe.  </p>
<p>you strummed the guitar, i sang<br>
about the little boat at sea -<br>
for two years i lived on sinking sand,<br>
and i sang and i sang and i knew,<br>
the salt was sinking deep in me.  </p>
<p>and you heard me on the balcony<br>
where i was drowning, 20 floors above<br>
the skyline.  </p>
<p>run the function still, accidental IKEA<br>
and a bad IP return<br>
you and me?  </p>
<p>and i said i loved you on the balcony<br>
where i was crying, 20 floors above<br>
my sea.  </p>

good UX | Poems by Caitlin Pequignot

December 2017

good UX

<h1>on tai shan (or, how i ate too many choco pies)</h1>
<p>i write my chinese name on every page,<br>
but it is fake —<br>
my surname made bai (白)<br>
as a joke upon my skin.<br>
for my first he named me kanglin (康琳)<br>
(healthy jade), yet another<br>
approximation of my being,<br>
until i grew into the ring of it,<br>
my voice pitched higher and higher,<br>
singing tones like a song<br>
until i could no longer hear him.  </p>
<p>i write my chinese name and won’t forget —<br>
but he has, so some have said that maybe<br>
so should i.  </p>
<p>not so, i argue, for where would the dreams<br>
of bai kanglin dwell —<br>
forged in the smog-laden streets<br>
or on tai shan, looking east<br>
or west<br>
when, she decided, that it<br>
no longer mattered?  </p>

on tai shan | Poems by Caitlin Pequignot

June 2016

on tai shan

<h1>orbitals</h1>
<p>there was a mahler chord in the anthem today,<br>
it struck like the light does at dusk, uncommonly.
angled, Dust between the panes —<br>
it was only f major, but.
in context it was more.  </p>
<p>i could have left after the resolution,<br>
but maybe it was the baptist in me —<br>
i climbed the balcony and let the sermon.
swing me into orbit, closer to the<br>
sun.  </p>
<p>what is it that draws me in,<br>
after proselytization turns me cold as<br>
pluto in its distant spin, unremarkably alone?  </p>
<p>is it the promise that i am a child of some vaster universe, that<br>
i am dust beyond that which i can fluoresce in basement labs,<br>
particulate and inarticulate in what i feel<br>
and can believe?</p>
<p>or is it my theory that even bridges built<br>
without straw can still fall short —<br>
what, in context on the brink of hell,<br>
angles us towards glory?  </p>
<p>God of the gaps begone, i need a gravity,<br>
i am a certain slant of distance from<br>
the center.  </p>
<p>but even on the edge there is still<br>
gold within f major, there is still sunlight<br>
in your eyes, and revolution in the space<br>
where we<br>
collide.</p>

orbitals | Poems by Caitlin Pequignot

May 2017

orbitals

<h1>practicing in my mid-twenties</h1>
<p>to do something repeatedly, hoping for<br>
a different outcome each time,<br>
is either insanity or<br>
practice —<br>
the difference rests in your analysis:<br>
“what could i have done better; what<br>
can i improve?”  </p>
<p>a teacher told me once:<br>
“f-sharp major is the only scale<br>
to practice. it has the most sharps;<br>
it is so awkward; it will bend<br>
all of your other scales into submission.<br>
play it, and only it — you will thank me.”<br>
for a dedicated summer i followed his advice,<br>
but i don’t play f-sharp major anymore; it is<br>
too easy to slide back into<br>
first position ease. g major, a major, these are the<br>
small insanities i allow while thinking about<br>
groceries, boyfriends, time —<br>
they are like breath to a yogi, the shallow end<br>
of the pool.  </p>
<p>but even easy waters seem like oceans<br>
when you’ve nothing to sail towards. the fruit  whispers<br>
within reach — why bother?<br>
the age of annual auditions fades, the stage recedes,<br>
the lights dim;<br>
g major seems like madness now.  </p>
<p>i ask the wood and the strings, “how do i find our<br>
way, now<br>
that the road is gone?”<br>
they answer, “we were born from the earth,<br>
a hundred twenty years before you lay a hand<br>
upon us. you will die, and we will still go on. we<br>
have known many hands other than yours —<br>
with them we have sung<br>
the same notes, yet many songs.”  </p>
<p>i have no song now, but i can make notes,<br>
so my fingers stammer out<br>
g-major like a child<br>
learning to speak.<br>
it is useless; i am spent, and at last, i find the words —<br>
“what can i do better; what can i improve?”<br>
and so i coax f-sharp major out the wood,<br>
straining towards the deep end,<br>
and maybe a song.  </p>

practicing in my mid-twenties | Poems by Caitlin Pequignot

August 2016

practicing in my mid-twenties

<h1>silent TV</h1>
<p>what is the point of silent tv,<br>
frames without sound<br>
pantomiming their points<br>
with near-human gestures<br>
in a fluid array?  </p>
<p>they shout into the void of my eyes,<br>
SLAM SLAM<br>
goes the door in comic-book effect,<br>
it’s color-loud but it’s sound-still<br>
in my ears.  </p>
<p>and i think the problem is me;<br>
i’m an echo unchecked<br>
of fear ringing louder<br>
than voices.  </p>
<p>so if you please, un-mute me gently,<br>
i will be well<br>
one day — take my head<br>
in your hands;<br>
you are softer than<br>
they.  </p>

silent tv | Poems by Caitlin Pequignot

October 2017

silent TV

<h1>styx</h1>
<p>what anime is this video still from,<br>
one where an electric violin is strewn on the floor,<br>
along with a rice cooker, a calculator, a boy —<br>
i can relate, or at least,<br>
i have known them all.  </p>
<p>but how can you know —
do you know my bamboo plant is not real?
nothing good grows in shadow corners, the
dark reactions cannot stem the swing
of death in
isolation.</p>
<p>and how do i decide —
from one moment to the next?
Lord i am stressed, the chaos
is metamorphic in its weight, i am at
the end of understanding, i have
thrown up my hands at the shoreline.</p>
<p>and still, i am wondering -
who will wait with those who cannot cross the river,
those who have heard but cannot decide,
those whose chemistry could not circumvent their
circumstance, those who had no need
to change their mind?</p>
<p>surely i am nothing but
the teeth of the wilderness,
i will lie on yoga mats and drain
myself of water, back
to the places where i am known.</p>

styx | Poems by Caitlin Pequignot

September 2018

styx

<h1>what right have i to grief?</h1>
<p>what am i running away from —<br>
the rain, or the cold night?<br>
behind glass i have shuttered myself,<br>
still,<br>
several reflections away<br>
from alone.  </p>
<p>i have no answers, only anger.<br>
it has anchored me immobile,<br>
i have made myself up in paint and plastic<br>
fibers, into pleasing shapes<br>
that warm in the cold night, yes<br>
the cold, cold night —<br>
and i wonder, what right have i to grief?<br>
it slips in along the ink of my gel pen<br>
strokes, sinks into the page<br>
and meshes with the<br>
atoms, maybe,<br>
i like to<br>
think about it.  </p>
<p>did i tell you i’m afraid of crying now?<br>
black mascara bleeding from my eyelids,<br>
the etchings just spreading from the<br>
cracks in my body.<br>
grief is a river, and i am<br>
sedimentary.  </p>
<p>one day, a dam will burst<br>
much closer to my orbit,<br>
and i’ll careen into the dark<br>
— into the cold, cold night —<br>
and there’ll be nothing left of me<br>
but water.  </p>
<p>maybe you can come over,<br>
if it rains tonight.<br>
i’ll put on some tea,<br>
and we can drink that soft<br>
leaf-water,<br>
drain ourselves of time<br>
and sleep.  </p>

what right have i to grief | Poems by Caitlin Pequignot

January 2021

what right have i to grief?

<h1>xeriscaping</h1>
<p>he tells me,<br>
he says -<br>
“i water your plants sometimes, when you’re out.”<br>
the palm-without-name bends, it drinks;<br>
i say, a derivative of thank you,<br>
but really i am thinking about how<br>
i water them, when i remember.<br>
it’s overdone, with the quickness of guilt<br>
pouring into the soil,<br>
and i try to make up for lost time with<br>
more.  </p>
<p>and i feel it, in the quiet spaces,<br>
i scratch out ten pages of thoughts,<br>
atonement through ink.<br>
when day to day, i am either waterlogged,<br>
or parched, left with a half-life<br>
in a middleground of care.  </p>
<p>i have made ritual out of everything but<br>
life,<br>
i have drowned in the dark reactions.  </p>
<p>and you have listened, in the day to day,<br>
you have bid me drink from a wellspring<br>
i can no longer name.<br>
one day, i will ask of myself,<br>
what you freely give.  </p>