Conversational AI: Flow Modeling
Portfolio | EY | IBM Watson (Hybrid) Chatbot
Challenge
EY employs, on average, one person every eight minutes. The purpose of the IBM Watson chatbot aimed to change and improve the way Millennials interacted with and experienced HR.
Prior to my arrival, EY had partnered with IBM and used Design Thinking to build out a successful proof-of-concept. I joined the effort for Phase II which was to broaden the corpus of available knowledge base material and improve user intent matching.
At the time, Watson functioned as a hybrid AI chatbot, combining rule-based logic with machine learning to address user questions and needs. Upon my arrival, the project faced key challenges, including a lengthy and inefficient stakeholder review process and frequent mismatches between user intent and system responses.
Additionally, the front-end chatbot experience was powered by Microsoft Teams UI and required an EY branded skin.
Supporting material
Article: How EY’s journey created a major business impact.
Example: Intent mismatches leading to low NPS scores
Responsibilities
➤ Liason between internal EY stakeholders and IBM
I served as the liaison between internal EY business stakeholders and IBM. My direct role was to facilitate the review and approval process for the user intents we were targeting in Phase II.
➤ Skin up a branded front-end chatbot UI using Microsoft Team’s kit
While the backend was powered by IBM Watson, the chatbot’s front end was designed for deployment within Microsoft Teams. I was responsible for applying EY’s branded design system to create a customized, cohesive EY chatbot experience.
Findings
➤ Deviation from conversational flow modeling best practices was causing approval and production delays and stakeholder frustrations
Phase II had already started prior to my arrival. IBM’s conversational flow diagrams lacked best practices and was overloaded with technical jargon. The flows were visually disorganized and difficult to interpret, demanding excessive time from stakeholders for review and approval. A bottleneck had been established and needed triaging.
➤ Customizing the Front-End UI for EY’s Watson-Powered Chatbot
The Phase I effort leveraged Microsoft’s Teams UI as they were out of the box. The Phase II effort would require branding the user interface controls and content components that the chatbot experience would rely on. Each component part would require inspection and branding.
➤ Intent matching didn’t accurately reflect EY’s information architecture
Traveling EY consultants needed the ability to manage up to 5 different addresses with their EY profile. The starting point of the “Change Address” conversational flow failed to reflect this architecture by first educating the user on the five options available and then clarifying the proper address to update. This simple dialog failed over 86% of the time during initial user testing.
➤ Not separating the dialog from the technical and content annotation layers created a heavy cognitive load for stakeholders during review cycles
Stakeholders are typically business oriented. Putting complicated flow diagrams in front of them can cause frustration and anxiety especially when they have multiple flows to review at the same time. We use swim lanes to separate different types of annotated context. The flows I was reviewing overlaid technical, narrative and logic into one swim lane.
➤ Some flow endpoints appear to leave the user in the lurch
Users are led to different conversational endpoints based on their needs. When reviewing the flow diagrams, stakeholders had a difficult time understanding what users would be left to do. The goal was to connect the user to a live agent when the chatbot was unable to assist but this was not always clear in the diagrams leading to additional rounds of edits, and meetings.
Approach
My first task was to streamline the partner-proposed flow diagrams using best practices—significantly reducing stakeholder review time, minimizing approval bottlenecks, and improving user utterance and intent matching accuracy.
Review resource
Identifying best practices
➤ Best practice sources
I scoured the IBM Carbon website on chatbot architecture and came across a recommended flow model (displayed below). I combined these partner-published best practices with the Jesse James Garrett Visual Vocabulary to re-draw the diagrams with more clarity.
Implementing best practices
➤ Redrawing the partner-provided flow diagrams using best practices
Some of the artifact improvements I brought to the table included switching the layout of the diagrams to a vertical stack (vs. horizontal), introducing swim lanes (for visual clarity) and including data research and metrics regarding user utterances to guide, inform and improve the “can the bot answer” loop.
| Swim Lanes | Research | Dialog | Supporting Flows | Technical Callouts | Content (In Chat) | Content (External) |
|---|---|---|---|---|---|---|
| This swim lane contained quantitative analytics and user verbatim utterances for targeted topics. | This swim lane provided the user/bot narrative dialog. This column was the chat experience. | This swim lane was used to call out and annotate supporting flows (e.g. "connect me to an agent"). | This swim lane offered technical annotations for our engineering partners. | This swim lane offered UI views like Adaptive Cards that were scheduled to display within the chat window. | This swim lane was reserved for declaring external content (content the user should interact with to complete their needs that occurred outside the chat window). |
