In a rush to incorporate Conversational AI into customer care talkpaths, enterprises increasingly rely on their Software-as-a-Service (SaaS) vendors to provide historical analyses, make actionable recommendations, and ensure compliance with regulations. Technologies to measure and even predict usage are now ubiquitous, but vendors have a long way to go in standing up processes to increase adoption. Solution providers typically assign Customer Success Managers (CSMs) to oversee these activities. While they are often seen as evangelists and problem solvers, their role has necessarily evolved from Advocacy to Forensics.
Back when software was sold as a perpetual license coupled with technical support, CSMs focused on meeting contracted SLAs and attaining overall satisfaction. As the human conduits between licensors and licensees, CSMs were expected to be empathic, passionate, and diplomatic. Their raison d’etre was to embody the voice-of-the-customer in matters technical and commercial alike.
Now that applications, including the ones that provide insights to both live and virtual agents, are sold as subscription or consumption services, CSMs’ updated mission is to drive usage. They are expected to be inquisitive, empirical, and analytical. At a minimum, CSMs need to continuously prove that the value of each application exceeds its costs.
Forensic ROI
In the common pay-for-resources-consumed model, metering usage is inextricably tied to utility of the application itself. At Monthly Business Reviews, CSMs must present extensive historical reports with colorful dashboards and insightful visualizations. They are also expected to propose adjustments that could improve future results. Since contracts are typically short-term with few if any fixed fees or minimum spend requirements, CSMs are in the unenviable position of having to articulate the incremental ROI each month. Their customers want to see demonstrable, impactful results; otherwise, the application could be relegated to the sidelines.
Forensic AI
Many applications include AI, which must be monitored continuously for compliance reasons. At Monthly Business Reviews, CSMs are expected to depict how AI models are adapting to changing conditions. They must provide assurances regarding data privacy, autonomous decisioning, explainability, and bias. If A/B testing is used, CSMs need to explain historical results and suggest ways to differentiate treatments going forward. Enterprise customers want to see the incremental value that AI provides and also know that new technologies are being applied ethically and compliantly.
Quants for Free or for Fee?
Given this new emphasis on forensics, CSMs who support large enterprise customers will need to be trained in Financial Modeling, Predictive Analytics, and Data Science. These “quants” will spend hours packaging historical analyses and recommendations for how to improve results. That’s a lot of talent and a lot of effort, all of which will shape how the enterprise manages the business itself.
The question is: who pays for all of this? Should the forensic responsibilities of the CSM role be embedded within the service offer itself—effectively absorbed by the vendor as a “cost of doing business?” Or should this work be itemized on the monthly invoice to compensate the vendor for customized, forensic services rendered? If chargeable, should it be optional or required? When multiple technology vendors are in the mix, should the work be delegated to a System Integrator, who will ensure consistency and completeness?
Implications for Enterprise Customers
While the onus is on Conversation AI vendors to describe past performance and make recommendations going forward, the enterprise-vendor relationship is symbiotic. Enterprises need to do their part by providing access to data and support from senior management to effect change. CSMs have always played a critical role in overseeing these bilateral activities, but their methods have recently shifted from advocacy on behalf of customers to packaging complex forensic analyses. Regardless of whether they are embedded in the offer or billable as a line-item, forensics will undoubtedly play an important role in driving successful adoption.
About the Author
Michael Sisselman, MBA, M.Sc. is an expert in Customer Experience platforms that delight customers, empower the workforce, and drive business outcomes. He most recently worked at Avaya, where he led a team of data scientists in Conversational AI, Intelligent Automation, and CX Analytics. During the (surreal) summer of 2020, Michael spent time developing methodologies pertaining to Business Value Consulting and Customer Success.
Michael lives in New York and plays classical guitar every day. For more information, click his LinkedIn page.
Categories: Conversational Intelligence, Intelligent Assistants, Articles
Upcoming Webinar with Interactions and Opus Research: Navigating the IVA Landscape 
Insights and Highlights from Enterprise Connect 2024 
Deadline Extended: Nominations for 2024 Conversational AI Awards 
Zoho’s AI Blueprint: Balance Model Size for Maximum Affordability
“Free Services” tend to be undervalued, leading to the nice readouts that go no where. Since the CSM would likely be recommending changes ultimately to improve outcomes, how far will those go if management is not paying for the recommendations? Of course, this is a two-edged sword: What will you find for me if I pay you? How can I know that before I’ve looked? I’ve faced this ROI conundrum before. One approach is to ask how much performance or outcome uplift would make the fee worth it. If it’s a minor bump, that’s better than it needing to be a significant uplift. That might get the for-fee relationship off the ground.
Just some thoughts.
Well you picked a big topic this time Michael and, as usual, your perspective is salient. There is a lot to unpack and rather than digging into the CSM role, maybe I can share a bit about the AI side of things. At #Pryon, we are building intelligent assistants and one market focus area is the call center. AI technology and agent assistants are not a well-defined, organized market today, so customers approach this space in many different ways.
While ROI is an underlying motivator, CSMs and CIOs are intensely curious about what AI can do and how it can change the customer experience. It doesn’t help that there are so many different companies with a cornucopia of claims out there, but no way to compare one to another. This leads to many proofs of concept and bake-offs before even a hint of potential ROI is visible. So even before you get to metrics and forensics, there are some major hurdles to overcome. I’ll share a few lessons learned:
– All AI solutions start with customer data. This is true whether you are building a chatbot, providing cognitive search, automating repetitive processes or predicting customer needs and behavior. If you don’t have data or your data isn’t organized, it is hard to even get started.
– Most AI projects involve analyzing a bunch of data, building a model, training the model, doing experimentation, tuning, etc. It can take many months before something is handling the first use case.
– Chatbots are often dumb and fragile. This is because they are text based IVRs that need to be programmed to understand all permutations of what is said, and walk users down a prescribed path in order to get to any kind of results. They need to be reprogrammed when anything changes, which makes them expensive to maintain. The only difference for users is it takes more effort to type “agent” than it does to hit zero on the phone to get out of the tree 🙂
– Even if you can find the right data, it is often bad. There are missing elements, it is contradictory or even wrong, and much of what is important is locked in the heads of agents and not written down somewhere other than in call transcripts.
– When you get through all of these hurdles, then you are still facing the questions of PII, data sovereignty, AI bias, transparency, etc.
I wish I could say that all of those issues are solved today, but they are not. We have been taking a pragmatic approach to these problems and seeing some success, although it is still in the early days. The underlying principles include:
– Don’t have humans organize the data, just point the AI engine at it and suck it all in. Let the technology organize it and figure it out
– Design the AI technology so that it doesn’t require analysis and training. Automate the process of ingesting the data, creating the model and producing results so this can take place in minutes or hours, rather than months. This also means that as data changes or new data is added, it can be immediately incorporated into the solution
– While there is no automated way to handle the dialog of a chatbot yet, the problem can be partitioned. Script dialog for the few scenarios that represent the majority of customer queries and use AI question answering to access all the long tail information without the need for any scripting
– Let agents score the results being produced from the system. This both provides insights into the quality of the current data and what information is most important to customers. Then enable agents to easily correct, update and add information to the system, and the CSM can track and manage this tacit knowledge without requiring changes to the source material. The state of the engine begins to reflect the reality of the agent experience over time
– Explainability is hard, especially when you are generating new models regularly based on changing data. This is mitigated somewhat by the source of all answers being the company’s own data. We have initially focused on PII and making it easy to redact information when necessary. The high levels of automation in our solution mean that redaction is as simple as changing the source information and pressing a button, or waiting for the next scan to catch the change
The desired outcomes from all of this are that agents can serve customers faster with a higher probability of a successful outcome. It is to simplify chatbot development and make them less brittle to changing information. It is that inexperienced agents can perform on par with experienced agents day 1. We have seen encouraging results but there is a long way to go in the industry and with AI technology. If these outcomes can be achieved, it is pretty easy to measure ROI. Thanks for the discussion.
Thanks for the thoughtful response, Greg. I’m curious whether advancements in Deep Neural Networking and Deep Learning are having an impact. Also, homing in on “explainability”, whether #Pryon is working with approaches to “AI” that address explainability in the face of regulatory audits and such. I’m skeptical about an approach that eliminates humans for organizing the data. “Supervised learning” (with a focus on customer success and, in turn, the customer’s customers success and experience). Would love to learn more.
Here are some of the comments and replies (slightly edited for clarity) that appeared on Michael’s LinkedIn post announcing publication of the article here:
Bernard Gutnick [https://www.linkedin.com/in/bernardgutnick/]
Terrific article and extremely relevant for businesses facing the upcoming unprecedented make or break fourth quarter in retail never seen before. There are so many companies who would benefit from your insight on what investment is needed to survive. It was such a pleasure seeing you present advanced CX concepts to Fortune 100 executives at Avaya. I look forward to hearing both businesses can quickly take action before it’s too late.
Michael Sisselman
Thinking about XMAS already? CSMs should be standing up a monthly cadence of forensic analysis, but you’re correct that an extra push is needed for seasonal planning. And coming off of a sluggish summer, success at the end of 2020 is imperative.
*****
Alison Dahlman [https://www.linkedin.com/in/alisondahlman/]
Great article. In fintech, technology alone isn’t going to cut it. Vendors need to provide continuous analysis of results and advice on how to optimize outcomes. The role of the CSM in all of this is critical.
Michael Sisselman
Forensics are especially important for the FSI vertical, since there are an inordinate number of compliance-related factors.
*****
Venkatesh Krishnaswamy [https://www.linkedin.com/in/venkyk/]
Well said. There is a rather naive buyer expectation that the “AI will continuously improve on its own”. In fact, the AI will only improve if it is monitored and “mentored”. The CSM in the hot seat will need the right tools to be able to help the AI get better.
Michael Sisselman
The CSM is the so-called “human-in-the-middle.” If not the data scientist that proactively tunes the model, then the resource that is tasked with managing that process.
*****
Richard English [https://www.linkedin.com/in/richard-english-84288/]
Balancing act for Conversational AI between its value and how to determine how it’s funded. It’s an important functionality that needs to be engaged and forensics is required to be help determine its value. Success needs to be measured.
Michael Sisselman
Or as Peter Drucker famously said: what gets measured gets improved.
*****
Jennifer Doyle [https://www.linkedin.com/in/jennifer-doyle-04b220a/]
Great article! As the CSM role evolves, I wonder who is benefiting – the customer from the forensic analysis and recommendations or the company for the incremental consumption that’s generated. Has to be both.
Michael Sisselman
When both the vendor and the customer are vested in the hard work that is needed, there is at least the potential for synergistic win-win. Needless to say, that’s easier said than done.
*****
Andrew Maher [https://www.linkedin.com/in/andrewfmaher/]
You raise some good questions here. As others have already noted, AI on its own won’t get you the ROI improvements but a person, or CSM is needed. However, you also state that “support from senior management to effect change” is also needed. This is where I’ve experienced the most challenges. Many times, the real improvements are not made with software but by changing the structure of the business. Together this could bring perpetual improvements.
Michael Sisselman
Forensic governance work best when companies: 1) have a written CHARTER; 2) dedicate full-time RESOURCES; and 3) assign an EXECUTIVE SPONSOR to lead the way.
*****
Peter Finney [https://www.linkedin.com/in/peterfinney1/]
Great article. Data Science starts with the right questions and in terms of ROI businesses need to align to a top down business dashboard, their MI/KPIs. Then a CSM can work with the wider data analysis teams to ensure that they are forensically analyzing the right data sets aligned to targeted MIs and individual stakeholder business outcomes. Whilst also identifying unknown, trends and tactical areas for continuous business improvement and customer excellence.
Michael Sisselman
Forensics that are top-down ensure congruence with broader business objectives. Bottom-up forensics, on the other hand, drive discovery of heretofore unknown phenomena. Needless to say, a combination of both approaches is usually best.
*****
Roy Schijins [https://www.linkedin.com/in/schijns/]
Fabulous article. I know of only a handful of people in the industry who can articulate this shift so clearly, expertly and eloquently. Your thought leadership in this area is much appreciated and valued. Keep writing, you’re great at it!!
Michael Sisselman
Couldn’t have done this without guidance and thought-leadership from Dan Miller (Opus Research). Rebecca Sisselman helped with edits…and more edits. That said, insightful comments from you and others make it all worthwhile, and are much appreciated.
*****
Robin Foster [https://www.linkedin.com/in/robinfosterroi/]
“Free” creates its own problems — Clients often undervalue what’s “free” and don’t commit to taking free advice with the same attention and rigor that they’d muster “for fee” advice.
In the specific case of a CSM, evaluating the operation and outlining points for improvement, the added issue is “how valuable will the CSM’s findings be?” That’s an ROI conundrum I’ve faced many times. What will be discovered, what would be the impact on business outcomes of making the change, and, how much would the change cost?
One option to break out of the stalemate about a CSM’s fees would be to ask how much uplift in business outcomes would be needed to cover the cost of the CSM plus some reasonable $ to implement the changes? Reverse engineer the problem and if the required uplift is small, the fees are “worth it” much more than if the uplift is large. But of course, that’s not to say a large uplift is unlikely — in the age of analytics and AI, we can all be panning for gold.
Michael Sisselman
Beyond “what gets measured gets improved,” you may be correct that “what gets purchased is deemed valuable.’” Meanwhile, the question of whether fees should be based on T&M or a percentage of improvements attained is extremely interesting. Perhaps that’s a topic for another article!
*****
Greg Pelton [https://www.linkedin.com/in/greg-pelton-631789/]
You have hit on a subject that is very topical in the industry and, for lack of a better word, thorny. LinkedIn is too limiting for detailed response so I commented on the original article at the Opus Research site, going into some details on the experiences we are having at #Pryon. Looking forward to your next post!
Michael Sisselman
Many thanks for writing extensive comments on the Opus site about ways to overcome numerous pitfalls in delivering Conversational AI solutions into production. No doubt this raises the bar significantly in what CSMs will need to orchestrate (no pun intended).
*****
Asli Uysal [https://www.linkedin.com/in/asli-uysal-929389a6/]
A great read. Thank you for sharing with us!
Michael Sisselman
We can learn from #genesyscloud a pioneer in Contact Center AI.
*****
Winsome Lee [https://www.linkedin.com/in/winsomelee/]
Thanks for the great insight, and how fantastic that you play the classical guitar daily!
Michael Sisselman
Finally, somebody noticed that what really makes me happy is playing the classical guitar every day (since 2012). In fact, check out the background graphics in my Linked In profile.
Some folks have commented offline that the artwork is reminiscent of Abbey Road tunnel on the Beatle album cover. But it’s meant to be an abstract rendering of my Jose Ramirez (Anos 130, cedar) that I bought from the luthier in Madrid several years ago.
I’m told that interests in AI and Classical Guitar are highly-correlated. What instrument do you play?
Winsome Lee
AI + Classical Guitar is an awesome combo! I play the piano.
*****
Tony Shrader [https://www.linkedin.com/in/tonyshrader/]
Is it the CSM role? or Market Opportunity? I believe the vendors’ CSMs will struggle to satisfy that large of a job description. That partnership can create stickiness for the vendor while extending the ROI for the customer. Pick on the #speechanalytics industry for a moment, rarely can business hire, deploy, train people and the model, build the searches, and have ROI in less than 12 -18mo. (DMG, ContactBabel) . I argue a CSM couldn’t speed that metric up. Although they could improve the client’s learning curve, they still must learn the customer’s business, while managing 4 other clients. There is an opportunity to layer capability with a partner to provide a quicker ramp for ROI, but also value-added services that can go beyond traditional speech analysis. So short answer is “Pay.”
Michael Sisselman
Emphatically agree that CSMs are necessary, but not sufficient. On-going governance is a team effort, and a RACI table should be created to assign roles and responsibilities holistically. Breakeven within 12 months is possible only if a concerted effort is made from the outset.
*****
Elaina Gemelas [https://www.linkedin.com/in/elainagemelas/]
Good stuff, and way to go on sharing your thought leadership in this space. It’s also nice to see the more human side of people, and sharing your classical guitar skills with the world is of great value!
Michael Sisselman
Understood: Bots don’t play classical guitar; humans do.
*****
Jacob Martin [https://www.linkedin.com/in/jacob-martin-8352402b/]
Great article and fascinating to look forward at AI sectors that are more mature relative to healthcare.
Michael Sisselman
In addition to data science, you remind me that CSMs need to have vertical expertise as well. It’ll be interesting to see how AI evolves in healthcare, and the extent to which it impacts evidence-based-medicine.
*****
Andrew Maher [https://www.linkedin.com/in/andrewfmaher/]
You have learned how this platform works. Keep doing what you do.
Michael Sisselman
Sensei, Heeding your advice, I developed a:
1. Bot to write articles
2. Bot to edit articles
3. Bot to publish articles
4. Bot to promote articles
5. Bot to respond to comments
Best regards, Grasshopper
*****
Steve Forcum [https://www.linkedin.com/in/steveavaya/]
Great thought-provoking article. First, I agree, the CSM role is crucial in delivering ongoing results through the use of AI. There’s a lot of curation necessary to ensure data integrity. Second, there’s no free lunch in this world…if you’re letting the vendor handle this, then hard questions need to be asked about output/data ownership, privacy, etc.
Michael Sisselman
In effect, the CSM is leading a team of resources who are providing PROFESSIONAL services. One way or another the vendor needs to pay. But you raise a very thorny question: Is this a work-for-hire, in which case the enterprise owns the work-product? Or does the vendor own derivative IP that is created in the process? Lawyers on the discussion thread…. this one’s for you.
*****
Will Nihan [https://www.linkedin.com/in/willnihan/]
Great article! I think it is too important to outsource to a vendor and from a governance point of view, it’s not sustainable. You could see a whole new set of roles and products emerging to handle this.
Michael Sisselman
Reminiscent of Newton’s Third Law: Every CSM that works for a VENDOR requires a COUNTERPART that works for the ENTERPRISE. The question is who will fill this role, and what skills in data science are needed to oversee the work product.
*****
Noland Bradshaw, PhD. [https://www.linkedin.com/in/nolandbradshawphdmphil/]
Insightful article. I agree with you that regardless of whether they are embedded in the offer or billable as a line-item, forensics will undoubtedly play an important role in driving successful adoption.
In my opinion, the client/buyer should pay for “… [the] … customized, forensic services rendered.” They can then measure their ROI, and explicitly hold the vendor accountable for the performance of their product.
Michael Sisselman
While precise measurement of “investment” (denominator) is possible, the vendor will struggle to measure “return” (numerator). Hence the onus of calculating ROI implicitly falls on the enterprise. But you raise an important point: minimally the CSM should be required to manage PERFORMANCE of the service itself—with FORENSICS.]
*****