4 ways to spot #fakeAI

You know the five love languages? Mine happens to be “Acts of Service,” and because tomorrow is Valentine’s Day 💜, I’m offering the following Act of Service to all of you: a list of the top ways you can spot #fakeAI when it comes to artificial intelligence (AI) in telco software.

Mobile World Congress (MWC) is just around the corner, and everyone will be talking about AI. But talk is cheap, and many vendors will just be riding the marketing wave. I want to help you figure out which are hawking #fakeAI so you can focus on the vendors selling the real deal.

To help, I created this quick primer to help you evaluate all the AI claims you’ll hear while you’re running around at the Fira.

1. How to evaluate a co-pilot

Want to juice up your product with AI? Then let me introduce you to the world of copilots. Copilots offer a straightforward method to integrate AI into software products, leveraging large language models (LLMs) for decision support, problem-solving, and content generation. They are particularly effective in systems rich in historical data, such as billing, CRM, and everyone’s favorite AI use case: customer support. 

The utility of a copilot can be enhanced when paired with a Retrieval-Augmented Generation (RAG) framework, merging information retrieval with natural language generation. This combination minimizes inaccuracies (AKA “hallucinations”), enriches content with current data beyond initial training sets, and cites sources to help validate responses.

Use these questions to learn more about a copilot:

• Which LLM was the copilot trained on? Can you change out LLMs?
• What additional information did you use to train the copilot?
• How do you make sure your copilot stays true to provided materials?
• How is the raw data cleaned and prepped for ingestion by the copilot?
• How do you charge for the API calls to the LLM?
• How do you protect my proprietary data?
• How does the copilot’s output help to increase the productivity of the user?
• What other systems can the responses be integrated into?

Once you get through your interview, place the copilot into one of these three categories:

• Basic: Offers decision-related information without significantly boosting productivity.
• Intermediate: Provides contextually relevant suggestions, enhancing productivity but still requiring human validation and intervention.
• Advanced: Fully automates tasks, including complex problem-solving and problem resolution, often outperforming the average human completing the same task.

An example of advanced copilot implementation is Totogi’s Plan Sidekick, which utilizes a RAG system for comprehensive plan design, generating legal documents, marketing materials, and sales emails, showcasing the potential of AI to enhance operational efficiency and creativity for a marketing team.

2. Validating predictive analytics

Another way vendors add AI to their products is by adding predictive analytics. Predictive analytics employs a variety of techniques and technologies, including statistical analysis, modeling, data mining, and machine learning (ML) to forecast future events based on current and historical data. It identifies patterns and trends to aid organizations in making informed decisions, optimizing operations, and tailoring strategies.

The field is particularly distinguished by advanced systems capable of real-time analysis. These use sophisticated models, such as deep learning and neural networks, to identify complex patterns in extensive and diverse datasets. Their proficiency in processing large volumes of data swiftly and their scalable nature allow for immediate, actionable insights. These advanced systems are adaptable, continuously learning from new data without needing manual updates. They provide highly accurate predictions due to their sophisticated algorithms. Their integration, automation, and customization capabilities make them invaluable across various industries.

A prime example is real-time fraud detection through deep learning, analyzing vast transactional data against historical fraud patterns and dynamically improving prediction accuracy. This reduces fraud, builds customer trust, and increases efficiency, demonstrating predictive analytics’ potential to evolve and combat financial fraud effectively.

Use these questions to learn more about a vendor’s predictive analytics capability:

• How is data ingested for use, and how long does it take to prepare it?
• How are your models trained, and how often are they updated with new data?
• What specific AI and ML technologies are utilized in your solution?
• How does your solution integrate with our existing data infrastructure?
• How do you ensure the quality and integrity of data used in your predictive models?
• What measures do you take to ensure the privacy and security of data?
• How customizable is your solution to fit specific business needs and objectives?
• What support and training do you offer to ensure successful implementation and maximization of the solution’s value?
• How do you measure the accuracy and performance of your predictive models, and what benchmarks do you use?
• Can you share case studies or examples where your solution has delivered measurable business outcomes?

Once you get through your interview, place the predictive analytic feature into one of these three categories:

• Basic: Uses simple statistical models for straightforward predictions based on historical data, focusing on linear relationships. Output is provided in the form of a summary report.
• Intermediate: Incorporates complexity with multiple data sources and variables, using techniques to identify both linear and non-linear patterns. Output is usually not real-time.
• Advanced: Employs complex algorithms for real-time analysis of vast datasets, capable of identifying intricate patterns, adapting to new data, and providing highly accurate, actionable predictions.

An example of this advanced application is Totogi’s real-time churn prediction ML model, which leverages charging data to predict customer churn, continuously refining its accuracy and offering real-time insights through an API. This exemplifies the advanced capabilities of predictive analytics in providing dynamic, data-driven churn prediction.

3. Prove it!

Any vendor with legit AI tools should be shouting about how it increases productivity, decreases costs, or increases revenue. While they may not have exact figures, they should be able to estimate metrics for you. Do the tools eliminate steps or handoffs? For example, at my company, AI tools are now handling all level-one customer support queries. We’ve been able to promote our level-one support people to higher-paying jobs doing things only humans can do. 

Ask if the tool has been deployed to customers, and seek out reviews and testimonials from current customers. Look for evaluations or assessments of the AI technology by independent third parties, such as research firms, industry analysts, or academic institutions. These sources can provide unbiased views on the technology’s effectiveness and innovation. While it’s early days on AI features and many vendors will be light on these external proof points, they should be coming in the next year, so be on the lookout for them.

Categorize the vendor’s metrics using the following guidelines:

• Basic: The tool improves performance. If the model is real, the vendor should have some basic documentation, benchmark results, and (favorable!) comparisons with other models. The metrics may include processing speed, compute or memory usage, cost savings, ROI, F-score, etc.
• Intermediate: There are real customers using it (and they like it). At this level, the vendor should have metrics from actual customers using the solution in production. Additional metrics would include relevant business KPIs, such as ticket resolution stats, customer satisfaction scores, customer acquisition or retention, revenue growth, adoption rate, etc.
• Advanced: The vendor is already talking about next-gen improvements. If your vendor has advanced-level AI chops, then company reps should be able to talk about all of the above metrics off the cuff—and even tell you about performance metrics on raw data from new customers, how to avoid built-in bias, compliance, security, etc.

At Totogi, our F1 score on our revenue optimizer is at 87% as we continue to drive towards 90%. At this point, it can predict customer churn six weeks in advance. That took a lot of work—like two years and millions invested in Google-trained data scientists. This kind of performance isn’t a press release throwaway line. It’s hard-ass shit.

Your vendor should have war stories about how hard it’s been to get it working. There is a lot of trial and error with new technology like this. We had false starts, internal throwdown fights about how to get things working, tried a bunch of third-party tools before we found the right one, delays while we cleaned our data, and further delays because we made an incorrect assumption and had to wait to accumulate more data and try again. It takes time. We shaved what felt like 1,000 yaks. We cleaned our data some more. Ask me, and I’ll be happy to whine to you about it.

4. Seeing is believing

With MWC coming up in just a couple of weeks, it’s the perfect time to explore firsthand the capabilities of various AI technologies. To really check out vendor’s AI capabilities, ask for a demonstration! Put them on the spot and inquire whether it is a pre-packaged or live demo. Bonus points to the vendor if it lets you bang on the demo yourself. The ability to interact directly with the technology, if permitted, can provide valuable hands-on experience; any reluctance to offer this might suggest the AI is not yet fully refined or produces variable results. If there’s not a demo available, ask if a pilot project or trial period can be done after MWC. This lets you see firsthand the benefits and potential challenges that might not surface through an in-booth demo. 

For vendors that are making progress with AI and claiming customers are using it, ask if any case studies are available. This can give you insight into the implementation process, challenges faced, and outcomes, focusing on metrics and data that support the touted benefits. Additionally, having discussions with existing customers about their experiences, from implementation to ongoing support and the realization of benefits, can offer critical insights.

Evaluating AI offerings can be structured into three categories:

• Basic: Exhibits through a canned demo. This is a prepackaged demonstration available at their booth, providing a superficial look at capabilities.
• Intermediate: Has a live demo that accepts your input for a personalized test, or a willingness to engage in a trial or a pilot project, allowing you to evaluate the technology directly.
• Advanced: Provides case studies and customer references, presenting detailed insights into real-world applications, challenges, and the value delivered, demonstrating the AI’s effectiveness and reliability.

Want to learn more about what REAL AI looks like in telco software? Come visit the Totogi booth in Hall 2, Booth 2B72, at MWC. We’ll be giving LIVE demonstrations of all our AI capabilities and letting attendees test them out for themselves. I also invite you to my talk at the MVNO Summit at MWC on Wednesday, February 28, 4 – 6 pm CET, in Hall 8, Theater 4. I’ll be talking about how the Totogi team is using AI to reinvent BSS. In fact, my talk will be the official launch of Totogi’s latest offering. See you there!

Recent Posts

Get my FREE insider newsletter, delivered every two weeks, with curated content to help telco execs across the globe move to the public cloud.

"*" indicates required fields

Email*

Comments

This field is for validation purposes and should be left unchanged.