The analyst’s day is full of research. Now, this is the age of AI and AI is here to help, isn’t it? As everyone is talking about copilots and AI agents, why not using the tools at hand to do a little research on research.

NB., no one really has a good definition of an AI agent, so this might become an additional topic for research.

But I digress.

Imagine the following project at hand, which is not only interesting for analysts, btw, but also for a variety of roles in the corporate world. Let’s call it vendor (competitor) monitoring. The job is the following:

• Research reputable sites for news about a number of vendors, relating to a set of keywords. Reputable sites are high quality news sites, high quality tech publications, high quality analyst sites and, of course the news pages of the vendors in question.
• Limit the time frame of the search matching to the cadence of my information requirement, e.g., “yesterday” for a daily update or “last week” for a weekly update
• Provide a summary of the news
• Give an assessment of how the news affects the positions of the vendors in the marketplace re the key words in question
• Provide these news with their assessments as a prioritized list, sorted from high impact to low impact
• Add an executive summary as a preface
• Send it to me as an email

So far, so simple. After all, a lot of folks, yours truly included, do this every day. And it is taking quite some time. So, this job is a perfect one for an automated update beyond a CSS feed. And it seems like a perfect job for an LLM turned agent – or is it a copilot?

Now, the basic question is: Which one to use? After all, there are plenty, from free to not so free ones. Answering this question turns into yet another interesting experiment: Why not ask some LLMs for their evaluation of suitability? Kinda meta, but an interesting one.

So, I did just that: I asked ChatGPT 4.5, Grok 3, and Gemini in its 4 versions 2.0 Flash Thinking Experimental, 2.0 Flash, 1.5 Pro with Deep Research, 2.0 Pro Experimental for their analysis of which of them is best suited for the research task at hand.

For this, I used the following, simple prompt:

describe the different capabilities and limitations of Gemini 2.0 Flash Thinnking Experimental, 2.0 Flash, 1.5 pro with deep research, 2.0 pro experimental, Grok 3 and chatGPT 4.5, both with and without deep reasoning. Which model is best to support the following use case:   

research the web for news on a given set of companies and a given set of topics. The news shall cover the past 2 days only   

assess the news regarding their impact on the companies' market positions re the given set of topics   

create this in the form of a report   

do this as a daily scheduled task  

accuracy, reasoning and reliability are of high importance. Speed is of lower importance.  

generate a comparison table, give a recommendation and justify the recommendation.

The results are quite interesting.

• Gemini 1.5 Pro with Deep Research resolves to Gemini 2.0 Flash because it offers “a balance of reasoning, accuracy, reliability, and tool use necessary for fulfilling the requirements of the specified use case. Its production-ready status, combined with its ability to handle complex analysis and generate comprehensive reports, makes it the ideal LLM for this task.” It looks at Grok as the runner up.
• Gemini 2.0 Pro Experimental recommends Gemini 1.5 Pro with Deep Research as it “offers the best balance of accuracy, reasoning, and reliability for your demanding research and reporting task, leveraging Google's strengths in both LLMs and web search.”. This is closely followed by Grok 3.
• Gemini 2.0 Flash Thinking Experimental suggests doing a competitive evaluation of ChatGPT 4.5 with Deep Reasoning and Gemini 2.0 Pro Experimental – as they are a tie. “Which is truly "better" in practice will depend on your specific prompts, data, and tolerance for latency and cost”.
• ChatGPT 4.5 (without Deep Research) suggests going for ChatGPT 4.5 with Deep Research as the model “offers enhanced reasoning capabilities, reduced hallucination rates, and a broader knowledge base, aligning well with the requirements for accuracy, reasoning, and reliability in daily scheduled tasks. While models like Gemini 2.0 Flash Thinking Experimental and Grok 3 also provide advanced features, ChatGPT 4.5's maturity and proven track record make it a suitable choice for generating comprehensive and reliable reports.​”.
• Grok 3 in Deep Research mode suggests using Gemini 2.0 Pro Experimental as its “advanced reasoning capabilities, as evidenced by its performance in complex tasks and 2 million token context window, make it ideal for researching news, assessing market impact, and generating daily reports (Gemini 2.0 Pro). The integration with Google Search ensures access to recent news, and as a Google product, it likely offers high reliability for scheduled tasks, aligning with the emphasis on accuracy and reasoning over speed. While Gemini 1.5 Pro with Deep Research is tailored for research, Gemini 2.0 Pro Experimental, being a newer model, likely offers superior capabilities”. Grok looks at Grok as the runner up is it offers advanced reasoning and Deep Search “but potential biases from X data integration”.

So, what does this tell me?

There is probably a bit of self-serving involved in the LLM’s assessments and suggestions. At least Google consistently suggests a Google model and ChatGPT suggests itself. What is a bit confusing is that

An interesting side remark is that only Gemini 1.5 Pro with Deep Research, ChatGPT 4.5 and Grok 3 provide the sources used for the research. Perplexity does this, too. Providing references is important for validating the results.

It looks like the results delivered by these LLMs seem to favor Gemini 2.0 Pro Experimental and ChatGPT 4.5, though, although I am impressed by Grok 3. On the other hand, one needs to know that “experimental” means exactly that – the models are not yet fully stable.

Having said this, if one needs to perform research tasks, as many of us need to, environment matters. Especially smaller businesses often run Google Workspace. In the case that they subscribed to the Business Standard Edition (like I am doing), Gemini is readily available, there is probably no immediate need to purchase an additional ChatGPT license (I have a pro subscription) or a Grok or Perplexity subscription. This is especially true as most of these tools use a lot of the data that users provide to improve their services, which is especially true for free services. Grok, in its privacy statement explicitly recommends to not input any personal data – as it will be used.

In summary, if and when I need to do research, I’ll use Google Gemini 1.5 Pro with Deep Research and Gemini 2 Pro Experimental as my preferred option, simply because ChatGPT 4.5 with Deep Research only offers limited runs per month. As it doesn’t cost much, additionally running the same research – potentially with a slightly changed prompt to cater for model differences – I will use ChatGPT and (if no sensitive data involved) Grok 3 in addition. Worst case, this gives me additional food for thought.

What do you think?

The news

During the Salesforce AI Day on June 12 as well as the Salesforce AI Industry Analyst Forum on June 20, Salesforce provided a lot of interesting information on how the company addresses the challenge – or should I say problem – of trust into artificial intelligence. Salesforce sees this gap caused by hallucinations, lack of context and data security as well as toxicity and bias. According to Salesforce, this gets compounded by the need for integrating external models into business software.

To address this problem, Salesforce has announced its AI Cloud that combines an “Einstein GPT Trust Layer”, Customer 360 and its CRM to offer AI-powered business processes that are built right into the system, based on an AI that can be trusted. The main vehicle is the Einstein GPT Trust Layer that takes care of

• secure data retrieval from business applications,

• dynamic grounding to reduce the risk of hallucinations and to increase response accuracy by automatically enriching prompts with relevant business-owned data,

• data masking, the anonymization of sensitive data to avoid its unintentional exposure of sensitive data to external tools,

• toxicity detection to make sure that generated content adheres to corporate policy, is free of unwanted words or images, and unbiased,

• creating and maintaining an audit trail,

• the external (or internal) AI not retaining, storing, any corporate information that gets sent to it via the request.

This trust layer sits in between the used AI models and the apps and the respective development environments. All requests to the models, along with their data, get routed through this layer, ensuring authorization protected retrieval of data, the grounding of prompts using it as well as data masking for anonymization. Responses by the models get routed through it as well. This enables an audit trail as well as toxicity detection. Models can be ones within Salesforce, ones developed and deployed by the customers in their infrastructure and third party models.

Figure 1 The Salesforce AI Cloud Architecture; source Salesforce

To round this off, Einstein Studio allows the building and deployment of models, their training using data within Salesforce and, at a later stage, the building of own models using a no-code environment.

The bigger picture

Although AI is not new, it is safe to say that generative AI is a game changer. OpenAI managed to get AI out of the realm of data scientists and into the hands of mere mortals. And most of us use business applications on a daily basis.

One of the most daunting problems of the use of AI is that there are a number of considerable risks involved with its usage. The one that is currently talked about most in the context of generative AI is the one of accuracy of responses to prompts, which is often referred to as hallucinations. This is not only problematic in consumer usage but even more so in business usage.

What comes on top in a business context is very much related to data privacy bias and profanity. Both have also been discussed in the consumer arena. Do you remember Microsoft’s infamous Tay bot? Or more recently of Samsung, Amazon, Apple, and other companies ordering their staff to not use ChatGPT et al.?

The management of all of these risks is of paramount importance to businesses, for regulatory reasons as well as for the need of protecting own intellectual property. No business can afford customer data and/or sensitive corporate data leak into external tools. This is doubly true in strongly regulated industries. But how to ensure this, when the models are not fully understood and when it is not even clear where and how data is stored? How to adhere in a GDPR-request to delete a customer’s data in this case? The management of these risks requires organizational, educational, and cultural measures in companies. These need to be supported or enforced with the help of technology. 

The obvious technical resolution for this is an AI security layer that I outline in my (upcoming, as of this writing) column article on CustomerThink as follows.

Figure 2 An AI security layer; source Thomas Wieberneit

This is of course simplified, not exactly trivial, but possible.

My analysis and point of view

One could, or rather should, say that trust and security are two of the most important assets in business. Customers need to trust that businesses do not collect an inordinate amount of data and that they furthermore use the data given by customers only for consented to purposes. In addition, they need to trust businesses that they keep their data safe. A multitude of regulations mandates this. Being trustworthy is even more important in times of AI as a service, when businesses cannot even tell anymore where customer data is stored, as it is learned by the AI and stored in a very decentralized manner – as part of an unknown number of parameters.

To enable this trustworthiness, what lies closer for a tier one platform vendor than ingraining an AI security layer directly into the own platform? The gateway to external services is already provided by the platform and can be reused by the AI security layer.

This is what Salesforce has done in an exemplary manner with the aptly named Einstein GPT Trust Layer. Kudos for this.

Figure 3 - How the Einstein GPT Trust Layer works; source Salesforce

In my opinion, the most interesting part is the zero-retention portion. Salesforce cannot guarantee on its own that external providers do not store any data. Whenever a prompt is sent to an external vendor, this data is leaving Salesforce’s systems boundaries. This means that external vendors assume temporary control of this data to provide their services. Masked or not, this data that can potentially be demasked, is handled by them.

To accommodate for this, Salesforce has established “zero-retention policies” with these vendors. According to information given during an analyst briefing, these policies ensure that the vendors won’t store any in-flight data, including inputs and outputs, nor won’t they use it for any purposes besides generating a response to the prompt.

This is quite an important statement that also indicates GDPR compliance, if “policy” can be translated to contract. On the other hand, this makes me curious how the refinement of prompts works in this case. Obviously, for highly security-oriented customers, this also suggests the preference of Salesforce or customer-owned models over external ones.

Overall, this is a great offering that addresses important concerns  of the C-suite.

The only qualm that I have is the price tag, which is quite steep, starting at currently $360,000 US. For sure, customers can derive good value out of it, this is not the problem. Where I see a challenge is that this is out of reach for most SMBs. I’d love to see an adaptation of this offering combined with offerings like Salesforce Easyor similar.

I wait to see when other vendors come forward with a comparable offering. Especially the other tier one but also the tier two vendors need to make a move now. Salesforce truly let the Genie out of the bottle and put them in a tight spot.

Kudos again!

Generative AI is here to stay.

It is not only a hype that probably gets worse before it gets better. And we clearly still are in a hype, as the following chart showing the search interest for ChatGPT between October 1, 2022 and April 12, 2023 from Google Trends shows.

Similarly, the Gartner Group sees Generative AI technology approaching the peak of inflated expectations in its 2022 hype cycle for artificial intelligence.

To be sure, we see only the tip of the iceberg when looking at voice, text or image based services that we all know and use. The Gartner Group also foresees many industrial use cases reaching from drug and chip design to the design of parts to overall solutions. 

You think that these scenarios lie far in the future? Read this Nature article from 2021 and think again.

And in contrast to some of the other hypes that we have seen in the past few years, there are actual use cases that support the technology’s survival of the trough of disillusionment. As there are viable use cases, unlike “Metaverse”, Blockchain or NFTs have shown, generative AI is not a solution in search of its problem.

Apart from OpenAI’s GPT and Dalle-E models that surely caught everybody’s attention in the past weeks and months, there are a good number of large language models that are just less known. A brief research that I recently conducted, unearthed more than 50 models that got published over the past few years. For their paper A Survey of Large Language Models that focuses on “review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques”, a group of AI researchers identified a similarly impressive number of large language models that got developed in the past few years.

Fig. 1: A timeline of large language models; source: A Survey of Large Language Models

This increased competition, along with research into how the resource consumption of large models can be reduced, e.g. through sparsification, will lead to improved pricing.

Consequently, businesses need to find a way to leverage this technology without harassing their own data security and/or exposing their intellectual property (IP).

That this is easily possible, has been already learned the hard way by companies as diverse as Amazon or Samsung, to name just two better known cases. And then, there are ongoing use cases around the unauthorized use of IP-protected data for training and even many questions around the ownership of generated (derivative?) work products of generative AI, not even speaking in unwanted bias. 

The way to be walked involves three major steps:

Education

It is necessary to educate the teams not only on the benefits of using generative AI but also on how to check the output and, importantly, when to not use generative AI. Every employee has a role in keeping the business protected from avoidable mistakes. And no employee wants to make one of those mistakes. Sources that are used for any content that is generated should to be attributed and/or verifiably not be copyrighted. Create some sensitivity about “when in doubt, don’t use it”. Enable your people. For the time being, this training should be frequently reviewed, as the whole area is evolving fast.

Second, it is important to enable personnel to fine tune and then train large language models. This is not necessarily a trivial task that requires some knowledge about machine learning.

Governance

As with most technologies, there is a need for governance to augment the training. Generative AI is a very powerful tool, so it needs to be used responsibly and ethically. To help the employees with their own decision making about when and how it is permissible to use generative AI services in the course of their work, and according to the corporate values, it is necessary to develop, publish and implement a policy that outlines and explains the relevant guidelines to follow.

Points that should be covered by the policy include;

• Who is allowed to use generative AI services. Depending on numerous factors like corporate culture, sensitivity of information dealt with, etc. it may be useful to explicitly limit the use of generative AI.
• Proprietary or confidential information must not be revealed or disclosed.
• Similar to other policies (e.g. use of the Internet in general), there may be a limitation to business use or the permission to also use it for private purposes.
• The services should not be used for tasks that are violating any law, regulation, or the company’s code of conduct. This involves discrimination, offensive or hate speech that should be prohibited. This also includes content that is intended to deceive or defraud others.
• Whatever the generative AI produces needs to be reviewed. 
• The use of generative AI to create a work product should be disclosed in order to maintain transparency.

The policy should also include a reference to the offered training that personnel should undertake. Last, but not least, and as bad as it sounds, the consequences of violating the policy must be made clear as well. Here, too, I would concentrate on explanation and rationalizing instead of just wielding a stick.

It might be useful to review this policy frequently, as we are still in a learning period.

Following my own suggestion here: I have used You.com and Google Bard to give me some points that the policy should cover.

Execution

In parallel to this effort, it is time to look for use cases that can meaningfully be implemented. Collect and assess them. Meaningful means that the use cases are important enough to add value and isolated enough to still be able to manage any risks. These use cases can include improving existing capabilities or adding additional ones. What is important is that for all use cases there need to be KPIs that shall get improved and a cost-benefit analysis.

Remember: Not every problem needs to be solved with the help of a LLM! For some problems it is more efficient to use more “traditional” technologies.

At the same time, it is important to be aware of the level of LLM-knowledge that is available in house. This level is, especially in smaller organizations, not too high. The degree of available knowledge is a boundary factor for project execution. So, it is best to start off with a problem that requires only minor fine-tuning of an existing LLM that is powerful enough to to support multiple of the collected use cases. This way, it is possible to learn fast while also getting results fast. Additionally, fine tuning a model comes at far lower cost than fully training a model, as the number of required training cycles is smaller and the amount of necessary training data is far lower. Out of the many available models (see above), one or more will be capable of supporting most of the selected use cases after fine tuning.

The way to success is thinking big while acting small and to walk before starting to run.

The hype around generative AI, in particular ChatGPT is still at a fever pitch. It created thousands of start-ups and at the moment attracts lots of venture capital. 

Basically, everyone – and their dog – jumps on the bandwagon, with the Gartner Group predicting that it is getting worse, before it is going to be better. According to them, generative AI is yet to cross the peak of inflated expectations. 

There are a few notable exceptions, though. So far, I haven’t heard major announcements by players like SAP, Oracle, SugarCRM, Zoho, or Freshworks.

Before being accused of vendor bashing … I take this is a good sign. Why?

Because it shows that vendors like these have understood that it is worthwhile thinking about valuable scenarios before jumping the gun and coming out with announcements just to stay top of the mind of potential customers. I dare say that these vendors (as well as some unmentioned others) are doing exactly the former, as all of them are highly innovative.

Don’t get me wrong, though. It is important to announce new capabilities. It is probably just not a good style to do so too much in advance, just to potentially freeze a market. This only leads to disappointments on the customer side and ultimately does not serve a vendor’s reputation. 

For business vendors, it is important to understand and articulate the value that they generate by implementing any technology. Sometimes, it is better to use existing technology instead of shifting to the shiny new toy. The potential benefits in these cases simply do not outweigh the disadvantages, starting from cost of running the new technology and extending to the added business value being marginal. Sometimes technology is a solution in search of a problem (anyone remember NFT or  Metaverse?), sometimes the new technology even turns out to be outright harmful.

Although the better is the enemy of the good, not everything new is actually better than the old. Vendors as well as buyers should keep this simple truth in mind.

Specifically looking at generative AI, it is therefore important to look at what the strengths and limitations of this technology are and to map out where business scenarios map to them. For this, I have outlined a simple framework a short while ago.

In brief, value comes out of solutions that adequately address the dimensions of fluency and accuracy. Not every business challenge needs to be addressed with equal fluency or accuracy. However, and this is important, accuracy also covers bias. Bias needs to be understood and managed. 

I have outlined a few examples in that article.

In the past few weeks, vendors like Cognigy, Microsoft, Salesforce and Google did some major announcements covering enterprise use cases of generative AI. In the meantime, Open AI announced version 4 of GPT. Let’s have a look what they were about and how they fit into the fluency and accuracy categories. Notably, all vendors emphasize on a human-in-the-loop functionality being embedded in their new AI features.

Cognigy

Cognigy is a vendor of conversational AI, currently focusing on the call center market, as both, agents and customers can receive a lot of benefit from a conversational AI system. Part of any conversational AI system is the ability to design and implement conversations. As such, the company implements technical as well as business scenarios with the help of generative AI.

The technical scenarios range from seemingly simple ones like the creation of seeding sentences to train the system’s intent detection to the development of conversation flows from written commands.

Both of these scenarios are in the high fluency area, while the creation of conversation flows also requires high accuracy. Generating seed phrases as a training set needs a solid understanding of synonyms and language use while the generation of whole conversation flows also needs precise formulations. After all, code is a precise language, even if visualized by symbols.

The business scenarios are all about improving the understanding of the customer and providing well worded responses, written or spoken – with the spoken ones obviously being more impressive. Cognigy basically combines the strengths of the conversational AI system – keeping focus on the task to be accomplished and connectivity to business systems – with the ability of the generative AI to formulate human-like sentences and to exhibit empathy by reacting on statements that deviate from the core task to be accomplished. Two examples for this are here and here.

Google

Google is an example that shows the hype that is generated by ChatGPT. The company is using generative AI features in its workspace products for quite a while now. For emails, smart reply exists since 2017, smart compose that suggests sentences of fragments thereof since 2018, auto summarizing of documents and spaces since 2022. Spam and phishing filtering is basically available forever and gets continuously improved. On March 14, Google announced the rolling rollout of additional features for Gmail, Docs, Slides Sheets, Meet and Chat. These features will base on Googles own PaLM LLM. Google will start with text generation from topical prompts or a rewriting of a given text.

In contrast to the other vendors, Google is focusing on collaboration efficiency, which is reasonable as Google is not a business applications vendor in the traditional sense. As these features will be made available only in the course of 2023, it remains to be seen how good they really are. However, all of these features need to score high on the fluency scale, with the drafting, replying, summarization and prioritization of texts also requiring high accuracy.

Microsoft

No need to explain what Microsoft in general does. Being the main investor into Open AI, it naturally has a front runner role when it comes to the integration of generative AI by Open AI into business and other applications. Microsoft did two main announcements in the past weeks that stretch the range of business applications. First, it announced an integration of generative AI into its Viva Sales product, already in February. On March 6, the company then announced an “AI copilot for CRM and ERP”.

In February, Microsoft announced the integration of GPT into Viva Sales, with the ability to automatically formulate emails for specific scenarios, like replying to an inquiry, or formulating a proposal, also utilizing data coming via Microsoft Graph. 

In March, this got enhanced to cover not only Viva Sales but functions covering sales, service, marketing and supply chain, followed by an announcement covering Microsoft 365 (formerly known as MS Office) on March 16. These are now being dubbed the Dynamics 365 Copilot and Microsoft 365 Copilot. The functionality in Viva Sales gets enhanced by some scenarios, an additional feedback loop and the capability to create meeting minutes including action items. These scenarios require a high fluency and varying degrees of accuracy, with the summarizing of meetings being at the higher end.

The same capabilities are available in Dynamics 365 Customer Service email and chat. Incoming information is analysed and used to create an answer that includes information from the knowledge base.

These scenarios basically require the same degree of fluency and accuracy as the summarization of meetings to lead to fast and efficient issue resolution.

Marketing scenarios include the prompting of data in natural language, for example to create a target group. The system basically generates the query to fetch the corresponding data, which requires good knowledge of the database schemata, i.e. has high demands to accuracy.

The second scenario is the creation of the text for a marketing email to support the campaign. Notably, there is no support for the creation of a landing page yet. In this scenario, the requirement to fluency is higher than the one to accuracy.

Microsoft Dynamics 365 Business Central gets enhanced by the ability to generate product descriptions based upon product title and product attributes. 

Last, but not least, Dynamics 365 Supply Chain Management now allows to generate emails to suppliers, carriers, etc., based on intelligence surfaced in the news module that gets correlated to existing orders. 

The capabilities of the Microsoft 365 Copilot are largely equivalent to the ones described above, tapping into the Microsoft Graph.

Most of these Copilot features are available in what Microsoft names a limited preview only.

Salesforce

Salesforce, the undisputed leader in CRM, announced support for generative AI, named Einstein GPT, on March 7, as part of its TrailblazerDX event as “the world’s first generative AI for CRM”. The functionality shall be able to create content across sales, service, marketing, commerce and IT. Similar to what Microsoft announced, Einstein GPT is able to generate personalized emails for salespeople, generate specific responses in service scenarios or to generate targeted content for marketers, which includes landing pages. To do so, Einstein GPT extends Salesforce’s proprietary AI models, takes advantage of Salesforce’s Data Cloud and offers out of the box connectivity to Open AI’s AI models. Alternatively, it is possible to utilize other models, e.g., Salesforce partners Anthropic, Cohere, Hearth.ai or You.com.

Salesforce will invest in these companies via its investment arm. Additionally, Einstein GPT is capable of auto creating code from user prompts.

Salesforce’s focus is slightly different from Microsoft’s. No support for ERP and supply chain tasks is obvious. Differences in sales, service and marketing functionalities are more subtle. Where Microsoft concentrates on meeting minutes in its sales functionality, Salesforce supports scheduling of meetings.

What is interesting is Salesforce’s ability to generate kb articles from case notes as part of its service capability. If this functionality also includes improving existing ones, it could be a real game changer in customer service.

Einstein GPT for Marketing supports the generation of content supporting email, mobile, web and advertisement engagements.

With Einstein GPT for Slack Customer 360 apps it is possible to deliver insights like generated summaries of opportunities and other information to Slack. This supports the increasing importance of conversational user interfaces and is a good fit for a generative AI.

Last, but not least, Einstein GPT for Developers enables the generation of code. This functionality uses a Salesforce Research proprietary LLM.

Out of these scenarios, the generation of kb articles and code certainly have the highest demands on accuracy.

At this time, Einstein GPT is in a closed pilot.

My analysis and point of view

In brief, I see much more of a future for these technologies than I have seen in the past major hypes: Web 3, Blockchain, and Metaverse. This is mainly because these three appear to be solutions in search of a problem while we see clearly described enterprise use cases for generative AI.

It is quite obvious that these vendors that I selectively chose are using a land and expand approach. They are starting with very specific scenarios which get extended over time. Many of these use cases extend existing conversational, or in general, AI based scenarios. From a technology point of view the new solution might even replace the older one, which is of no consequence if there is a transparent migration.

The selected scenarios are regularly in the high accuracy and high fluency quadrant, which caters to one of the main use cases of a generative AI.

It is also quite obvious that there is a tremendous struggle for mindshare. All these announcements are coming at about the same time and they are mostly talking about limited availabilities, i.e. betas or trials, indicating work in progress. The actual releases are at some time in the future. As much as I do not like this, this seems to be the way the business works.

Not surprisingly, the vendors are mainly focusing on similar capabilities. This is partly due to my selection. The good news is, that all vendors argue with business benefits instead of promoting technology for technology sake. To provide these capabilities, and this is important, all vendors connect the generative abilities with data that is available in the organization, be it from databases, file repositories, business applications, or chat- and email conversations. 

In my eyes, all these functionalities are helpful in a sense that they take tedious work away from persons; so, they are definitely worthwhile to be trialled.

The challenge

For now, all these features require a human in the loop, i.e., they need active confirmation of a user. Which is a good thing, given that a generative AI still has a tendency to hallucinate, even though e.g. Open AI claims that GPT 4 has a 40 percent higher likelihood to produce factual responses than GPT 3.5.

The goal is to gain trust while the human is still in control. But what do humans do, when they trust, or trust enough? All of the sudden, the suggestion implicitly becomes a decision. This is OK, as long as it is really sure that these “decisions” are good. And that includes that a number of things are guaranteed.

The models must be trained to have minimum bias. There must be a verifiability of this. This does not jibe well with news like Microsoft laying off an ethical AI team while not explaining how this job is done in future. Principles are good, control is necessary. Similarly, Open AI’s turn to not disclosing anymore how the training data got created nor how many parameters it has, etc., citing the competitive environment. As PROs chief AI strategist Dr. Michael Wu recently said in a CRMKonvo, it is impossible to avoid bias, but it must be understood. Then the AI can be a real helper. I’d like to add that the user must be able to understand.

This leads to the second point. The system’s “reasoning” needs to be explainable by the system in a way that not only data scientists understand it. This is an area that many AI’s sorely lack and actions like the ones above are not helpful in this endeavour as they may raise the impression of delivering a black box that is not subject to strict governance.

Lastly, the data that gets used in training and operation must be clean enough. This is a job for not only the vendor but also for the organization that runs the AI. One reason is the reduction of bias, the other one going forward is good recommendations/decisions. While data cleansing is a lost cause, frameworks and regular clean-up work needs to be in place to be sure that the used data is good enough – not perfect, but good enough.

My advice

Given all this, I advise cautious use of these tools in a controlled environment, and to measure the results, both in terms of effort reduced and quality of the output. This is not an easy, but a mandatory, task to fully assess the value of these tools and to establish the necessary trust level. 

Additionally, have the vendors demonstrate which AI principles govern the development and what procedures they have in place to effectively make sure that these principles are baked into the product.

In that sense, the vendors’ early announcements and long closed trial periods are very helpful. 

Now, that we are in the middle of – or hopefully closer to the end of – a general hype that was caused by Open AI’s ChatGPT, it is time to reemphasize on what is possible and what is not, what should be done and what not. It is time to look at business use cases that are beyond the hype and that can be tied to actual business outcomes and business value.

This, especially, in the light of the probably most expensive demo ever, after Google Bard gave a factually wrong answer in its release demo. A factual error wiped more than $100bn US off Google’s valuation.

I say this without any gloating. Still, this incident shows how high the stakes are when it comes to large language models, LLM. It also shows that businesses need to have a good and hard look at what problems they can meaningfully solve with their help. This includes quick wins as well as strategic solutions.

From a business perspective, there are at least two dimensions to look at when assessing the usefulness of solutions that involve large language models, LLM.

One dimension, of course, is the degree of language fluency the system is capable of. Conversational user interfaces, exposed by chatbots or voice bots and digital assistants, smart speakers, etc. are around for a while now. These systems are able to interpret the written or spoken word, and to respond accordingly. This response is either written/spoken or by initiating the action that was asked for. One of the main limitations of these more traditional conversational AI systems is that they are better in understanding than in – lacking a better word – expressing themselves. Relying on well-trained machine learning models, they are also quite regularly able to surface a correct solution for problems in the problem domain that they are trained for. They usually work based on pretrained intents.

And, based on the training data, they usually give quite accurate responses to questions in their domain.

The problem: They are usually limited to a fairly small number of domains.

LLMs, on the other hand, are generally trained “to understand the relationships between words, phrases and sentences in a language. The goal is to have the LLM generate outputs that are semantically meaningful and reflect the context of the input.” This is part of ChatGPTs answer to the question what the purposes of an LLM is. The training set of an LLM is usually a vast amount of “real world” knowledge that usually comes from publicly available sources – aka the Internet. The output itself can be in written, graphical or other formats.

What LLMs excel in is generating responses to questions in a human way. And they can respond to a wide variety of topics. When focusing on text, they are built to generate coherent and meaningful responses.

The problem: They sometimes lack accuracy and give wrong output with full confidence. Even worse, wrong or inaccurate output is not easily identifiable by a user without the requisite knowledge. Again, refer to the Google Bard example that (temporarily, at least) wiped off $100 billion US from Googles valuation. Not picking on Google, there are plenty of examples around that call out ChatGPT or You.com or other tools.

Consequently, the other dimension to look at is accuracy.

The question is whether both dimensions always matter equally or not. In a business sense, one can argue that accuracy matters always. Receiving factual errors in a business conversation is not only a poor customer experience but may in extreme cases even lead to legal issues.

What is also important to understand is that the more accuracy is required the more the necessity of integrating additional systems to augment the LLM increases. An LLM on its own is not much more than some form of entertainment. Even in search engines, LLMs only augment the search by enabling natural language queries and the delivery of results in human language instead of a mere link list.

At least they should do this.

With all this being said, what are business use cases involving a large language model? As said, there needs to be a reasonable accuracy. Obviously, they require fluency as a precondition, as fluency is the core differentiator of an LLM.

Let’s look at some use cases in no particular order of priority.

• I’d start with something that I’d call “storytelling”. This is basically the creation of market-relevant documents that describe the capabilities and differentiating factors of a product, solution, or service. Being somewhat marketing related (no offence intended) and a first point of contact for customers, it needs to be easy to understand without requiring a great deal of technical accuracy. At the same time, it must not be wrong. A stripped-down version of this could be the (improved) generation of social media content, e.g., tweets. Benefits are faster creation of high-level content for general websites but also, more specifically, for ABM scenarios and landing pages. To be able to create this text, an LLM needs to be connected to internal systems holding requirements, specifications as well as communications between the involved persons. This is also a use case that should be implement-able near-term.
• One of the main tasks of people is the writing of, and more so, responding to emails. Especially, in sales scenarios, customer inquiries can get formulated and suggested based upon previous emails and the context given by the CRM system, e.g., about proposals made. This scenario would already require quite a high accuracy to avoid sending out faulty information that might be legally binding. The benefit of this scenario is a significant reduction time needed to send emails, resulting in increased productivity. It is a scenario that Microsoft has already implemented in its Viva Sales solution.
• Generation of documentation is a scenario that somewhat varies in the requirement for fluency. It can be mainly divided into technical and user documentation. While user documentation needs to be extremely readable, the writing style is somewhat less important for technical documentation. Conversely, technical documentation likely needs to have a high degree of technical accuracy that is not needed in user documentation, which means that either different repositories or different parts of source documents need to be used to create the texts and potentially diagrams and images.
• One of the most promising use cases in the short term is customer service, including enterprise search. Here, users want answers to their questions, not just links or something actioned. To achieve this, it is necessary to connect to a conversational AI, business systems and a well-functioning knowledge base that helps in generating accurate answers when searching for something. The actioning of issues is very similar to what conversational AIs do already now. The differences are that the intent detection can be far better as the LLM can create more than enough training sets for this and that the answers given by the system are far more fluent. The same holds true for an inquiry scenario. However, as a word of caution, the accuracy of responses to inquiries depends heavily on the kb content that gets searched by the enterprise search. Therefore, the kb needs continuous and rigorous scrutiny. If this is given, the benefits lie in increased call deflection and customer satisfaction. Properly implemented, benefits include an improved call deflection as more cases can get handled by the system, combined with an increased customer satisfaction as issue handling can become quite easy and efficient for the customer. Cognigy has recently presented some very good examples (here and here) that also include voice in- and output.
• Agent assistance is somewhat easier to implement, as it mostly needs to connect to the customer service application, including the chat history. Having complete access to sales and marketing data, of course is helpful, too. Combined with a sentiment analysis, the LLM can suggest text blocks for the agent to use. The benefits of this are an increased agent efficiency and quite possibly also higher customer satisfaction as the text blocks do exhibit more empathy with the customer’s situation than texts generated without an LLM.

In summary, these five scenarios show use cases involving an LLM that are beyond the hype. They can get implemented in a short time and they can also be easily tied to business outcomes. That way, their benefits can get measured.

Which other use cases do you see? And how would you tie them to business value?