Microsoft's Knowledge Graph and Copilot
And again with the re-bundling...
Welcome to Tidal Wave, an investment and research newsletter about software, internet, and media businesses. Please subscribe so I can meet Matt Levine one day.
In my last post, I wrote that AI-enabled workflows represent one of the core ways that Salesforce and other SORs can drive re-bundling across their product suite over the next decade. The post was focused on Salesforce and its Data Cloud, but Hubspot seems to be similarly embracing ChatGPT with its ChatSpot.ai product1.
For Microsoft, who has been beating the drum the loudest around enterprise applications of AI, the opportunity is not constrained to simply suites around the SOR (e.g., Dynamics 365). The company’s ambitions are much broader because of its Collaboration and Productivity Suite (e.g., Office 365).
And on March 16th, Microsoft “showcased” the next iteration of its broader ambitions for Office 365 at its Future of Work event. And two days prior, on March 14th, Google showcased its own AI-enabled productivity suite. I use the word “showcased,” but neither product suite is in GA.
In the months ahead, we’re bringing Copilot to all our productivity apps—Word, Excel, PowerPoint, Outlook, Teams, Viva, Power Platform, and more. We’ll share more on pricing and licensing soon.
As we embark on this next journey, we will be bringing these new generative-AI experiences to trusted testers on a rolling basis throughout the year, before making them available publicly.
Both companies are slightly more focused on beating each other to the punch above all else. Jokes aside, the implication of both “launches,” especially Microsoft’s, have far-reaching implications for how the software stack looks for the next decade.
While both Salesforce and Hubspot are driving towards re-bundling around the customer record, Microsoft is pushing towards a re-bundling around their knowledge graph with their Copilot and Business Chat. In doing so, the company seems to have either created or adopted a new way of reliably working with the LLMs and searching within a business context.
The Copilot System and Information Network Effects
There were three aspects of Microsoft’s recent launch: Generative AI for Microsoft Apps, Business Chat, and the Copilot System.
The Layup — Generative AI for Apps
Let’s get the easy one out of the way first – Generative AI for Microsoft Apps. Microsoft launched their broadly expected adoption of Generative AI features within the Office Suite. The Generative AI features allow users to automate tasks, summarize information, create content, generate analysis, and even create automations across workflows.
Copilot is integrated into Microsoft 365 in two ways. It works alongside you, embedded in the Microsoft 365 apps you use every day — Word, Excel, PowerPoint, Outlook, Teams and more — to unleash creativity, unlock productivity and uplevel skills….
With Copilot in Word, you can jump-start the creative process so you never start with a blank slate again. Copilot gives you a first draft to edit and iterate on — saving hours in writing, sourcing, and editing time.
Copilot in PowerPoint helps you create beautiful presentations with a simple prompt, adding relevant content from a document you made last week or last year. Copilot in Excel, you can analyze trends and create professional-looking data visualizations in seconds. Copilot in Outlook helps you clear your inbox in minutes, not hours. And every meeting is a productive meeting with Copilot in Teams. It can summarize key discussion points — including who said what and where people are aligned and where they disagree — and suggest action items. Copilot in Power Platform, anyone can automate repetitive tasks, create chatbots and go from idea to working app in minutes.
Again, this was a largely expected launch. Beyond making the core Microsoft applications potentially more valuable to its users, the launch has near-term implications for two segments of the market:
Copywriting startups built on top of OpenAI or other commercial models → unsure2 what happens to these companies. Maybe they move towards becoming hyper-specialized for specific verticals.
High-volume, low-value freelance and agency markets → these companies may have an arbitrage for a short period of time where they can actually use AI to make themselves more efficient and maintain price. But over time, they’ll need to (or get to) focus on higher-order projects/tasks.
Business Chat — Powered by the CoPilot System
The second announcement was Business Chat, which is the company’s Chat Interface that allows users to query/ask for information on the company’s corpus of knowledge (documents, emails, spreadsheets, etc.).
More or less, it's ChatGPT that has access to enterprise files and applications. But if two months ago, after the error-prone demos of BingChat and Google’s Bard (and the awakening of Sydney), you had pitched someone “ChatGPT for the Workplace,” there would have been two responses:
(1) “Oh god”
(2) the training costs may be prohibitive.
So the more interesting and important aspect of the announcement is what’s under the hood of Business Chat, the CoPilot System. Here’s the company’s visual representation of the Copilot System:
The CoPilot System has three components:
Microsoft Application Suite – Teams, Outlook, Powerpoint, Word, and Excel.
Microsoft Graph3 – the data layer that sits across the Microsoft apps that have structured information such as email, contacts, text within Word docs, etc.
Large Language Model – OpenAI’s LLM, which is what users query effectively.
When a user enters a query into Copilot, the following things happen:
The user’s prompt is grounded on Microsoft Graph data, and a modified prompt is created (pre-processing)
The modified prompt is then sent to the LLM for a response or app command (processing)
The response is then grounded again against the Microsoft Graph (post-processing)
Finally, the response is sent back to the user.
At this point, you are probably wondering why I keep bolding grounded.
If we rewind the tape a month, BingChat launched, everyone clapped, Google launched Bard, everyone mocked its errors, and then it turned out that BingChat had the same problem. The issue is that LLMs have a “hallucination” problem, they come up with an answer even if they don’t know the answer. For certain queries, this is probably fine – but for some contexts (e.g., business setting), it can actually be quite harmful.
Imagine that in the scenario above, there was no meeting with “Fabrikam.” Because LLMs are designed to be predictive, they “predict” responses, so an unconstrained LLM may simply predict that “Yesterday we shared the following quote” and reference a made-up sales quote. The VP of Sales may reference that made-up quote in a message to the CEO, and we’re off to the chaos races.
Grounding and LLM-Augmenter
Grounding is designed to obviate this risk, and what seems to be happening under the hood is that Microsoft is using a version of the LLM-Augmenter they wrote about two weeks ago, which is designed to prevent this hallucination issue.
Large language models (LLMs), such as Chat-GPT, are able to generate human-like, fluent responses for many downstream tasks, e.g., task-oriented dialog and question answering. However, applying LLMs to real-world, mission-critical applications remains challenging mainly due to their tendency to generate hallucinations and their inability to use external knowledge.
This paper proposes a LLM-AUGMENTER system, which augments a black-box LLM with a set of plug-and-play modules. Our system makes the LLM generate responses grounded in external knowl- edge, e.g., stored in task-specific databases. It also iteratively revises LLM prompts to improve model responses using feedback generated by utility functions, e.g., the factuality score of a LLM-generated response…
What the LLM-Augmenter system is doing is:
Based on the user query/prompt, the LLM-Augmenter retrieves evidence (i.e., information) from a defined data source (e.g., database or Wiki) using an NLP-enabled search. For CoPilot, the data source is the Microsoft Graph, and only information in the graph can be used as “evidence.”
LLM-Augmenter then takes the “evidence” and the prompt (modified prompt) and queries the fixed LLM (e.g., ChatGPT). The LLM can then generate a response to the prompt grounded in the evidence from the Graph.
The LLM-Augmenter then checks the response against the Graph again to confirm that the response has information that can be directly attributed to data and documents in the Graph.
The verified and cited response is sent back to the user.
You can actually see these steps happen one by one in the demo. And the annotated steps below:
In addition to reducing the risk of LLM hallucinating, there are other benefits to this approach:
Microsoft can use a “fixed” or static LLM, so the users do not have to constantly re-train models to incorporate new corpus of data → cheaper, flexible, and more scalable approach.
Only the relevant or required data is passed to the LLM → more secure and compute-friendly.
Information Network Effects
Driven by the CoPilot system, Business Chat provides users to effectively and reliably search, query, extract, summarize, and build top of all their company’s documents, files, and knowledge in one place. Strategically, the CoPilot System allows Microsoft to foster intercompany network effects. In other words, the summaries and content generated by CoPilot are more powerful if every node (i.e., user) is using tools that feed into the Microsoft Graph because that’s the corpus of data Copilot uses as “evidence.”
In an end-state where knowledge workers are trained to use Business Chat to find information at their company, there’s meaningful value leakage if everyone at the company is not using the same tools. This highlights the primary issue of the last ~10 years of software buying. For the last decade, SaaS applications and productivity tools have thrived on bottoms-up adoption and the departmentalization of software spending, which has inevitably led to the fragmentation of information and tools.
At many companies, especially ones with multiple business units, and a history of acquisitions, it's not uncommon for different teams to use different filesystems (e.g., Sharepoint, Box, Dropbox), collaboration tools (e.g., Teams, Slack, Confluence), and project management tools (e.g., Smartsheet, Asana). For investors, that has been the beauty of software this last decade, bottoms-up GTM and adoption of the cloud have allowed teams to select the tools that best fit their needs. But the corollary to that has been information and knowledge have increasingly fragmented across the enterprise.
The cost has shown up in two areas:
The unquantifiable productivity loss of having information so disaggregated
The very quantifiable seat bloat (seatware) at organizations. For example, the finance team may not directly need access to Notion4, but that is where the Product and Sales prefer to do their planning/collaboration so everyone in Finance needs a seat because you need the transparency and information for EoY planning.
Historically this has been bearable for a few reasons: (1) everyone had unlimited SaaS budgets so getting access to the information in another system was not an issue and (2) there was no information or business cost of not standardizing around one system.
Microsoft Copilot and Business Chat will be effectively highlighting the previously cost of unquantifiable cost of productivity loss from searching for information across the company’s various systems.
More recently, there have been entire companies built to address this problem, such as Glean. From the company’s press release:
Knowledge workers waste 20 percent of their time — more than one day per week — looking for the information to do their jobs (McKinsey).
They average over 6 hours per week duplicating work that has already been done by a current or former coworker (Asana, 2020).
On top of that, context switching between a dozen apps 30 times per day and dealing with interruptions every 3-11 minutes (UC Irvine's Donald Bren School of Information and Computer Science) leads to burnout. It is fueling the "Great Resignation" trend.
Standardizing Around the Microsoft Graph
If Microsoft is successful in convincing enterprise users and, more importantly, managers and executives of the value of the Business Chat, there’s going to be a push to re-bundle/standardize on the Microsoft Graph. The cost of not standardizing will become apparent when users query information and get incomplete or incorrect responses because the was pertinent information that sits outside of the graph.
There are two forms of standardizations: (1) all SaaS applications need to build very tight integrations with the Microsoft Graph so their data can be queried by Copilot, and (2) any SaaS tool that is not frictionlessly compatible with the Graph is replaced by a Microsoft equivalent if one exists.
The Microsoft Graph is not new per se. It’s been around since 2016, and many popular SaaS applications, such as Confluence, integrate directly with the Graph, but there are limitations:
The Confluence Cloud connector has the following known limitations in its latest release:
Confluence Cloud connector does not index attachment files and comments.
Indexing Server and Data Center deployments will be released as a separate connector.
Even companies, like Box, have clunky implementations and limitations when it comes to integrating with the Graph. And none of these are the fault of Box or Confluence, this is Microsoft developer APIs, we’re talking about.
Microsoft 365 users, however, do not have to worry about this issue, they have end-to-end integrations that will allow frictionless movement of data and information across their apps and into the LLMs. But nevertheless, this is a problem that other companies need to solve at some point if they want to sell into the Microsoft installed base. Otherwise, here’s how most conversations will go:
VP of Sales: Hey Tim, did we end up sharing the quote with XYZ Client after the meeting?
Sales Rep: Yeah, it’s saved here in IODT (Indiscriminate Other Document Tool).
VP of Sales: Ah okay. Can you save it to Sharepoint or Presentation next time? It’s not showing up when the CEO asks for a meeting summary in Teams.
Sales Rep: Sounds good. Will do.
VP of Sales: Also, can you stop saving client materials in IODT moving forward. Third time this month this happened.
[6 months later]
CFO: Hey is anyone using this IODT?
VP of Sales: No, we stopped using that.
CFO: Great! Will remove it from the budget.
Obviously, this is a bit cartoonish, but for Microsoft365-centric companies, this is bound to happen in the world that Microsoft is pushing towards. And the sale to managers is very simple because Business Chat is probably one of the greatest enablers of micromanagers the world has ever seen. Managers and executives can now, with a simple query, know anything they want regarding meetings and content5.
What Microsoft is doing with the combination of CoPilot, Business Chat, and Microsoft 365 is creating information network effects within companies. Any tool that does not augment those network effects or, worse, adds friction to those network effects, will likely be jettisoned.
The Non-Microsoft World
Not every company, especially in the tech world, is a Microsoft company. Most companies in Silicon Valley are G-Suite users.
Fortunately for the productivity tool companies around the Bay Area, Google has not been as aggressive as Microsoft with its productivity suite. Microsoft has been, over the last few years with regards to co-opting very semi-successful productivity suite ideas or features. Google does every now and then co-opt features, but they’re worse at shamelessly copying and distributing when compared to Microsoft.
And importantly, G-Suite’s APIs have been more developer-friendly, specifically Calendar and Gmail. Unlike Microsoft, which has Dynamics, Google does not have a large ERP and CRM suite to upsell its customers onto after it pushes them to standardize everything around G-Suite, so the juice may not be worth the squeeze.
If G-Suite decides to forgo the aggressive re-bundling opportunity that Microsoft keeps going after, there’s probably an opportunity for a 3rd party vendor to create the “CoPilot System” for the non-Microsoft ecosystem.
And there are probably a few wedges for the 3rd party CoPilot System: the collaboration layer, search, or neutral middleware platform.
Notion is the most likely candidate at the collaboration layer and has acquired three companies or teams adjacent to collaboration (Calendar, Workflow, and Search).
Glean is a well-funded enterprise search startup that is more or less built to solve the disaggregated knowledge problem, and they have the integration footprint required.
Middleware platforms are tougher to identify because they would have to become a standard across companies to really work.
Algolia has domain knowledge but lacks an integration footprint.
Companies like Merge have the integrations but are a bit removed from the domain problem.
There’s also just the very real possibility that Google goes after this opportunity as well, but their investments around G-Suite leave much to be desired.
Either way — any tool that does not fall neatly into the information graph (Microsoft or G-Suite specific) will be at risk of falling out of the enterprise stack and/or have its TAM severely constrained.
If you’re finding this newsletter interesting, share it with a friend, and consider subscribing if you haven’t already.
Always feel free to drop me a line at ardacapital01@gmail.com if there’s anything you’d like to share or have questions about.
And in the immortal words of @dalibali: “Not investment advice, obviously”. Do your own due diligence.
Just a random footnote, but seeing Dharmesh crank away at ChatSpot has been pretty amazing. The guy clearly pulled a late night before the most recent ChatSpot demo.
Well, I’m sure of what will happen to ones that cannot adapt to the world where the core part of what they offer their customers (i.e., a canvas with access to OpenAI/Anthropic’s model) is broadly available.
Ever since the company’s move to the cloud, Microsoft has been pushing developers to build to this standard and has tried to create an ecosystem around the graph with moderate success.
I’m picking on Notion because of its ubiquity and reference ability. I’m a loyal Notion user.
Sales reps think Sales Managers listening to Gong calls incessantly is bad, Copilot is going to be next level.