<h2>Key Takeaways</h2>
<ul>
<li>Klarna&#8217;s AI customer service story is really two announcements eighteen months apart, and a UK C-suite should read both, not just the first one that went viral. In February 2024, Klarna said its OpenAI-built assistant handled 2.3 million conversations in its first month, doing the equivalent work of 700 full-time agents, cutting resolution time from 11 minutes to under 2, and was projected to add $40 million to 2024 profit.</li>
<li>By May 2025, chief executive Sebastian Siemiatkowski told Bloomberg the strategy had gone too far. &#8220;We focused too much on efficiency and cost,&#8221; he said. &#8220;The result was lower quality, and that&#8217;s not sustainable.&#8221; Klarna started recruiting human customer service agents again, on flexible, remote, gig-style contracts.</li>
<li>The headcount numbers are genuinely mixed, and worth reading precisely rather than as a clean before-and-after. Klarna&#8217;s workforce fell from roughly 5,000 to almost 3,000 employees, about 40%, but Siemiatkowski himself credited much of that to ordinary staff attrition, which he put at 15 to 20% a year, not to the AI system directly replacing people.</li>
<li>The AI didn&#8217;t get switched off. As of the May 2025 reversal, it was still handling two-thirds of all customer inquiries, response times were 82% faster than before the AI launched, and repeat contact rates were still down 25%. What changed was where the company drew the line between what AI should handle and what a person should.</li>
<li>Klarna reversed its AI-first messaging five months before pricing a $1.37 billion IPO on the New York Stock Exchange in September 2025 at a $15.11 billion valuation, itself a steep comedown from the more than $45 billion the company was valued at in 2021. For a UK C-suite, the lesson isn&#8217;t that Klarna&#8217;s AI failed. It&#8217;s that a cost-driven AI rollout and a customer-quality AI rollout are different projects with different failure modes, and Klarna ran the first one before realising it needed the second.</li>
</ul>
<p>Klarna&#8217;s AI customer service assistant is the most cited AI case study in enterprise circles for a reason: it is one of the few with a genuinely public, multi-year arc, a triumphant launch, a candid reversal, and enough concrete numbers at each stage to actually compare. Most AI case studies a UK C-suite reads are either brand-new and unproven or old enough that no one is still checking the follow-up. Klarna&#8217;s is neither. It has been checked, restated and partly walked back by its own chief executive, on the record, twice.</p>
<h2>What Klarna actually announced in February 2024</h2>
<p>Klarna&#8217;s AI assistant, built in partnership with OpenAI, went live globally in Klarna&#8217;s app in February 2024. In a press release dated 27 February 2024, Klarna said the assistant had handled 2.3 million conversations in its first month, two-thirds of Klarna&#8217;s total customer service chat volume, work the company said was equivalent to that of 700 full-time human agents.</p>
<p>Average resolution time fell from 11 minutes to under 2, and repeat customer inquiries, people re-contacting support because their issue hadn&#8217;t actually been resolved, dropped 25%.</p>
<p>Klarna said the assistant was on par with human agents on customer satisfaction scores, available around the clock in 23 markets, and could communicate in more than 35 languages. The company estimated the assistant would drive a $40 million improvement to 2024 profit. &#8220;This AI breakthrough in customer interaction means superior experiences for our customers at better prices,&#8221; said Sebastian Siemiatkowski, Klarna&#8217;s co-founder and chief executive, in the announcement.</p>
<p>OpenAI&#8217;s own chief operating officer, Brad Lightcap, called Klarna &#8220;at the very forefront among our partners in AI adoption and practical application.&#8221;</p>
<h2>The hiring freeze nobody outside the company saw happen in real time</h2>
<p>What followed the launch wasn&#8217;t publicised with the same fanfare. In a Bloomberg Television interview reported in December 2024, Siemiatkowski said Klarna had effectively stopped hiring around a year earlier as the AI system took on more of the workload, with headcount falling 22%, to roughly 3,500 employees, over that period, driven mostly by natural attrition rather than layoffs.</p>
<p>That framing matters: Klarna didn&#8217;t announce mass redundancies tied to the AI launch. It quietly stopped backfilling roles as people left, and let the AI system absorb the resulting gap in customer service capacity.</p>
<h2>May 2025: the reversal, in Siemiatkowski&#8217;s own words</h2>
<p>By May 2025, in another Bloomberg interview reported by trade publication CX Dive and others, Siemiatkowski said the strategy had gone further than it should have. &#8220;We focused too much on efficiency and cost,&#8221; he said. &#8220;The result was lower quality, and that&#8217;s not sustainable.&#8221; He was specific about what he thought the company had got wrong: not that AI itself was the mistake, but that Klarna had let cost-cutting outrun the customer experience it owed people who needed genuinely human help. &#8220;I just think it&#8217;s so critical that you are clear to your customer that there will always be a human if you want,&#8221; he told Bloomberg, adding, &#8220;Really investing in the quality of the human support is the way of the future for us.&#8221; Klarna began recruiting customer service staff again, piloting what CX Dive described as an &#8220;Uber-type&#8221; flexible, remote, gig-style arrangement aimed at &#8220;highly educated students, professionals and entrepreneurs&#8221;, with one stated goal being to bring in-house some of the customer service work Klarna had previously outsourced to third-party contractors, rather than simply re-hiring for the original 700 roles the AI had absorbed.</p>
<p>Separately, in a CNBC interview around the same time, Siemiatkowski gave the fuller headcount picture: Klarna&#8217;s total workforce had fallen from about 5,000 to almost 3,000 employees, a roughly 40% reduction.</p>
<p>But he was careful not to credit all of that to the AI system directly. &#8220;Natural attrition in a company like ours is 15-20% per year, so we shrink naturally 15-20% by people just leaving,&#8221; he said, framing the AI&#8217;s role as enabling the company to not backfill those departures rather than actively cutting jobs itself.</p>
<p>That distinction, between AI-enabled attrition and AI-driven layoffs, is one most retellings of the Klarna story flatten, and it&#8217;s a materially different claim about what the technology actually did.</p>
<h2>The AI didn&#8217;t get switched off. The target changed.</h2>
<p>The detail most useful to a UK C-suite is what didn&#8217;t change in May 2025: the AI assistant was still handling roughly two-thirds of all customer inquiries even after Siemiatkowski&#8217;s public reversal, CX Dive reported, with response times still 82% faster than before the assistant launched and repeat contact rates still down 25%. Klarna didn&#8217;t retreat from AI-handled customer service as a category.</p>
<p>It retreated from treating &#8220;AI handles it&#8221; as sufficient for every kind of interaction, and rebuilt a deliberate lane for cases, complex disputes, financial hardship, anything where getting it wrong costs a customer real money or trust, that get routed to a person instead of resolved end-to-end by the assistant.</p>
<p>The rollout&#8217;s actual failure mode wasn&#8217;t that the AI performed badly against the metrics Klarna originally chose to measure. It was that those metrics, speed and average satisfaction, didn&#8217;t capture whether harder, lower-volume, higher-stakes cases were being handled well.</p>
<h2>The timing a UK C-suite shouldn&#8217;t miss</h2>
<p>Context that rarely makes it into the Klarna story: the May 2025 reversal came five months before Klarna priced its US initial public offering. Klarna listed on the New York Stock Exchange under the ticker KLAR in September 2025, pricing shares at $40 and raising $1.37 billion at a $15.11 billion valuation, a real recovery from the roughly $6.7 billion trough Klarna&#8217;s valuation fell to in 2022, but still a steep comedown from the more than $45 billion it commanded at its 2021 peak.</p>
<p>A company preparing to go public in 2025, after a valuation collapse, had every commercial incentive to tell an aggressive cost-efficiency story through 2024, and then to be seen correcting course on customer quality before investors started asking pointed questions about churn and brand damage.</p>
<p>None of that makes Siemiatkowski&#8217;s admission untrue. It&#8217;s a reasonable, evidence-based reading of why the timing looked the way it did.</p>
<h2>What a UK C-suite should actually take from this</h2>
<p>Klarna&#8217;s AI assistant did what Klarna said it did in February 2024: handled millions of routine conversations faster and, on the metrics Klarna chose to track, at least as well as the humans it replaced.</p>
<p>That part of the story holds up. What failed wasn&#8217;t the AI&#8217;s competence at the task it was measured against; it was the decision to let a cost and efficiency target stand in for a customer-quality target, and to let that substitution run for the better part of a year, through a hiring freeze and a real headcount reduction, before correcting it in public.</p>
<p>For a board weighing its own AI-driven customer service plans, the specific, transferable lesson isn&#8217;t &#8220;AI customer service doesn&#8217;t work,&#8221; which the two-thirds figure that survived the reversal directly contradicts.</p>
<p>It&#8217;s that the metrics chosen to declare an AI rollout a success will determine what gets missed, and that a board should decide, before launch, which failure it would rather catch early: a bad quarter of efficiency numbers, or a public reversal after a year of customers quietly getting worse service than they were promised.</p>


Klarna Cut 700 Jobs’ Worth of Work With AI. Its CEO Now Says That Was the Mistake

<h2>Key Takeaways</h2>
<ul>
<li>Twelve AI deployments published on this site reduce to three real categories: internal knowledge retrieval, customer- or citizen-facing communication, and narrower plain-language interfaces over an existing system.</li>
<li>The largest, most specific results came from the narrowest, most rules-based tasks, such as Amey&#8217;s bid-writing time falling from four days to 30 minutes.</li>
<li>Four of the twelve, moinAI, contexxt.ai, Yepic.AI and Policy-Insider.AI, all run on OVHcloud infrastructure specifically for GDPR and European data-sovereignty reasons.</li>
<li>Jaguar Land Rover&#8217;s case study describes a roadmap and a secured business case rather than a finished, measured deployment, unlike the other eleven.</li>
<li>Across the set, the size and specificity of a reported result tracks how narrow and well-bounded the automated task was, not how ambitious the AI project sounded.</li>
</ul>
<h2>TLDR</h2>
<ul>
<li>Twelve AI deployments published on this site, spanning automotive engineering, water utilities, infrastructure services, legal tech, aged care, local government, customer chat, enterprise knowledge search, generative video and public-affairs monitoring, describe a narrower range of actual mechanisms than the industry variety suggests. Most are built to solve one of two problems: finding information that already existed but was too slow to locate, or communicating with a customer or citizen faster than a human team could manage alone.</li>
<li>The clearest, most repeatedly measured successes cluster around document-heavy, rules-based tasks. Amey&#8217;s bid-writing platform cut drafting time from four days to 30 minutes; an unnamed Dynamiq legal client cut contract review from 1.5 hours to 45 minutes; Anglian Water cut manual governance effort by up to 6,000 hours a year. In each case, the AI was retrieving and structuring information a human already had the standing to evaluate, not making a new judgement call.</li>
<li>A distinct European vendor cluster runs through four of the twelve deployments, moinAI, contexxt.ai, Yepic.AI and Policy-Insider.AI, all built on OVHcloud infrastructure specifically because of GDPR and data-sovereignty requirements their customers demanded. None of the four cite performance or cost as the primary reason for that infrastructure choice; all four cite European hosting as close to non-negotiable.</li>
<li>Not every case study reports a completed, measured result. Jaguar Land Rover&#8217;s engagement describes a roadmap and a secured business case, not a deployed tool with a time saving attached, a genuinely different, earlier stage of the same underlying problem the other eleven case studies describe as finished.</li>
<li>For a UK C-suite, the practical value of reading these twelve together isn&#8217;t a list of AI success stories to imitate individually. It&#8217;s the pattern underneath them: the deployments with the clearest, most defensible results all automated a narrow, well-bounded, previously manual task, and the ones with the vaguest results were the ones asked to do more than that.</li>
</ul>
<p>Twelve AI case studies published on this site look, on the surface, like twelve unrelated stories about twelve unrelated organisations. Read together, in their own words, they describe a much smaller number of actual problems being solved, and a genuinely useful pattern in which of those problems produced results specific enough to trust.</p>
<h2>Twelve deployments, and what each one was actually built to do</h2>
<p>At <a rel="nofollow" href="https://www.networksjournal.com/content/jaguar-land-rover-ai-investment-roadmap-engineering">Jaguar Land Rover</a>, the problem was strategic rather than operational: building a business case for a new, data-driven engineering operating model and prioritising which AI use cases were worth pursuing across a 16,000-strong engineering workforce. Aiimi applied its AI Strategy Framework to build a roadmap measured against impact, feasibility, affordability and enterprise alignment, and separately extended Enterprise Search over JLR&#8217;s product engineering documentation. &#8220;They helped us show where data and AI could make a real difference to engineering, aligning that to business value and shaping an AI roadmap that builds momentum quickly,&#8221; said Dave Hird, JLR&#8217;s Head of Data Driven Engineering &#038; AI.</p>
<p><a rel="nofollow" href="https://www.networksjournal.com/content/anglian-water-asset-document-governance-automation">Anglian Water</a> solved a related but more advanced version of the same problem. Asset and site documentation was scattered across physical &#8220;blue boxes,&#8221; shared drives and specialist applications; manual audits consumed up to 17,000 hours per cycle, and inspectors repeatedly flagged document control issues because field teams couldn&#8217;t reliably find current information on site. Aiimi&#8217;s Virtual Blue Box automatically classifies and tags documents across all 7,000 of the utility&#8217;s sites, running continuous compliance checks. &#8220;By harnessing the capabilities of Machine Learning and Natural Language Processing, it addresses critical challenges faced by technicians in having access to the most relevant and latest information when they&#8217;re on-site and need it most,&#8221; said Stuart Rawdon, Anglian Water&#8217;s Enterprise Content Governance Manager.</p>
<p><a rel="nofollow" href="https://www.networksjournal.com/content/amey-generative-ai-knowledge-platform-bid-writing">Amey</a> had &#8220;decades of asset management expertise&#8221; locked inside individual silos rather than something anyone could quickly search, according to this publication&#8217;s own account. FOIL built Gennie, a generative AI knowledge platform integrated into Amey&#8217;s Microsoft ecosystem. &#8220;We have this vast, deep experience of how we manage assets for our clients. What if you could bring together all of that knowledge and make it searchable and instantaneous?&#8221; said Rob Curley, Amey&#8217;s Project Director.</p>
<p>An unnamed client of <a rel="nofollow" href="https://www.networksjournal.com/content/dynamiq-ai-legal-agent-contract-review">Dynamiq</a> faced the same shape of problem inside a legal department: reviewing thousands of contracts and regulatory documents on tight timelines while relying on costly outside counsel to fill the gaps. Dynamiq built an AI legal agent on the IBM watsonx platform. &#8220;IBM watsonx.data provides a very robust platform for integrating and embedding proprietary client data for AI agents,&#8221; said Vitalii Duk, Dynamiq&#8217;s founder and chief executive.</p>
<p>At <a rel="nofollow" href="https://www.networksjournal.com/content/regis-aged-care-generative-ai-clinical-handover">Regis Aged Care</a>, Australia&#8217;s largest residential aged care provider, clinicians spent two to three hours per shift reviewing more than 200 pages of clinical notes during handovers. Cognizant built a generative AI assistant on Microsoft Copilot Studio, benchmarked against senior clinician standards throughout development. &#8220;Be ambitious about the question you want answered. Don&#8217;t limit yourself in terms of what you think is possible,&#8221; said Imtiaz Bhayat, Regis Aged Care&#8217;s Chief Information Officer.</p>
<p><a rel="nofollow" href="https://www.networksjournal.com/content/ferndown-town-council-decisions-ai-meetings">Ferndown Town Council</a> moved the same underlying idea into local government: a Clerk previously managing agenda preparation and action tracking entirely by hand adopted GovMeetings, a platform powered by Decisions AI. &#8220;Decisions AI has completely reshaped how we run our meetings. It&#8217;s efficient, professional, and intuitive,&#8221; said Liz Bishopp, the council&#8217;s Clerk.</p>
<p>The remaining six deployments move away from internal knowledge retrieval into customer- and citizen-facing communication, and generative content. <a rel="nofollow" href="https://www.networksjournal.com/content/moinai-secures-german-health-insurer-chats-with-eu-based-managed-databases">moinAI</a>, a Hamburg-founded chatbot company serving more than 100 German customers including major insurers, processes several million customer interactions a year. &#8220;Our customers, particularly in the German health sector and SMEs, require the strictest data protection and security standards, and prefer a provider based in Germany or Europe,&#8221; said Florian Nommensen, moinAI&#8217;s Managing Director and CTO.</p>
<p><a rel="nofollow" href="https://www.networksjournal.com/content/contexxt-ai-generative-ai-knowledge-platform">contexxt.ai</a>, a DeepTech startup founded within an academic setting in 2023, built its Sesame platform on a zero-knowledge architecture so that neither it nor its cloud provider can access customer data being processed. &#8220;Research shows we spend up to two hours a day searching for information. Sesame gives that time back,&#8221; said founder and chief executive Torsten Katthöfer.</p>
<p><a rel="nofollow" href="https://www.networksjournal.com/content/yepic-ai-generates-1-million-videos-on-high-end-gpu-cloud-instances">Yepic.AI</a> builds generative AI video and voice translation technology. &#8220;Twenty-five million Americans don&#8217;t speak English, and in the UK nearly 1 million people can&#8217;t speak English well enough to access healthcare without an interpreter,&#8221; said Aaron Jones, the company&#8217;s CEO, framing the tool as an access problem rather than a productivity one.</p>
<p><a rel="nofollow" href="https://www.networksjournal.com/content/policy-insider-ai-processes-5000-daily-policy-documents-on-gdpr-compliant-public-cloud">Policy-Insider.AI</a> processes around 5,000 new political documents a day across more than 40 institutions and 8,000 daily social media posts from over 4,000 policymakers, replacing manual monitoring of public policy across languages and borders. &#8220;The availability of powerful CPU and latest GPU architectures&#8230; is crucial to our platform effectively handling AI workloads,&#8221; said Agata Chudzinska, the company&#8217;s co-founder and Head of AI.</p>
<p><a rel="nofollow" href="https://www.networksjournal.com/content/velotix-ai-assistant-development-time-compliance">Velotix</a> embedded a conversational assistant into its data access platform so users could manage compliance policy in plain language rather than through specialist tooling. &#8220;With AI supporting natural language queries, we give users the freedom to stay focused on their work, while intelligent policy enforcement runs silently in the background,&#8221; said Yoram Segal, the company&#8217;s Chief Technology Officer and Chief AI Officer.</p>
<p>And at <a rel="nofollow" href="https://www.networksjournal.com/content/akzonobel-corporate-event-app-guest-communication">AkzoNobel</a>, the problem was smaller in scale but familiar in shape: fragmented guest communication at corporate events, information scattered across emails and PDFs. QuantumXL built an AI-powered guest app. &#8220;They really listened to the brief and delivered above and beyond,&#8221; said Tina Waller, AkzoNobel&#8217;s Events Manager.</p>
<h2>Comparisons of the projects</h2>
<p>Laid side by side, these twelve deployments split cleanly into three categories rather than twelve unique stories. Six, Jaguar Land Rover, Anglian Water, Amey, Dynamiq&#8217;s client, Regis Aged Care and Ferndown Town Council, are internal knowledge-retrieval tools: AI built to find, summarise or structure information a human employee already had standing to use, just not quickly. Four, moinAI, contexxt.ai, Yepic.AI and Policy-Insider.AI, are customer- or public-facing communication and content tools, chat, translation, video and document monitoring, aimed outward rather than at internal process. Velotix and AkzoNobel sit closer to the first group in mechanism, plain-language interfaces over an existing system, but serve narrower, more specific functions, data compliance and event logistics respectively, than the broader knowledge-retrieval cluster.</p>
<p>A distinct vendor pattern runs through the outward-facing cluster specifically: moinAI, contexxt.ai, Yepic.AI and Policy-Insider.AI are all built on OVHcloud infrastructure, and all four cite the same reason, GDPR compliance and European data sovereignty, rather than raw performance or cost, as the deciding factor. contexxt.ai&#8217;s Torsten Katthöfer put it most directly: &#8220;We were looking for a partner that thinks and acts with an European mindset.&#8221; That&#8217;s a genuinely different kind of infrastructure decision than the internal-tooling cluster made: Dynamiq&#8217;s client and Velotix both chose IBM&#8217;s watsonx platform specifically for enterprise integration and on-premises deployment flexibility for regulated finance and healthcare clients, while Aiimi&#8217;s two clients, Jaguar Land Rover and Anglian Water, and FOIL&#8217;s client, Amey, chose their vendors for domain-specific knowledge architecture rather than infrastructure sovereignty at all.</p>
<p>The clearest outlier among the twelve is Jaguar Land Rover, and it&#8217;s worth reading as a genuine methodological control rather than a weaker result. Every other case study in this set reports a completed deployment with a measured outcome attached. JLR&#8217;s reports a secured business case, leadership buy-in and use cases &#8220;moving into active development,&#8221; a roadmap-stage engagement rather than a finished one. That&#8217;s not a flaw in the case study; it&#8217;s evidence the other eleven organisations had already passed the stage JLR was still working through when this publication&#8217;s account of its project was written.</p>
<h2>Successes</h2>
<p>Where these case studies report a measured result, the results are large and specific rather than vague. Amey&#8217;s bid drafting time fell from four days to 30 minutes, a 95% reduction, with 80% of tender content now generated through the platform and, in the organisation&#8217;s own account, 100% of previously siloed institutional knowledge retained rather than lost to staff turnover. Dynamiq&#8217;s client cut contract review time from 1.5 hours to 45 minutes, cut the time to answer business queries from two days to 60 minutes, and cut clause identification from 20 minutes to two. Anglian Water&#8217;s Virtual Blue Box cuts manual governance effort by up to 6,000 hours a year, delivered zero minor risks in a subsequent BSI audit, reduced Environment Agency and EPA visit actions by 40%, and is estimated to avoid up to £450,000 in annual costs. Regis Aged Care cut clinical handover review from two to three hours down to under one, with zero hallucinations recorded during trials and nearly 60% of users engaging with the assistant daily or several times a week. Ferndown Town Council cut meeting preparation time by 40%.</p>
<p>Velotix&#8217;s result is the most dramatic in relative terms: a conversational assistant that would otherwise have taken six months or more to build launched in one month, an 80%-plus reduction in development time, with an expected 60 to 70% cut in compliance costs and up to 95% fewer manual tasks. Yepic.AI&#8217;s growth figures are the clearest commercial validation in the set: Yepic Studio generated 1 million videos, and the company&#8217;s monthly revenue grew from $2,000 to $50,000 within five months. AkzoNobel&#8217;s app achieved adoption rates above 95%, and Policy-Insider.AI now handles roughly 100,000 queries a day at peak, scaling with major political events rather than breaking under them. Across all eleven completed deployments in this set, not one reports a negative or null result; where a case study is candid about a gap, as Amey&#8217;s and Dynamiq&#8217;s client&#8217;s accounts are about the absence of any downside mentioned at all, that absence is itself worth reading cautiously rather than as proof nothing went wrong.</p>
<h2>Things to learn</h2>
<p>The first lesson from reading these twelve together is that the size of the result tracks the narrowness of the problem, not the ambition of the AI. The deployments with the largest, most specific, most confidently reported numbers, Amey, Dynamiq&#8217;s client, Anglian Water, Velotix, all automated a single, well-bounded task with a clear existing correct answer: a bid response format, a contract clause, a compliance policy, an audit record. The deployments with output that&#8217;s harder to verify independently, generative video, chatbot conversation quality, event-app satisfaction, report adoption and growth figures rather than accuracy or time-saved figures, because there often isn&#8217;t a single correct answer to measure against.</p>
<p>The second lesson is that infrastructure choices in this set were rarely about the AI model itself. Four companies chose the same cloud provider for the same non-technical reason, GDPR and data sovereignty, and two chose IBM&#8217;s watsonx platform for enterprise integration and on-premises flexibility rather than model quality. A board evaluating a vendor for a similar deployment should expect the infrastructure conversation to be dominated by compliance and integration requirements specific to its own regulatory environment, not a comparison of model benchmarks.</p>
<p>The third lesson is the one Jaguar Land Rover&#8217;s case study makes visible by not fitting the pattern of the other eleven: a roadmap-and-business-case stage is a legitimate, necessary part of an AI programme, not a sign of a stalled one, and a board should expect its own early-stage AI initiatives to look like JLR&#8217;s account rather than Amey&#8217;s or Velotix&#8217;s finished numbers. Reading a roadmap-stage case study against a completed-deployment one as though they represent the same level of achievement is exactly the kind of imprecise comparison that makes AI case studies less useful than they should be, and precisely the comparison a UK C-suite should avoid making when it reviews its own AI programme&#8217;s progress against outside examples.</p>


Twelve AI Deployments Compared: What Networks Journal’s Own Case Study Archive Actually Shows

<h2>Key Takeaways</h2>
<ul>
<li>Lockheed Martin, KPMG and Nationwide Building Society, a defence contractor, a professional services firm and a UK financial mutual, all sequenced their <a rel="nofollow" href="https://www.networksjournal.com/ai">AI</a> programmes the same way: consolidate and govern the underlying data first, then build AI on top of it. That pattern held across three organisations with almost nothing else in common, which is itself worth taking seriously.</li>
<li>Lockheed Martin&#8217;s own published pilot numbers are worth reading precisely rather than as an unqualified success: an early generative AI question-answering system improved from 45.45% response accuracy to 56.8% with prompt engineering and to 66.0% after further query optimisation. Against general, inexact industry benchmarks for enterprise retrieval-augmented Q&#038;A systems, which commonly range from roughly 60% to the mid-80s depending on task and tuning, though no single authoritative figure exists, 66% sits at the lower-to-middle end, a credible, improving, honestly reported result rather than a standout one.</li>
<li>KPMG&#8217;s case study contains no quantified outcome at all, but the underlying confidentiality stakes are real and specific, not generic: professional codes including the ICAEW Code of Ethics impose a standing duty of client confidentiality that survives the end of an engagement, and ICAEW&#8217;s own technical guidance requires firms to actively manage conflicts of interest, precisely the risk an AI system with poorly controlled access across many clients&#8217; data could create if access controls aren&#8217;t built first.</li>
<li>Two of the three, Lockheed Martin and Nationwide Building Society, built their governance layer with <a rel="nofollow" href="https://www.ibm.com/solutions/artificial-intelligence">IBM</a>, whose watsonx.governance platform was independently named a Leader in 2025&#8217;s Gartner Magic Quadrant, Forrester Wave and IDC MarketScape for AI governance platforms, a genuine, analyst-confirmed market position distinct from the consumer-facing model providers, <a rel="nofollow" href="https://openai.com/solutions/">OpenAI</a>, Anthropic, Google, that dominate headlines but don&#8217;t compete in this specific governance-platform category.</li>
<li>Context worth holding onto throughout: MIT&#8217;s NANDA research group reported in August 2025 that 95% of enterprise generative AI pilots deliver no measurable return on investment, and only 5% reach production. Judged against that backdrop, all three organisations here, imperfect, still-in-progress results included, are already ahead of where most enterprise AI efforts currently stall.</li>
</ul>
<p>Three organisations with almost nothing in common, a defence and aerospace contractor, a Big Four professional services firm, and a UK building society, arrived at the identical sequencing decision for their AI programmes: govern and consolidate the data first, then build on top of it. That&#8217;s a genuinely useful pattern for a UK C-suite to notice, and it&#8217;s worth reading each of the three cases precisely, including where the published numbers are modest rather than triumphant, rather than treating every AI case study as either a success story or a cautionary tale.</p>
<h2>Lockheed Martin: real numbers, honestly modest</h2>
<p>Lockheed Martin, the global aerospace and defence company, had valuable data locked away across 46 disparate systems spanning multiple data lakes, data management, analytics and business intelligence tools, making it difficult to put that data to consistent use across the enterprise. &#8220;Data is the foundational element by which we pull everything together across the enterprise,&#8221; said Stephan Gerali, Chief Architect and Senior Fellow at Lockheed Martin. Working with IBM, Lockheed Martin replaced those 46 systems with a single, connected and accessible data environment built on IBM watsonx.data, along with watsonx.data integration and intelligence tools. With a unified, scalable data foundation in place, the company built out a broader AI ecosystem, culminating in the Lockheed Martin AI Factory, a secure environment where 10,000 engineers build, iterate on and deploy AI solutions at scale, accelerated by IBM Granite open AI models. Agentic frameworks and virtual agents developed inside the AI Factory, powered by watsonx.governance and watsonx Orchestrate, now help manage complex workflows including handling HR-related inquiries through natural language. &#8220;We&#8217;re getting closer each day to a data-driven culture where folks can take the data they have and immediately make use of it,&#8221; Gerali said.</p>
<p>The specific figure worth reading carefully: in an early generative AI question-answering pilot, response accuracy rose from 45.45% to 56.8% with prompt engineering, and to 66.0% after query optimisation. That&#8217;s a real, transparently reported improvement, and it&#8217;s worth being honest about where 66% actually sits. Enterprise retrieval-augmented Q&#038;A systems commonly report accuracy figures ranging from roughly 60% up into the mid-80s depending on the task and how heavily the system has been tuned; there&#8217;s no single authoritative industry-wide benchmark, figures vary hugely by domain and evaluation method, but against that general spread, 66% reads as a credible, still-improving result at the lower-to-middle end, not a standout one. That&#8217;s not a criticism of Lockheed Martin&#8217;s disclosure, if anything the opposite: publishing an honest, incrementally improving figure rather than a rounded-up success number is precisely the kind of transparency a defence contractor&#8217;s own governance context would predict. The US Department of Defense adopted five formal AI Ethical Principles in February 2020, Responsible, Equitable, Traceable, Reliable and Governable, developed by the Defense Innovation Board, followed by a 2021 implementation plan and specific AI ethics guidance for contractors from the Defense Innovation Unit. That framework applies across a defence contractor&#8217;s AI use generally, not only to weapons systems, which is a plausible, if not explicitly stated, part of why Lockheed Martin&#8217;s own account of this pilot reads as a careful progress report rather than a marketing claim.</p>
<h2>KPMG: no number at all, and a genuinely specific reason the governance mattered first</h2>
<p>KPMG, one of the Big Four professional services firms, needed better mechanisms for securing sensitive information across a complex data estate while maintaining regulatory compliance and data governance standards, and wanted its consultants to be able to access valuable insights safely rather than being blocked by fragmented controls. Following a successful pilot, KPMG signed a three-year engagement with Aiimi to deploy its AI-powered Workplace AI platform, helping the firm classify, refine and govern its data landscape. The platform lets teams identify, secure and intelligently classify data while respecting retention rules, access controls and client obligations. &#8220;To have trust in AI outputs you must first have trust in your underlying data. The Aiimi platform gives us that trust,&#8221; said Chris Allen, Chief Data Officer at KPMG UK. The three-year engagement gives KPMG a structured foundation for governing its data estate as it scales its data and AI ambitions across the firm.</p>
<p>Unlike Lockheed Martin, KPMG&#8217;s case study contains no quantified outcome whatsoever, no accuracy figure, no time saving, no cost reduction. That&#8217;s worth naming plainly rather than filling in with assumption. What&#8217;s genuinely worth understanding, though, is why the governance-first sequencing wasn&#8217;t optional for a firm like KPMG in a way that goes beyond generic caution. Professional codes governing accountancy and professional services firms, including the ICAEW Code of Ethics, impose a standing duty of client confidentiality that survives the end of a specific engagement, and ICAEW&#8217;s own technical guidance specifically requires firms to identify and actively manage conflicts of interest. For a firm serving competing clients simultaneously, an AI system with poorly controlled access isn&#8217;t a hypothetical risk, it&#8217;s precisely the mechanism through which one client&#8217;s confidential information could inadvertently inform work product delivered to a competitor. That&#8217;s a real, citable, professional-standards basis for KPMG&#8217;s approach, not just sound general practice, and it&#8217;s a more specific reason to build access controls before scaling AI than the case study itself spells out.</p>
<h2>Nationwide: the third data point in the same pattern</h2>
<p>Nationwide Building Society took the same sequencing approach at a third kind of institution again, a UK financial mutual rather than a defence contractor or a professional services firm, building an AI Centre of Expertise with IBM Consulting and Microsoft Azure OpenAI Service before scaling deployment, a case study this publication has examined in detail elsewhere, including the real, separately-reported improvement in customer letter response times that emerged after the original case study&#8217;s governance-first framing, alongside Nationwide&#8217;s own acknowledgement, reported separately, that broader return on investment across its wider AI rollout remains difficult to quantify. The relevant point for this comparison is narrower: Nationwide is the third organisation in this set to build the governance layer before the measured result, reinforcing rather than complicating the pattern visible at Lockheed Martin and KPMG.</p>
<h2>The vendor pattern worth naming plainly</h2>
<p>Two of these three organisations, Lockheed Martin and Nationwide Building Society, built their governance layer specifically with IBM, either directly through watsonx.governance or via IBM Consulting. That&#8217;s not incidental: IBM&#8217;s watsonx.governance platform was independently named a Leader in Gartner&#8217;s first AI Governance Platforms Magic Quadrant, The Forrester Wave for AI Governance Solutions, and the IDC MarketScape for Unified AI Governance Platforms, all in 2025, a genuine, three-firm analyst consensus specifically in the enterprise AI governance category, distinct from the consumer-facing model providers, OpenAI, Anthropic, Google, that dominate general AI coverage but don&#8217;t compete in this particular governance-platform space at all. KPMG&#8217;s approach with Aiimi sits outside that specific IBM pattern, a reminder that &#8220;governance-first&#8221; is a genuine cross-industry principle, not proof that every large enterprise is converging on identical vendor infrastructure to get there.</p>
<h2>What a UK C-suite should actually take from all three</h2>
<p>The most useful context for reading any of these three case studies is one none of them state directly: MIT&#8217;s NANDA research group reported in August 2025 that 95% of enterprise generative AI pilots deliver no measurable return on investment, and only 5% ever reach production. Against that backdrop, Lockheed Martin&#8217;s honestly modest 66% accuracy figure, KPMG&#8217;s unquantified but professionally well-justified governance build, and Nationwide&#8217;s governance-first sequencing followed by a real, if partial, measured result, are not underwhelming stories. They&#8217;re evidence of three organisations that got further than the overwhelming majority of enterprises currently attempting the same thing, precisely because each one treated the unglamorous data and governance work as the actual project, rather than a delay before the real one began. For a board weighing its own AI investment, the specific lesson isn&#8217;t which vendor to pick or which sector to imitate. It&#8217;s that the sequencing itself, data and governance before scale, appears to be the genuine differentiator, and that a case study reporting a modest, improving number is, on the evidence here, a better sign than one reporting no number and claiming full success.</p>


Enterprise AI’s Real Bottleneck Isn’t the Model, It’s the Data: Comparing Lockheed Martin, KPMG and Nationwide’s Governance-First Approach

<h2>Key Takeaways</h2>
<ul>
<li>Camping World, the NYSE-listed retailer that describes itself as the world&#8217;s largest RV retailer, built a customer-facing virtual agent, named Arvee, on <a rel="nofollow" href="https://www.ibm.com/solutions/artificial-intelligence">IBM</a> watsonx Assistant integrated with the LivePerson conversational cloud platform. Since deployment, customer engagement across its platforms grew 40 percent, agent efficiency improved 33 percent, and average wait times dropped to 33 seconds, a figure that sits at the edge of what call centre industry benchmarking generally treats as acceptable rather than exceptional, useful context rather than a knock against the result.</li>
<li>Avid Solutions, a research and development firm working with government agencies on self-sustaining agricultural systems, took the opposite approach: <a rel="nofollow" href="https://www.networksjournal.com/ai">AI</a> that customers never see. Using IBM watsonx Orchestrate to automate back-office workstreams, customer onboarding, project management and expense reporting, the company cut project management errors 10 percent and onboarding time 25 percent, with a stated target of 50 percent and 20 percent respectively as the automation expands further.</li>
<li>The two deployments answer genuinely different questions, and comparing them directly on a single metric would miss the point. Camping World&#8217;s AI has to perform live, in front of a customer, in real time, where a bad answer is immediately visible. Avid Solutions&#8217; AI works entirely behind the scenes, where the standard for success is quieter: fewer errors, less time lost, work a customer never has to notice happened at all.</li>
<li>Both tools sit on the same underlying IBM watsonx platform but do meaningfully different jobs. Watsonx Assistant, paired with LivePerson&#8217;s Conversational Cloud, a real integration with roots in a 2017 partnership between IBM and LivePerson, is built for live conversational interaction. Watsonx Orchestrate is IBM&#8217;s agentic workflow automation product, explicitly positioned as complementary to, rather than a replacement for, traditional robotic process automation, adding a layer that interprets intent and adapts across multi-step workflows rather than only executing pre-scripted rule-based steps.</li>
<li>Independent, authoritative benchmarks for both companies&#8217; specific figures are limited. Contact centre wait-time benchmarking is dominated by vendor and industry-blog sources rather than a single independent research body, and no credible independent baseline exists for typical onboarding time or project error rates at small professional and technical services firms, so both companies&#8217; percentage improvements are best read as real, specific, self-reported results rather than performance measured against an audited external standard.</li>
</ul>
<p>Two AI deployments, built on the same underlying IBM watsonx platform, doing almost opposite jobs. Camping World&#8217;s virtual agent talks directly to customers, in real time, across web and phone, where every response is visible the moment it happens. Avid Solutions&#8217; automation never talks to a customer at all, it sorts emails, tracks onboarding steps and flags project errors entirely inside the business, where success looks like nothing going wrong rather than anything dramatic happening. Read together, the two cases are a useful reminder that &#8220;AI for customer experience&#8221; and &#8220;AI for operations&#8221; aren&#8217;t the same investment, aren&#8217;t judged by the same standard, and shouldn&#8217;t be evaluated with the same checklist.</p>
<h2>Camping World: AI that has to perform live</h2>
<p>Camping World, which describes itself as the world&#8217;s largest retailer of recreational vehicles, relies heavily on its contact centres to serve three distinct groups of customers: retail buyers, financial services and insurance customers, and its dealership network. A post-pandemic surge in customer volume exposed gaps in agent management and response times, and the retailer had no 24-hour call centre, so questions arriving outside business hours were delayed, dropped, or simply went unanswered. &#8220;We are a unique business where we serve three very distinct sets of customers who are into the RV lifestyle. There&#8217;s the retail side, financial services like insurance or goods, and the dealership side,&#8221; said Saurabh Shah, Chief Digital Officer and Chief Information Officer at Camping World. &#8220;We have a decently-sized call center, but I can&#8217;t have one agent meeting the needs of each of the three different business units. That creates significant complexities in staffing up our call centers.&#8221;</p>
<p>Brenda Wintrow, Senior Vice President of Sales and Customer Experience at Camping World, said the resulting wait times had become a real problem for customers, compounded by the team&#8217;s limited visibility into performance and chat volume. Camping World turned to IBM Consulting to modernise its contact centres, deploying a solution powered by IBM watsonx Assistant integrated with the LivePerson conversational cloud platform across its web properties and phone channels. The resulting virtual agent, named Arvee, routes and manages capacity dynamically, connecting customers to a live agent for anything too complex for the assistant to resolve, including after-hours lead generation the team previously had no way to capture. &#8220;We were looking to create more free time for our agents to build meaningful and impactful conversations with our clients. That meant removing noisy, quick, simple queries that could be answered faster with automation,&#8221; Shah said. Since deployment, customer engagement across Camping World&#8217;s platforms has grown 40 percent, agent efficiency has improved 33 percent, and average wait times have dropped to 33 seconds.</p>
<p>That 33-second figure is worth reading against a benchmark rather than taken as self-evidently excellent. Contact centre industry commentary generally treats an average wait time under 20 seconds as strong performance and the 20 to 30 second range as acceptable, with times beyond 30 seconds flagged as needing improvement, a widely used framing though it comes from vendor and industry-blog benchmarking rather than a single independent research body. On that scale, Camping World&#8217;s 33 seconds sits right at the edge of acceptable rather than in the standout range, which doesn&#8217;t diminish the achievement, going from an unstaffed after-hours gap to a 24-hour, capacity-managed system is a genuine operational improvement, but it&#8217;s a more accurate way to read the number than treating &#8220;33 seconds&#8221; as an unqualified win in isolation.</p>
<p>The integration itself is real rather than a marketing pairing invented for the case study. LivePerson, a publicly listed conversational AI and customer engagement company, has partnered with IBM on Watson-powered customer care products since at least 2017, an arrangement that has evolved into today&#8217;s watsonx Assistant integration with LivePerson&#8217;s Conversational Cloud platform, still documented in LivePerson&#8217;s own developer resources. At least one third-party industry analysis describes LivePerson&#8217;s revenue as having weakened in its most recent reported quarter, positioning it as a mid-tier player behind larger conversational AI vendors, a characterisation worth knowing as context, though it comes from a single secondary source rather than an audited industry ranking and doesn&#8217;t affect whether the specific integration Camping World deployed is a genuine, supported product.</p>
<h2>Avid Solutions: AI that never talks to a customer</h2>
<p>Avid Solutions, a research and development firm working with government agencies to build self-sustaining agricultural systems both on Earth and in space, found that the day-to-day logistics of running its consultancy, customer onboarding, project management and expense reporting, were time-consuming and error-prone, pulling focus away from its core scientific work. &#8220;Customer onboarding, project management and expense reporting are critical tasks for our business, but they are also time-consuming and error-prone,&#8221; said Dr. Malcolm Adams, Chief Executive Officer of Avid Solutions. Adams and his team adopted IBM watsonx Orchestrate to automate the repetitive steps within these workstreams for customer service representatives, project managers and accountants. One early use case has watsonx monitor incoming emails for a specific subject line and attachment, then automatically route the document to the right repository in the company&#8217;s cloud-based filing system, removing a manual sorting step entirely.</p>
<p>&#8220;By automating tedious workstreams with Orchestrate, we are now able to free up our employees to focus on more strategic work and their own development, while also improving the overall efficiency of our operations,&#8221; Adams said. The automation has cut errors in Avid Solutions&#8217; project management processes by 10 percent and reduced the time it takes to onboard new customers by 25 percent. &#8220;We have also seen a number of qualitative benefits from Orchestrate. Our employees are more satisfied with their jobs because they are no longer bogged down by repetitive tasks. We have also seen an improvement in customer satisfaction because we are able to respond to customer inquiries more quickly and efficiently,&#8221; Adams said. The company is now targeting a 50 percent reduction in onboarding time and a 20 percent reduction in project management errors as it expands the automation further.</p>
<p>Avid Solutions itself is worth being precise about: what can and can&#8217;t be independently confirmed beyond IBM&#8217;s own case study and the company&#8217;s public materials matters here. Its work on precision agriculture and controlled-environment food production tied to space-habitat research is consistent with what the company describes about itself publicly, but this publication could not independently verify which government agencies or contracts Avid Solutions works with, details the case study itself does not specify, and readers evaluating this case study on its own terms should treat those specifics as company and IBM-sourced rather than independently audited.</p>
<h2>The tool behind each deployment is built for a different job</h2>
<p>Both companies used products under the same IBM watsonx umbrella, but the two products are built for genuinely different work. Watsonx Assistant, the engine behind Camping World&#8217;s Arvee, is built for live, real-time conversational interaction, understanding a customer&#8217;s question as it arrives and either answering it or routing it to a human. Watsonx Orchestrate, the tool behind Avid Solutions&#8217; automation, is IBM&#8217;s agentic workflow automation product, and IBM itself explicitly positions it as complementary to, rather than a wholesale replacement for, traditional robotic process automation. The distinction is a genuinely useful one for a UK C-suite evaluating either kind of tool: conventional RPA executes pre-scripted, rule-based steps exactly as defined, while an agentic layer like watsonx Orchestrate is designed to interpret intent and adapt across multi-step workflows, useful for exactly the kind of judgment-light but variable task, sorting an email by its subject line and attachment, that Avid Solutions automated first.</p>
<h2>Two different jobs, two different standards for success</h2>
<p>It&#8217;s worth resisting the temptation to declare one of these deployments more impressive than the other, because they&#8217;re not really comparable on the same axis. Camping World&#8217;s AI is judged by customers directly and immediately, every interaction is a live test, and the cost of getting it wrong is a frustrated customer in the moment. Avid Solutions&#8217; AI is judged by whether errors and delays quietly stop happening inside the business, a standard that produces no dramatic customer-facing moment at all when it works. Neither company&#8217;s percentage figures come with an independently audited external benchmark to compare against, contact centre wait-time standards are themselves drawn from vendor and industry commentary rather than a single authoritative body, and no credible independent baseline exists for onboarding time or project error rates at small technical services firms, so both sets of results are best read as genuine, specific, self-reported outcomes rather than measured against an external yardstick. What the two cases together show a UK board is that &#8220;we&#8217;re investing in AI&#8221; isn&#8217;t one decision, it&#8217;s at least two distinct ones, whether the AI needs to perform in front of a customer or quietly get better at not making mistakes behind one, and the two require different products, different success measures, and different tolerance for visible failure while the system is still learning.</p>


AI That Talks to Customers vs AI That Never Does: Camping World and Avid Solutions Compared

<h2>Key Takeaways</h2>
<ul>
<li>Trove, whose case study is published by the consultancy Hashorn, was running its machine learning models on manually managed EC2 instances with no monitoring, no model registry and no drift detection in place, a setup where the risk stays invisible until it isn&#8217;t. It wasn&#8217;t: a customer SLA breach hit two weeks before the engagement began, and support tickets, not any internal alert, were how Trove&#8217;s team found out.</li>
<li>Eight weeks later, after Hashorn rebuilt the pipeline on AWS EKS with proper orchestration, a model registry, packaging and drift detection, the new system caught a feature schema change in week six and stopped it from degrading recommendations for a single customer. That&#8217;s the core of the story: not that Trove replaced its models, but that it built the capability to notice when a model was quietly going wrong before a customer had to report it.</li>
<li>What Trove was missing has a name and an academic pedigree. A 2015 Google paper presented at NIPS, &#8220;Hidden Technical Debt in Machine Learning Systems&#8221; by D. Sculley and colleagues, describes exactly this pattern: ML systems accumulate forms of technical debt that ordinary software doesn&#8217;t, including undeclared data dependencies and the absence of monitoring for the specific ways a model can silently degrade, distinct from the debt of writing the model itself.</li>
<li>The specific failure mode described, drift between online and offline features, is a named, well-documented problem often called training-serving skew, where a model behaves differently in production than it did in training because the two environments compute features in different ways. It&#8217;s one of the most common causes of a model that still runs without erroring, while quietly making worse predictions.</li>
<li>Post-deployment monitoring is also becoming a regulatory expectation for a defined category of <a rel="nofollow" href="https://www.networksjournal.com/ai">AI</a> systems, not only an engineering best practice. Under the EU AI Act, providers of high-risk AI systems must maintain continuous post-market monitoring for the system&#8217;s full operational life under Article 72, with serious incidents reportable within 15 days under Article 73, though the compliance deadline for the underlying high-risk obligations was postponed from 2 August 2026 to 2 December 2027 by the EU&#8217;s Digital Omnibus on AI. Whether Trove&#8217;s own recommendation models would fall into that specific high-risk category isn&#8217;t addressed by the case study, and most recommendation engines don&#8217;t, but the direction of travel, formal expectations for monitoring after deployment, is the same one Trove&#8217;s rebuild moved towards on its own.</li>
</ul>
<p>The most useful detail in Trove&#8217;s case study isn&#8217;t the latency number, though it&#8217;s a real one. It&#8217;s the sentence describing how the company found out its recommendation system had a problem: support tickets started landing. Not a dashboard, not an alert, not an internal review. A model had been serving degraded predictions for long enough that customers noticed before anyone at Trove did. That&#8217;s a specific, common and well-studied failure in machine learning systems, and Trove&#8217;s response to it, an eight-week infrastructure rebuild rather than a quick patch, is worth reading as a case study in what &#8220;shipping a model&#8221; is missing if it doesn&#8217;t also include a way to know when that model quietly stops working.</p>
<h2>What was actually missing</h2>
<p>Trove was running seven machine learning models on manually managed EC2 instances with no monitoring, no model registry and no drift detection in place. The risk that setup carried stayed invisible until it wasn&#8217;t: a customer SLA breach hit two weeks before the engagement began, exposing undocumented training data lineage, unnoticed drift between online and offline features, and a complete lack of deployment isolation between models. Worth noting precisely: the case study&#8217;s own figures shift partway through, from seven models at the outset to &#8220;all 12 models&#8221; deployed by the end, with no explanation given for the difference. That&#8217;s not something this article can resolve, the source itself doesn&#8217;t address it, but it&#8217;s worth stating plainly rather than smoothing over, since a &#8220;fully factual&#8221; reading of the case study means reporting the inconsistency rather than picking whichever number reads better.</p>
<p>Hashorn built a full MLOps platform on AWS EKS over eight weeks, using Argo Workflows for orchestration, MLflow as the model registry, BentoML for packaging, and Evidently AI to handle drift detection. All four are genuine, actively maintained open-source projects rather than proprietary or obscure tooling: Argo Workflows is a Kubernetes-native orchestration engine and a graduated Cloud Native Computing Foundation project; MLflow, an ML lifecycle and model registry platform, was created and open-sourced by Databricks in 2018 and has more than 20,000 GitHub stars; BentoML, a model packaging and serving framework founded the same year, has thousands of stars of its own; Evidently AI is an open-source library specifically built for the kind of drift and data-quality monitoring Trove lacked. Models were migrated incrementally rather than in one risky cutover, with canary rollouts and parity testing at every step to catch problems before they reached production traffic. &#8220;We had a customer SLA breach that nobody on our team saw coming until support tickets started landing. Eight weeks later, drift detection caught a feature schema change before any customer saw degraded recommendations. That&#8217;s the difference,&#8221; said Vikram Shah, Head of ML at Trove.</p>
<h2>Why &#8220;drift&#8221; is the specific word that matters</h2>
<p>It&#8217;s worth being precise about what drift actually means, because it&#8217;s not a vague catch-all term. Data drift describes a change in the distribution of the input data a model sees over time, the relationship between inputs and the correct output hasn&#8217;t changed, but the data no longer looks like what the model was trained on. Concept drift is different and often more serious: the underlying relationship between inputs and the correct output itself changes, so the same input now genuinely calls for a different answer than it used to. The specific problem Trove&#8217;s case study names, &#8220;drift between online and offline features,&#8221; is a related but distinct failure often called training-serving skew, where the features a model sees in production are computed differently, from a different pipeline or a different code path, than the features it saw during training. A model can pass every offline test and still degrade silently in production if the two environments disagree about how a feature is actually calculated, which is precisely the kind of gap invisible monitoring would have caught and invisible infrastructure did not.</p>
<p>This isn&#8217;t a niche concern specific to Trove. A 2015 Google paper presented at NIPS, &#8220;Hidden Technical Debt in Machine Learning Systems&#8221; by D. Sculley and co-authors, argued that machine learning systems accumulate their own distinctive forms of technical debt beyond ordinary software, including undeclared data dependencies, entangled components where changing one part silently affects another, and the absence of the kind of monitoring that would catch these problems before they reach a customer. Trove&#8217;s pre-engagement setup, undocumented training data lineage, no deployment isolation between models, no drift detection, reads as close to a checklist of the exact debt categories that paper describes a decade earlier, evidence that this is a recurring, well-understood pattern in how ML systems fail rather than a one-off oversight.</p>
<h2>What actually changed, and what it&#8217;s worth against a real benchmark</h2>
<p>All 12 models are now deployed with zero downtime, and the three to four manual deploys Trove&#8217;s team used to push out every week have dropped to zero. Inference latency fell 63 percent, from 240ms to 88ms, and in week six the new drift detection system caught a feature schema change before it could degrade recommendations for a single customer. The latency improvement is worth placing in context rather than taken as a bare number: moving from ad hoc, single-request serving on manually managed instances to a proper packaging and serving framework with batching support is a documented source of substantial latency and throughput gains independent of the model itself, since much of the improvement comes from how requests are served and batched rather than from the underlying model changing. That makes a 63 percent reduction plausible and consistent with the kind of gain this category of infrastructure change typically produces, though this publication could not independently verify Trove&#8217;s specific figure against a matching external benchmark, so it should be read as a credible, company-reported result rather than an independently audited one.</p>
<p>The week-six catch is the more important result of the two, even though it comes with no percentage attached. A monitoring system&#8217;s value isn&#8217;t measured by a single dramatic save, it&#8217;s measured by whether it keeps working quietly in the background indefinitely, and one confirmed catch within the first six weeks of operation is real, if necessarily limited, evidence that the system does what it was built to do. Whether it keeps catching problems a year or three years from now is not something any single case study, this one included, can answer in advance.</p>
<h2>The regulatory direction this points towards</h2>
<p>Trove&#8217;s rebuild happened for a customer-facing, commercial reason, an SLA breach and the support tickets that followed, not because of a regulatory requirement. But the broader shift towards treating post-deployment monitoring as mandatory, not optional, is increasingly showing up in regulation as well as good engineering practice. Under the EU AI Act, providers of high-risk AI systems are required under Article 72 to establish a continuous post-market monitoring system covering the system&#8217;s entire operational lifetime, with serious incidents reportable within 15 days under Article 73. Those specific obligations attach to systems classified as high-risk, and the compliance deadline for the underlying high-risk category obligations under Annex III was recently postponed, from the original 2 August 2026 to 2 December 2027, by the EU&#8217;s Digital Omnibus on AI. Most recommendation engines, including what Trove&#8217;s case study describes, don&#8217;t fall into the AI Act&#8217;s high-risk categories, so this specific legal obligation likely doesn&#8217;t apply directly to Trove&#8217;s own system, and the case study makes no claim that it does. What it does show is that the standard Hashorn built towards for entirely commercial reasons, know when your model is degrading before a customer does, is the same standard regulators are now writing into law for the AI systems they&#8217;ve decided carry the highest stakes.</p>
<h2>What a UK C-suite should take from this</h2>
<p>The specific lesson isn&#8217;t &#8220;hire a consultancy&#8221; or &#8220;adopt these four tools.&#8221; It&#8217;s that a machine learning model in production isn&#8217;t a finished deliverable in the way a piece of conventional software often is treated as one; it&#8217;s a system that can degrade continuously and silently while technically still running without errors, and the only way to know that&#8217;s happening is to build the capability to notice, deliberately, before it started. Trove found that out from a customer complaint. The genuinely useful takeaway for any board sponsoring a machine learning initiative is to ask, before the model ships rather than after the first SLA breach, exactly how the organisation would find out if this specific model started quietly getting worse, and whether the honest answer today is a dashboard, or a support ticket.</p>
