Trust, Betrayed: What AI Providers Bake In When Nobody's Watching | The Ethical AI Series

AI in the News | Ethical AI

Three stories broke within days of each other this week, and none of them were about a model getting smarter. They were about something more fundamental: whether the people and companies building AI can be trusted to tell you what it actually does, and whether they're even asking your permission before they decide for you.

A consulting firm found that most large enterprises still can't tell if their AI spend is paying off. Microsoft quietly flipped on a new AI memory feature for Teams that nobody requested and few will notice. And a lone developer spent a few hundred dollars to fake "the world's best AI model," fooling tens of thousands of people for about 24 hours before he outed himself. Put together, they sketch two separate but related trust problems every IT leader now has to manage: what AI providers choose to disclose, and what gets baked into the AI itself before you ever open the settings menu.

If this pattern feels familiar, it's because it is one: it's the old comic-strip routine where Lucy holds the football, promises this time will be different, and yanks it away the instant Charlie Brown commits to the kick — and he lines up for another try anyway. Enterprises keep running at the ball. They trust a vendor's benchmark number, trust that a new default was chosen with their interests in mind, trust that this quarter's headline claim is sturdier than the one that got quietly walked back last quarter. The fix isn't more optimism. It's refusing to line up for the next kick until someone independent has checked exactly where the ball is.

The ROI illusion: why leadership doesn't trust the transformation it's buying

West Monroe's new "Building the AI-Native Enterprise" research, based on a survey of 417 U.S. business leaders at companies with at least $250 million in revenue, delivers an uncomfortable statistic: 58% of organizations say AI now connects across their business, but 40% are still bolting it onto legacy infrastructure rather than redesigning around it. Even more telling, only 36% measure AI's success by revenue growth, despite majorities reporting faster decisions and lower costs (PR Newswire). West Monroe's own published research names "growing trust and infrastructure constraints" as one of three cross-cutting pressures shaping every organization's AI journey right now (West Monroe).

Read plainly, that's a trust gap hiding inside an efficiency story. Companies are willing to buy AI, deploy AI, and even report time and cost savings from AI — but a majority still won't stake a revenue number on it. That's not a technology failure; it's a governance failure. If leadership can't or won't measure business value, it's usually because nobody set up the accountability structure to do so before the tools went live — the AI equivalent of buying a car and never checking the odometer. For IT and business leaders, the lesson isn't "AI doesn't work." It's that AI transformation requires the same rigor you'd demand of any other capital investment: a baseline, a named owner, and metrics agreed on before rollout, not retrofitted after the fact.

Opt-out by design: what Microsoft baked into Teams without asking

The second story is about consent design, and it hits closer to home for anyone running Microsoft 365. Starting late August 2026, Microsoft Teams will begin generating AI-written "archive" files for eligible meetings — condensed summaries of what was discussed, built to give Copilot and the Facilitator agent long-term memory across meetings. Microsoft's own documentation is explicit that this capability is "default on for your organization," stored in a tenant-owned SharePoint Embedded container, and retained for five years (1,825 days) unless an admin changes it (Microsoft Learn). The official Message Center notice confirms admin controls won't even be available until roughly two weeks before rollout begins (Microsoft 365 Message Center, MC1429018), and independent reporting flags that because Copilot licenses are often bundled into 365 subscriptions, most organizations will meet the activation conditions automatically, without anyone flipping a switch (Windows Latest).

To be fair to Microsoft, the design has real privacy guardrails: no raw transcript text is stored, only AI-generated summaries; access is restricted to meeting participants; encrypted or restricted meetings are excluded entirely. This isn't a surveillance feature dressed up as a memory feature. But "on by default, opt out if you notice" is a pattern worth naming every time a major vendor uses it, because it puts the burden of governance on the customer's admin team rather than the vendor's launch checklist. If your organization runs Teams, the actual action item is concrete: review Meeting Policies > AI Memory and Archive before late August, decide what retention period actually matches your compliance posture, and don't assume five years is the right number just because it's the one Microsoft picked.

The hoax that proves the rule: when "world's best AI model" is just marketing

The third story is almost a parable for the first two. On July 17, a previously unknown outfit calling itself "Basalt Labs" posted on X that its new model, "Monolith-1.0," had just topped every major AI benchmark: 99.4% on Humanity's Last Exam, 95.9% on GPQA Diamond, 96.2% on MMLU-Pro, a 1.6-trillion-parameter architecture, a paper, a team, open weights on Hugging Face (Basalt Labs). For about a day, a meaningful slice of the AI community took it seriously.

It wasn't real. The model actually downloadable from the repo was, in Basalt's own later admission, "an inflated version of the original Qwen 2.5 7B Instruct model" — a small, already-public model padded with junk weights to look enormous, fine-tuned directly on public benchmark answer sets to inflate its scores, with no entry on any credible Humanity's Last Exam leaderboard to corroborate any of it (Ground Truth). The mastermind, a solo developer named Max Scherf, later published his own reveal, walking through exactly how a fake founder team, a jargon-heavy paper, and a purchased batch of social engagement got a fabricated model onto thousands of screens before anyone checked the primary source.

The technical trick here — fine-tuning directly on benchmark answers — is a literal example of what gets "baked in" to a model: the difference between a system that has learned to reason and one that has memorized the answer key is invisible from the outside unless someone runs an independent evaluation. That's true whether the fabrication is a full hoax, as with Basalt, or something subtler: a vendor's cherry-picked internal benchmark, a test run on a setup nobody else can reproduce, a capability claim that only holds on the exact test cases the vendor chose to publish. Basalt Labs is the extreme case that makes the ordinary case visible. If a solo developer with a few hundred dollars of GPU time can manufacture "the world's best model" convincingly enough to fool a chunk of the AI industry for a day, no enterprise buyer should be accepting a vendor's self-reported benchmark score without asking who ran it, how it was graded, and whether it's reproducible against an independent leaderboard.

Even honest benchmark scores don't mean what you think

Here's the part that should worry IT buyers more than the hoax: you don't need an outright fraud like Basalt Labs to get a misleading benchmark number. Epoch AI, the nonprofit research group that runs some of the field's most-cited evaluations, has documented how much a single benchmark score can swing just from how it's measured. Testing the same open model on GPQA Diamond under different prompts and temperatures produced scores ranging from 74% to 80%. On SWE-bench Verified, a widely cited coding benchmark, simply changing which "scaffold" (the surrounding tooling that lets a model use code, search, or other tools) is used swung results by up to 11 percentage points for GPT-5 and up to 15 points for Kimi K2 Thinking. Epoch's own conclusion: "the provider of the model itself" and the choice of scaffold are often bigger factors in a published score than the model's actual capability (Epoch AI).

Two real examples show what that means in practice. When OpenAI unveiled o3 in December 2024, it said the model solved more than 25% of problems on FrontierMath, a benchmark of expert-level math problems that Epoch AI itself built. When Epoch independently re-tested the model months later, it scored closer to 10% — less than half the headline number, with the gap attributed to a more powerful internal scaffold and heavier test-time compute used in OpenAI's original run (TechRepublic). Separately, Epoch measured DeepSeek's R1-0528 model at 33% on SWE-bench Verified using its standard evaluation setup — "well short of Claude 4" and a fraction of the higher numbers that circulated when DeepSeek's own preferred tooling was used (Epoch AI on X). Neither case was a hoax. Both were legitimate labs publishing legitimate numbers from legitimate runs — that simply weren't reproducible under someone else's testing setup.

To its credit, OpenAI has since gotten out ahead of one version of this problem itself: in February 2026 it announced it would stop reporting SWE-bench Verified scores entirely, after its own audit of 138 problems found that at least 59.4% contained flawed test cases that could reject a functionally correct answer — meaning the benchmark itself, not just any one model's score on it, had become unreliable (OpenAI). And it's worth being fair in the other direction too: when Epoch compared self-reported GPQA Diamond scores against its own standardized runs across every major lab, it found no evidence of cheating — the differences were small and within statistical noise (Epoch AI). The honest takeaway isn't that every vendor is gaming the numbers. It's that you can't tell the difference between honest and inflated reporting from the outside — the only way to know is an independent re-check, and right now almost nobody is asking for one before signing a contract.

Who's actually auditing the auditors

If the fix is "someone independent should check," the natural next question is: who's already doing that, and how far does it reach? The honest answer is that the landscape is thinner and more fragmented than it should be.

ForHumanity is a real nonprofit — a 501(c)(3) public charity that runs the Independent Audit of AI Systems (IAAIS), a certification scheme built with more than 1,500 contributors covering 16 governance areas including ethics, bias, privacy, trust, cybersecurity, and compliance with frameworks like the EU AI Act and GDPR (ForHumanity). It's genuine, substantive work — but it's important to be precise about what it audits. ForHumanity certifies an organization's AI governance and compliance practices, not whether a specific model's benchmark score is real. It wouldn't have caught a Basalt Labs-style fabrication, because that's not the layer it operates on.

MLCommons, the consortium behind the long-running MLPerf benchmarks, is closer to the benchmark-integrity layer — it explicitly designs for reproducibility through working-group-defined rules, and its AILuminate suite extends that to chatbot safety testing (MLCommons). But its core benchmarks are mostly about training and inference speed, not the kind of open-ended capability claims (best model on math, best model on expert reasoning) that made the Basalt hoax possible.

Epoch AI is the closest thing that exists today to what this problem actually needs: an independent group that runs its own standardized re-evaluations rather than just aggregating what vendors publish, and that shows its work — full prompts, responses, and per-question scores, using open evaluation tooling anyone can inspect (Epoch AI). The FrontierMath and SWE-bench examples above are exactly the kind of check the industry needs more of. But even here, worth naming the wrinkle honestly: Epoch AI itself was criticized in early 2025 for not initially disclosing that OpenAI had helped fund FrontierMath, the very benchmark it uses to evaluate OpenAI's models (TechCrunch). To Epoch's credit, it responded by maintaining a private hold-out set OpenAI has never seen — but the episode is a useful reminder that "independent evaluator" still needs its own disclosure standards. Auditors need auditing too.

Scale AI, which co-created Humanity's Last Exam with the Center for AI Safety, runs the field's most rigorous public leaderboard for that benchmark. Its methodology ranks models using 95% confidence intervals rather than raw scores, and — most relevant to the Basalt story — it maintains an additional held-out private question set specifically to catch training-data contamination and benchmark hacking (Scale AI / SEAL Leaderboard). That's the actual leaderboard Basalt Labs never appeared on, despite claiming to have topped it.

What the real leaderboard looks like next to a fabrication

None of the credible sources above have ever come close to Basalt's claimed 99.4% on Humanity's Last Exam. As of a late-July 2026 snapshot, the real top scores on the benchmark cluster in the high 30s to low 50s — with the actual frontier leader roughly 46 points below what Basalt claimed for a repackaged small model:

Scores per an Artificial Analysis snapshot from July 20, 2026, as reproduced on Wikipedia's Humanity's Last Exam page. Worth noting for the same trust reasons discussed above: different third-party aggregator sites report meaningfully different "top" HLE numbers depending on the date and tool-use conditions they track, which is itself a small illustration of the underlying problem — when in doubt, check the benchmark owner's own leaderboard (Scale AI / SEAL) rather than a third-party recap.

The gap between the real leaderboard and Basalt's number wasn't subtle. It didn't need Epoch-level statistical rigor to catch — just one look at where the actual frontier models stood.

A rotating cross-check might be the fix — and it's not a solved problem yet

Here's a proposal worth putting on the table: instead of relying on a single evaluator (even an independent one like Epoch AI) to grade a model once, use an independent, untainted ground-truth data source that no vendor curated or trained on, have one AI model verify another model's output against that ground truth, then rotate to a different evaluation model and repeat the check — like running the same laundry through several different wash cycles instead of trusting the first rinse.

That idea isn't just intuition — it lines up with real, active research. A 2024 Cohere paper proposed exactly this kind of setup: a "Panel of LLM evaluators" (PoLL) made up of several smaller models from different model families, rather than one large judge model. The panel correlated better with human judgment than a single GPT-4 judge, showed less bias toward any one model's own outputs, and cost roughly seven times less to run (Verga et al., arXiv). Separately, a 2025 academic survey of "LLM-as-a-judge" research confirmed the underlying problem a rotating panel is meant to solve: a single model grading other models' work shows measurable self-preference bias, favoring outputs that resemble its own style — and while multi-model panels, voting, and debate setups reduce that bias, the research is explicit that none of them eliminate it yet (Li, Jiang, Huang et al., EMNLP 2025).

So the honest framing is: this is a real, actively-researched idea, not a fringe one, and the closest thing running in production today — Epoch AI's independent re-evaluation against a held-out data set, refreshed as models change — is a partial version of it. But nobody yet runs the full version: independent ground truth, rotating evaluator models, repeated cycles, done as standard practice before a vendor's benchmark claim gets repeated as fact. That's a gap. For an industry that will happily spend on model licenses, compute, and consulting, funding a rotating, multi-model verification layer — even a modest one — would buy more real trust than another quarter of benchmark press releases.

Four questions to ask before you trust any AI claim or default

These stories point at the same governance gap from different angles: leadership can't verify AI's business value, vendors can quietly bake consequential defaults into products without real consent, and even a legitimate benchmark score can mean something different depending on who ran it and how. None of that means AI can't be trusted — it means trust has to be built, not assumed. Before signing off on the next AI tool, feature update, or vendor claim, it's worth running four checks:

• Who measures it, and against what? If a vendor cites a benchmark score, ask which leaderboard it's on, whether it's the benchmark owner's own leaderboard (not a third-party aggregator), and whether an independent group like Epoch AI has re-run it.

• Has anyone independently reproduced it? A number nobody outside the vendor has been able to match — the way Basalt's 99.4% never appeared on any real leaderboard — is a claim, not a result.

• What's on by default, and who decided? For any new AI capability touching your tenant — memory features, data retention, model access — check the admin console before rollout, not after a compliance audit flags it.

• What's the actual business metric, not the activity metric? Faster decisions and lower headcount costs are real, but they aren't revenue. If your organization can't tie AI investment to a number leadership already tracks, that's the gap to close before scaling further, not after.

The organizations that get AI-native transformation right won't be the ones that deployed the most tools first. They'll be the ones that never stopped asking who's accountable for what the AI actually does — by design, by default, and by claim. Everyone else is still lining up for the kick. Subscribe to be notified when the next article in our Ethical AI series is posted.