|
|
This week's Deep Dive is personal. I rushed to get last Tuesday's edition of the AIE out as I headed to my Great Aunt Janet's funeral. That's also why this week's Deep Dive is a day late. Tuesday was rushed, and I leaned too heavily on AI. The quality gap between my normal process and what I published is the best argument I can make for why human-in-the-loop review isn't optional—it's the whole point. |
Organizations that skip review checkpoints see measurably worse outcomes—an IDC/Lenovo study found that just 4 of 33 AI proofs of concept graduated to full production, often due to quality and trust gaps. The value of AI isn't in the output—it's in the judgment layer around the output. Speed without review is just fast failure. Every organization using AI needs a defined review protocol—not as bureaucracy, but as the mechanism that turns AI drafts into work you'd put your name on.
|
As agentic AI makes it easier than ever to delegate entire workflows, the temptation to "just let the AI handle it" will only grow. This edition is about why that instinct is dangerous and what to do instead.
[Big thank you to my little sister, Melinda, who was the additional human in the loop for this edition to make sure it wasn't AI Slop.] |
|
ALL THINGS AI LUNCH AND LEARN SCHEDULE |
| | |
|
|
As cofounder of All Things AI, I try to host a lot of live events in Raleigh, NC, but have found that even locals are strapped for time, so we've done many virtual events to make it convenient for everyone. I expanded that program to include my other communities to help provide high-quality content. |
Agentic Systems: How AI Actually Gets Work Done
Tuesday, February 10 · 12:00 PM EST · Online |
The Missing Link: Adding Your Data to Your App
Tuesday, February 24 · 12:00 PM EST · Online |
Lunch and Learn: How Small Orgs and Non-Profits are Getting Value out of AI
Tuesday, March 3 · 12:00 PM to 1:00 PM EST |
Missed a session? Catch up on recent recordings. Here are some past events: |
|
|
MORE FROM THE ARTIFICIALLY INTELLIGENT ENTERPRISE NETWORK |
| | |
|
|
|
|
|
The Cost of Trusting AI Too Much |
A Cautionary Tale about a Funeral, a Deadline, and the Review I Skipped |
Yesterday, we buried my great Aunt Janet. She made it ninety-nine years and four months—and of those, ninety-nine years and three months were genuinely good. This last month, she got sick, and that was that. She was one of the best ladies I've known. |
Her funeral meant two days away from work. Two days doesn't sound like much until you publish a newsletter on a fixed schedule. I was behind before I started, and instead of doing the careful dance I normally do between AI tools and my own editorial judgment, I delegated more than I should have. I leaned on the machine and let it carry the weight I normally carry myself. |
The result is here. I'm not going to pretend it's terrible—it's not. But it's not up to the standard I hold myself to, and the difference between that piece and what I normally publish is the difference between AI-assisted work and AI-delegated work. If you've been reading this newsletter long enough, you can probably tell which sections had less of me in them. |
I'm sharing this because the lesson is too valuable to waste: the human in the loop isn't a nice-to-have. It's what makes AI output worth publishing. |
The phrase "human in the loop" gets used so often in AI circles that it has lost its edge. It sounds like a governance checkbox—something you tell regulators you do. But for anyone actually producing work with AI, the loop is where value gets created. Remove the human, and what you're left with is fast, fluent, and frequently wrong. |
What "Human in the Loop" Actually Means |
In machine learning, human-in-the-loop (HITL) refers to systems where people provide feedback, correction, or approval at defined checkpoints in an AI workflow. The concept originated in manufacturing and aviation domains, where autonomous systems needed human oversight at critical decision points. |
In the enterprise AI context, HITL has expanded to mean something broader: the practice of maintaining human judgment as a required step before AI output reaches its audience, whether that audience is customers, investors, colleagues, or the public. This applies to content, code, analysis, customer communications, financial models—anything where the output carries consequences. |
The reason it matters now is that the tools have gotten good enough to skip it. Two years ago, most AI output required heavy editing. Today, Claude, GPT-5.2, and Gemini produce text that reads well on the first pass. Code assistants generate functional scripts. Image generators produce professional-quality visuals. The quality floor has risen dramatically—and that's precisely what makes the oversight question more dangerous, not less. When the output looks good, you stop looking carefully. |
Why the Data Says You Can't Skip It |
The numbers tell a consistent story. An IDC/Lenovo study found that just 4 of 33 AI proofs of concept graduated to full production, a staggering failure rate driven by familiar problems: lack of executive ownership, low user adoption, and weak infrastructure. But buried in those failure modes is a quality and trust problem. Teams that deploy AI without clear review processes produce inconsistent output, lose stakeholder confidence, and eventually abandon the tools. |
The good news: more recent data from Lenovo's 2026 CIO Playbook suggests improvement, with 46% of AI POCs now progressing to production. The difference? Organizations that invest in structured oversight, including human review checkpoints, are the ones making it through. |
A separate study found that companies using AI content workflows produce five times more content while maintaining quality, but only when they build explicit review checkpoints into the process. The "maintaining quality" part doesn't happen automatically. It happens because someone designs a system that catches errors before they ship. |
People who use AI regularly know it hallucinates, takes shortcuts, and occasionally produces output that is confidently, fluently wrong. The practical experience of working with these tools teaches a consistent lesson: AI is an extraordinary first-draft engine, but the value lives in what happens after the draft. |
The gap between AI-assisted and AI-delegated work isn't subtle. Here's what it looks like: |
AI-assisted: You use AI to generate a first draft, research summary, or code scaffold. You then review, restructure, fact-check, add your judgment, and publish something you'd put your name on. |
AI-delegated: You give AI the task, skim the output, and ship it. The result is technically competent but lacks the editorial decisions, strategic framing, and quality checks that make work valuable. |
I lived the difference this week. The gap is real. |
How to Build a Review System That Works |
The challenge with HITL isn't convincing people it matters—it's making it practical. Review systems fail when they're too heavy for the pace of work. They succeed when they're embedded in the workflow, not bolted on after. |
Phase 1: Define Your Quality Threshold |
Before you can review effectively, you need to know what "good enough" looks like. For my newsletter, here are the editorial standards: no unverified statistics, no generic advice, concrete business scenarios, honest limitations, and a voice that sounds like a person with opinions—not a language model summarizing the internet. |
Your threshold will differ by context. For code, it might be passing tests plus a security review. For customer communications, it might be brand voice consistency and factual accuracy. For financial analysis, it might be source verification and assumption documentation. The point is to make the standard explicit so review isn't subjective. |
Phase 2: Design Checkpoints, Not Bottlenecks |
The best review systems have two to three checkpoints maximum. Having more than that, people route around them. Here's a practical structure: |
Checkpoint 1 — Structural Review (before detailed work begins): Does the AI output address the right question? Is the framing appropriate? Is the scope correct? This is a two-minute check that prevents two-hour rewrites. Checkpoint 2 — Quality Review (before delivery): Does the output meet your quality threshold? Are facts accurate? Is the tone right? Are there hallucinations, missing context, or logical gaps? This is where most value gets added. Checkpoint 3 — Final Sanity Check (before it ships): Read it as your audience would. Does it make sense? Would you put your name on it? This is the "sleep on it" step, compressed into five minutes.
|
Phase 3: Make It Sustainable |
The reason I skipped my process last week wasn't laziness—it was time pressure. And that's the most common failure mode. When deadlines compress, review is the first thing that gets cut. |
The fix isn't discipline; it's design. Build your review steps into your calendar, not your to-do list. Set up templates that force you through checkpoints. Use tools that automatically flag common issues. And critically, give yourself permission to delay or reduce scope rather than skip review entirely. A shorter piece you're proud of beats a longer piece that undermines your credibility. |
Key Success Factors: |
Define quality standards before you start, not after you finish. Keep checkpoints to no more than three per workflow. Build review into the schedule, not around it. When time is short, reduce scope rather than skip oversight
|
Common Missteps |
Treating AI review like proofreading: The most common mistake is reviewing AI output only for grammar and spelling. The real risks are structural—wrong framing, missing context, hallucinated facts, and generic analysis that sounds smart but says nothing. Review needs to be substantive, not cosmetic. |
Over-automating the review itself: Some organizations try to solve the AI quality problem with more AI—using one model to check another. This can help catch factual errors, but it doesn't replace human judgment about relevance, tone, strategic framing, or whether the output actually serves the audience. AI checking AI is useful for specific tasks (grammar, code linting, fact extraction), but the editorial layer remains human. |
Skipping review when the output "looks good": This is the trap I fell into. Modern language models produce output that reads well even when the substance is thin. The better AI gets at surface quality, the more important it becomes to check depth, accuracy, and originality. Fluency is not the same as quality. |
Business Value |
The business case for HITL isn't abstract. Organizations that maintain human review in their AI workflows see measurable advantages in three areas. |
Quality consistency: Gartner predicted that 30% of legal tech automation would include human-in-the-loop by 2025—not because the technology can't handle it, but because the stakes require it. The same logic applies to any domain where errors carry consequences: finance, healthcare, customer-facing communications, and strategic analysis. |
Trust and credibility: When nearly half of organizations cite skill gaps as a major barrier to scaling AI, the message is clear: the bottleneck isn't technology. It's the human capacity to direct, review, and improve AI output. Organizations that invest in review capability—not just AI capability—build the trust required to scale. |
Competitive differentiation: Here's the counterintuitive insight: when everyone has access to the same AI models, the quality of human oversight becomes the differentiator. Two companies using Claude to draft customer proposals will produce different results, based entirely on the review process around the AI. The one that catches errors, adds context, and applies strategic judgment wins. The one that ships the first draft loses. |
ROI Considerations: |
Reduced error correction costs—catching mistakes before publication is cheaper than fixing them after. Higher audience trust and engagement—consistent quality builds subscriber loyalty and reduces churn. Faster scaling of AI adoption—when teams trust the review process, they're more willing to experiment with AI across new use cases.
|
Competitive Implications: As agentic AI expands—with tools like OpenClaw, Claude Cowork, and Claude Code giving AI direct access to your files, browser, and desktop—the review question becomes even more critical. Autonomous AI that can take actions, not just generate text, raises the stakes for every checkpoint you skip. |
|
FROM ZERO to 1,200 MONTHLY VISITORS-ON AUTOPILOT |
| | |
|
|
The SEO game changed. Now you need to rank on Google and get cited by AI assistants like ChatGPT and Perplexity. That's a lot of content. |
Run your content marketing autonomously—from keyword research to publishing—so you can focus on closing deals instead of managing writers. Tely AI optimizes both traditional SEO and the new "GEO" (Generative Engine Optimization) landscape. |
Real results: customers see 64% reduction in customer acquisition costs. |
|
AI TOOLBOX |
Prompt of the Week: AI Output Review Checkpoint Designer |
Every team using AI to produce work—reports, code, customer communications, and analysis— faces the same question: where do we put the human? Too many checkpoints and you lose the speed advantage. Too few and errors ship. Most teams default to reviewing everything or reviewing nothing, neither of which scales. |
This prompt uses role-based framing to position the AI as a quality operations strategist, then applies structured criteria — error consequences, audience sensitivity, and reversibility—to design review checkpoints that align with the actual risk profile of each output type. The tiered output format prevents the common failure of treating all AI work as equally risky. |
The Prompt |
You are a quality operations strategist designing human review checkpoints for an AI-assisted workflow. Your goal is to create a review system that catches high-consequence errors without creating bottlenecks that eliminate AI's speed advantage. ## Context Our team in [DEPARTMENT/FUNCTION] at a [COMPANY SIZE/TYPE] company uses AI to produce [DESCRIBE AI-GENERATED OUTPUTS — e.g., draft reports, code, customer emails, Marchketing copy, data analysis]. ## Current AI Outputs [LIST 5-10 TYPES OF AI-GENERATED WORK PRODUCTS] ## Instructions For each output type, evaluate against these criteria: 1. Error consequence — What happens if this output is wrong? (Minor embarrassment → Financial loss → Legal/safety risk) 2. Audience sensitivity — Who sees this? (Internal team → Cross-functional → Customer-facing → Public) 3. Reversibility — How easily can an error be corrected? (Instantly → Within hours → Permanent/difficult) 4. Pattern consistency — How predictable is quality? (Highly consistent → Variable → Unpredictable) 5. Volume — How many of these do we produce per week? ## Output Format Provide a tiered review system: **Tier 1: Auto-Approve** (Low consequence + internal + reversible) - Output type: [Why it's safe to skip review] - Spot-check frequency: [e.g., 1 in 10] **Tier 2: Light Review** (Moderate consequence OR external audience) - Output type: [What to check and what to skip] - Review time target: [Minutes per item] - Reviewer: [Role/skill level needed] **Tier 3: Full Review** (High consequence + external + hard to reverse) - Output type: [What requires detailed review] - Review checklist: [3-5 specific items to verify] - Reviewer: [Role/skill level needed] - Escalation trigger: [When to flag for senior review] **Implementation Note:** Recommended total review time budget as percentage of AI production time.
|
Example Use Case |
A marketing director pastes their list of AI-generated outputs: social media posts, blog drafts, email campaigns, press releases, executive presentations, and internal meeting summaries. The prompt classifies internal meeting summaries as Tier 1 (auto-approve with weekly spot checks), social posts and blog drafts as Tier 2 (quick tone and fact check, three minutes each), and press releases and exec presentations as Tier 3 (full review with brand voice checklist and source verification). The result: 70% of outputs flow without delay, and review effort concentrates where errors actually matter. |
Variations |
For engineering teams: Replace "audience sensitivity" with "blast radius". How many systems or users are affected if the code is wrong? For regulated industries: Add a "compliance flag" criterion that auto-escalates any output touching financial, medical, or legal claims to Tier 3 regardless of other scores.
|
I am sorry about the rushed Tuesday version and the late Friday edition. I hope you understand. |
I appreciate your support. |
|
|
|
No comments:
Post a Comment