AI
AI Content Generation Quality Control: What I Learned the Hard Way
May 20269 min read
AI
I use AI-generated content in my projects: workout descriptions, exercise technique guides, and blog post drafts. Early on, I shipped AI content with minimal review and paid for it with user complaints about inaccurate exercise form descriptions.
AI content fails in three distinct ways: factual errors (wrong information presented confidently), voice inconsistency (correct information that sounds nothing like your brand), and structural problems (correct information in the wrong format, missing required sections, inconsistent terminology).
The most dangerous AI content failure is confident misinformation. The detection approach: run a separate 'fact check' LLM call that reviews the generated content and flags claims that seem uncertain. This catches about 60% of factual errors.
Embed a set of 20 'reference voice examples' (content I've manually approved) and embed every new AI-generated piece. If the cosine similarity between the new piece and the average of the reference set falls below 0.75, flag for human review.
AI Content Quality Pipeline
Content Request
│
▼
┌───────────────────────────────────────────┐
│ Generation Stage │
│ - Voice calibration (2-3 examples) │
│ - System prompt with style guide │
│ - Few-shot examples (dynamic selection) │
└──────────────────┬────────────────────────┘
│
▼
┌───────────────────────────────────────────┐
│ Automated Gates (all must pass) │
│ ① Length check (500-1500 words) │
│ ② Structure check (required H2s) │
│ ③ Terminology check (exercise glossary) │
│ ④ SEO check (keyword density) │
└──────────────────┬────────────────────────┘
│
┌───────┴───────┐
│ Gates passed? │
└───────┬───────┘
Yes │ No
│ └──────────── Back to generation
▼
┌───────────────────────────────────────────┐
│ Voice Similarity Check │
│ embed(content) vs embed(reference_set) │
│ cosine_similarity < 0.75 → flag │
└──────────────────┬────────────────────────┘
│
▼
┌───────────────────────────────────────────┐
│ Human Review (structured checklist) │
│ 10-item checklist, not "does this look │
│ good?" — specific questions that force │
│ real checking │
└──────────────────┬────────────────────────┘
│
▼
Publish ✓Add a 'voice calibration' step at the start of every AI content generation session: provide the model with 2-3 examples of your best existing content and explicitly say 'Match this author's voice, vocabulary choices, and structural approach.' Refresh the calibration examples quarterly as your voice evolves.
I run every AI-generated piece through four automated checks: length check, structure check (does it have the required H2 sections?), terminology check (does it use correct terms from our standard glossary?), and SEO check (does it naturally include the target keyword 2-4 times?). These four checks catch 80% of structural failures.
For content that passes automated gates, a structured review checklist is far more effective than open-ended review. My checklist has 10 items: specific questions that force the reviewer to actually check the things that matter.
// Automated quality gate pipeline — TypeScript
interface QualityCheck {
name: string
pass: boolean
details?: string
}
async function runQualityGates(content: string, keyword: string): Promise<QualityCheck[]> {
const wordCount = content.split(/s+/).length
return [
{
name: "length",
pass: wordCount >= 500 && wordCount <= 1500,
details: `Word count: ${wordCount}`,
},
{
name: "structure",
pass: (content.match(/^##s/m) !== null) && content.includes("## "),
details: "Has required H2 sections",
},
{
name: "terminology",
pass: EXERCISE_GLOSSARY.some(term =>
content.toLowerCase().includes(term.toLowerCase())
),
details: "Uses standard exercise terminology",
},
{
name: "seo_keyword",
pass: (() => {
const count = (content.match(new RegExp(keyword, "gi")) || []).length
return count >= 2 && count <= 6
})(),
details: `Keyword '${keyword}' appears ${...} times`,
},
]
}
// Voice similarity check
async function checkVoiceSimilarity(content: string): Promise<number> {
const referenceEmbeddings = await loadReferenceVoiceEmbeddings()
const contentEmbedding = await embed(content)
const avgReference = average(referenceEmbeddings)
return cosineSimilarity(contentEmbedding, avgReference) // target > 0.75
}Fitness content has a category of sensitive topics where AI confidently generates dangerous advice. I handle this by adding a 'sensitive topic filter' to my generation pipeline: a secondary LLM call that reviews the generated content specifically for safety-critical claims.
AI content quality often degrades over time as you keep generating without updating your calibration examples. The model anchors to the patterns it sees, and if your reference examples are from 6 months ago, the generated content drifts toward that older style. I refresh my voice calibration examples every quarter.
The metrics that actually predict user satisfaction: accuracy complaint rate (aim for under 0.5% of published pieces), engagement rate, repeat visit rate, and explicit feedback rating. AI content rates 15% lower than my manual content on first publish, but after one revision cycle, they match.
AI content is excellent for: high-volume, templated content; first drafts that you rewrite significantly; and research-heavy content where the AI can synthesize information. AI content is poor for: thought leadership and personal opinion pieces, content where being provably wrong has serious consequences, and content requiring very recent information.