AWS Expert: Measure AI Success with Core Business Metrics, Not New AI Scores

Article Summary

Colleen Aubrey, SVP at AWS, advises IT leaders struggling with AI pilots to shift focus from creating novel AI performance metrics to improving established business outcomes. She argues that true AI ROI is measured by metrics a CFO already tracks, such as reduced inventory or faster resolution rates, rather than metrics like 'return on tokens.' Her playbook, drawn from successful AWS applications like Amazon Connect, emphasizes identifying a single, acutely painful decision point—a 'sharp wedge'—and making that process dramatically better, rather than attempting to achieve feature parity across entire workflows. Key takeaways include focusing on narrow wins, budgeting for necessary downstream infrastructure fixes, and redesigning roles around measurable outcomes rather than tasks.

Key Points

• Enterprises must tie AI project success to existing, measurable business metrics that executives already track, dismissing vanity metrics like 'return on tokens.'
• Successful AI implementation requires identifying a single, acute pain point—a 'sharp wedge'—rather than trying to replicate entire, complex workflows.
• Organizations must budget for and fix the 'plumbing' (testing, pipelines, data quality) that AI exposes, as this often dictates real velocity gains, not the AI tool itself.