TradecraftEvaluation
Looking past the demo
6 min read
Most AI demonstrations are built to impress in three minutes. Here is how I pressure test them for the work that actually matters.
A good demonstration removes friction so you can see the idea. A good deployment adds friction back, because the real world is full of edge cases, governance and people.
Start with the failure modes
Before I am impressed by what a system does well, I ask what it does badly, and how often.