Tool Field Test evaluates AI products through predeclared, reproducible workflows designed around real buyer decisions.
1. Define the job
Each comparison starts with a specific audience, job to be done, and fixed test scenarios. Inputs, expected answers, time limits, and pass/fail checks are set before products are run.
2. Use controlled evidence
We use synthetic or appropriately licensed data, preserve fixture versions and hashes, and test corrections, negation, missing information, dates, numbers, citations, noise, overlap, exports, and deletion behavior when relevant.
3. Repeat core tests
Core scenarios are normally run three times without silently repairing prompts. Invalid setup attempts are logged as friction but excluded from scored results.
4. Score observable outcomes
We score task completion, reliability, setup and handling time, effective cost, portability, privacy and administration, and documentation. Confident invention is penalized more heavily than omission.
5. Label evidence honestly
Every dossier is labelled Executed, Partially executed, Desk reviewed, or Blocked. Marketing claims are not treated as hands-on findings.
6. Audit before publication
Draft comparisons undergo a claims audit and human editorial review. Mutable prices and policies are recaptured immediately before publication.
