Method
How we test
A FitScore is a reviewed run of a published test, not a vibe and not a vendor's number. Until that run exists, the tool is listed as not yet tested and the score cell stays empty.
The same task for every tool
Each job has one test pack: the task, the inputs, and a time limit. The pack version is on the job page. We do not change the task because a tool is struggling, and we do not give one tool a warmer prompt.
Packs start as specified. That means the test is written and has not been run. When the inputs are frozen for a real run, the pack is marked frozen and the version is the one printed next to every score from that run.
An agent runs it. A person publishes it.
The run is executed by an agent where the product is something an agent can drive, and by a person where the product is a UI we have to operate. Either way, a person reviews the output before anything is published. The test-run file records who executed it and who reviewed it.
We publish the date, the tool version, the plan, and the test-pack version. We publish failures, not only the tools that did well. Screenshots and other evidence are attached to the run when we have them. If a field is empty, we did not record it. We do not fill it in.
Tested, research-only, or not yet tested
Tested means the pack was run and reviewed. The seven criterion scores and the FitScore are on the page.
Research-only means we have read the docs or the product page and have not run the pack. There is no score. The label is a flag, not a polite way to rank a tool we have not touched.
Not yet tested means the tool is a candidate and nothing has been published. That is the state of the whole launch catalog.
Rankings are per job
A tool can be a good coding agent and a poor notetaker. We do not publish a global "best AI" list. Order on a job page comes from FitScore, highest first. Untested tools stay on the page with a dash instead of a rank. Equal scores keep catalog order and get consecutive ranks. We will publish a tie-break before the first scored run if we need one that is fairer than that.
The scorecard
FitScore = weighted average × 20, rounded to the nearest integer. The weighted average is the sum of (criterion score × weight) divided by 100.
Each criterion is an integer from 1 to 5. Weights sum to 100. All 5s is a FitScore of 100. All 1s is 20. A missing run is not a zero. It is no score.
| Criterion | Weight | What a 5 means |
|---|---|---|
| Output quality | 30 | The task result is usable with no fixes. |
| Reliability | 15 | Three runs finish the task the same way. |
| Cost and free tier | 15 | A free tier covers the task, or the paid cost is low and obvious. |
| Speed | 10 | Finishes comfortably inside the limit. |
| Setup friction | 10 | Install, auth, and the first successful run are straightforward. |
| Agent, API, and MCP support | 10 | An agent can drive the tool through an API or MCP, or the tool is itself that path. |
| Privacy and data | 10 | We can say what left the machine, and it was no more than the task required. |
The 1–5 scale
- 5 Excellent. No fixes.
- 4 Good. One minor fix.
- 3 Usable. Several fixes.
- 2 Poor. Partial result.
- 1 Fails. Does not complete the task.
Setup friction is scored so that 5 is easy and 1 is "we could not get it set up". Privacy is scored so that 5 means we could see what left the machine and it was no more than the task needed. Agent support is 5 when an agent can drive the tool through an API or MCP, or when the product is itself that path.
Corrections
When a published result changes, the change is logged with a date. We do not quietly replace a number. Vendors can ask for a retest. We rerun the same pack. Sponsors do not get a preview.
No corrections published yet.