webbenchGitHub

8 web data APIs for agents, graded on whether the job got done

Every provider gets the same tasks, the same model and the same limits. A task only counts when a script confirms every answer.

How a search task runs

The task1 of 60Latest stable releaseof fastapi on PyPI, andits install command?The modelClaude Haiku 5.5, same promptfor every providerOne toolsearch(query)API testedBrowserbaseContext.devFirecrawlLinkupTinyFishJSON answer{ "version": "0.143.3", "installCommand": "pip install fastapi"}Script checksSame as PyPI's own record?it countsup to 3 calls,60 s, $0.10
60 tasks. Answer keys come from npm, PyPI, crates.io, GitHub releases, Wikidata and SEC filings.

Tasks done every answer correct, checked by script, with the cost of each finished task

Firecrawl85%3.04¢ per task
Linkup83%2.55¢ per task
Context.dev75%1.82¢ per task
TinyFish73%1.31¢ per task
Browserbase38%5.97¢ per task
Model alone23%1.07¢ per task

The only gap big enough to rule out chance: Firecrawl finished 47 points more tasks than Browserbase. The rest can't be told apart yet.

Cost per finished task all API and model spend, failures included, divided by tasks done

TinyFish1.31¢
Context.dev1.82¢
Linkup2.55¢
Firecrawl3.04¢
Browserbase5.97¢

Time per finished task median, lower is better

Context.dev5.2 s
Linkup5.3 s
TinyFish5.3 s
Firecrawl11 s
Browserbase12 s

Raw output, no model does the API's own output contain every answer? Cost is the API alone

Linkup68%0.83¢ per task
Context.dev55%0.46¢ per task
Firecrawl47%2.40¢ per task
TinyFish47%free
Browserbase2%30.0¢ per task

5 of the 60 tasks have two defensible answers: three founding years where Wikidata differs from the year most sources give, and two releases just after midnight UTC that providers dated in US time. Dropping them, or accepting both answers, moves Firecrawl to 92% at most and changes no provider's place.