About this project

Who made this

I am Francisco Perdigon Romero, AI Technology Lead & Founder, based in Montreal. I run Core-AI, a practice that builds private, self-hosted generative-AI systems for small law and engineering firms — the kind of work where the model has to run on hardware the client owns, not on someone else’s API. Day to day I am AI Lead and Data Scientist at Ericsson, working on agentic AI for 5G and LTE networks. Before industry I spent years as a deep learning researcher at École Polytechnique de Montréal, on medical image analysis and ECG signal processing, and I still maintain DeepFilter, an open-source ECG baseline-wander removal tool.

• LinkedIn
• GitHub
• Core-AI

Choosing hardware for a private deployment is a decision I make for real, with a client’s budget. What is usually missing is not the benchmark — it is knowing who ran it, when, on what settings, and whether anyone else got the same number.

Where this came from

This project started with Alex Ziskind’s video What Computer Should You Buy for Local AI, on his channel. In it he turns years of his own local-AI benchmarking into a small web app that answers the question in its title, hosted at ai-hardware-bench.replit.app.

The question is the right one. Its numbers, though, arrive with no source behind them, no date attached, and no way to tell a measurement someone ran from a figure filled in to complete the table.

How this one is different

Every number names its sources. A record without at least one source URL, with an author and a date, does not enter the dataset.

Confidence is computed, not claimed. Two or more independent sources agreeing within 25% makes a row verified. One source makes it single-source. Sources that disagree beyond that make it disputed, and the disagreement is shown rather than averaged away. Independence is keyed on the author, so one person cross-posting the same run to three forums counts once.

Dates are first-class. Local inference gets faster on hardware that has not changed, because the backends improve. Every measurement here shows the date it was taken, sorts by it, and sources more than 270 days apart are kept as separate records instead of being blended into one.

Estimates are labelled and bounded. Where nobody has measured a combination, the site can predict one from memory bandwidth — but that row is generated, wears an estimated badge, is never hand-authored, and is only produced inside the range where the underlying fit has real data behind it. Outside that range it says nothing. The Arena takes the same line: it will not declare a winner on a speed row unless both machines were actually measured.

It is data reviewed like code. The whole dataset is YAML in a repository. A schema, a validator and a test suite run over every change, and every published figure is recomputed from the sources at build time.

Where the numbers come from

From the community, in the places people actually post their runs: 261 source citations across 17 sites, from 115 distinct contributors, behind 197 measured records.

Those sites are hardware forums (Level1Techs, the Framework community, the NVIDIA developer forums, Radxa), GitHub issues and discussions on the inference backends themselves, Hugging Face, and independent write-ups by people who benchmark this hardware repeatedly and show their settings.

Collecting from the open web means most of the work is rejection. Benchmark numbers on the modern web include AI-generated articles inventing models that do not exist, content mills restating each other, “calculator” pages that quietly compute a figure and present it as a measurement, and spam repositories on GitHub.

The method, in short

Read a source, extract the run and its conditions (model, quantisation, backend, context length, batch size, offload), attribute it to an author and a date, and store it. Confidence falls out of how many independent sources agree and by how much. Every figure is checked against a physical ceiling: a decode rate cannot exceed what the machine’s memory bandwidth allows.

The full version, including what this dataset cannot tell you and the sources that were rejected and why, is on the methodology page.

Credit and independence

The framing of the advisor comes directly from Alex Ziskind’s video. This project is not affiliated with him, his channel or the Replit app; it is not endorsed by them; and no data has been copied from that app. Everything here was collected and re-sourced independently.

Corrections and contributions

Found a wrong number, a stale source, or a hardware spec that has changed? Open a correction — corrections are the most valuable thing anyone sends. Nine records have already been removed after re-verification showed they were fabricated or misattributed, and finding another is a service.

Know of a measurement that should be here, or ran one yourself? Submit a benchmark. 115 people have already contributed a source, almost all of them without ever being asked — they published a run and it was cited.

The form asks for the source URL, its author and date, and the quantisation, context length, backend, batch size and offload, because a record missing any of those cannot be published: a tok/s figure without its conditions is not comparable to anything else here. Most submissions are still rejected, and that is the intended outcome rather than a judgement of the sender.

The tracker carries no code and no dataset — it exists so the numbers can be argued with in public. Questions and methodology arguments belong in its discussions.