Performance testing is still incredibly important. It’s how we determine improvements from generation to generation, and it’s how we discover there might be issues with products.

Someone with a Snapdragon 865 probably didn’t receive a better phone overall by “upgrading” to an 8 gen 1. Someone with an iPhone 13 might not understand an iPhone 15 doesn’t bring battery improvements, and might throttle more in heavy lifting tasks.

The problem is, all our chips now are in a race to score well for synthetic benchmarks. Those scores rarely help us predict performance in real world apps. The design of the 888 and 8 gen 1 seemed tailored to delivering a big number score in a synth bench, at the expense of battery life and sustained performance. Consumers don’t user their phones by randomly testing a batch of little pieces of apps and math.

We would mock a laptop or desktop or gaming system review if all they had to offer was a pair of synth scores.

The issue is, phones aren’t ONE thing. Performance testing for a communicator phone matters more that we look at performance per watt and battery longevity. Performance testing for a gaming phone should lean harder on a collection of different game types and genres, and looking at sustained game play. A productivity phone needs a healthy balance of CPU and GPU performance to tackle laptop grade compute tasks.

We live in a time where phones are the first computers people reach for, and in many regions the only computers someone might own. We increasingly use them for tasks that required laptops and desktops, and it matters how we evaluate the claims of phone and component manufacturers.

The problem is, we’ve bought into the “average consumer” narrative which only describes the most basic level of use from the broadest swath of users in more affluent developed nations. The next billion Internet users won’t have the luxury of owning multiple computers to do individual tasks. They’ll own a phone. Probably only a phone.

We need more folks who will test devices in a variety of tasks, and will grade more than the performance of one or two games, to help consumers make more informed purchasing decisions. That takes a lot more work, which does not pay well on YouTube.

Running geekbench and antutu is insufficient to educate consumers, and aggregate scores will often misrepresent the actual daily performance aspects those consumers are likely to care about most.