Performance in ImageNet is measured with metrics such as “top 1 accuracy” and “top 5 accuracy.” An image classifier gets a 0.98 score on “top 5 accuracy” if its five highest predictions include the right label on 98 percent of the test photos in ImageNet. ImageNet contains millions of images labeled for more than a thousand categories. An example is ImageNet, a popular benchmark for evaluating image classification systems. Benchmarks for specific tasksīenchmarks are datasets composed of tests and metrics to measure the performance of AI systems on specific tasks. “We do not deny the utility of such benchmarks, but rather hope to point to the risks inherent in their framing,” the researchers write. The scientists warn that progress on benchmarks is often used to make claims of progress toward general areas of intelligence, which is far beyond the tasks these benchmarks are designed for. In a paper accepted at the NeurIPS 2021 conference, scientists at University of California, Berkeley, University of Washington, and Google outline the limits of popular AI benchmarks. But while benchmarks can help compare the performance of AI systems on specific problems, they are often taken out of context, sometimes to harmful results. Especially in the past few years, with deep learning becoming very popular, benchmarks have become a narrow focus for many research labs and scientists.
BENCHMARK AI SERIES
This article is part of our reviews of AI research papers, a series of posts that explore the latest findings in artificial intelligence.įor decades, researchers have used benchmarks to measure progress in different areas of artificial intelligence such as vision and language.
0 kommentar(er)
