The letters ‘AI,’ are only a measly two syllables, yet become bundled with a bevy of trepidations and an equal amount of hype. It has made diagnosticixans and even prophets out of researchers. Helping us separate the artificial from the intelligence was Ray Perrault of the Stanford Institute for Human-Centered Artificial Intelligence (AI). 

Perrault spoke about the Stanford AI Index Report on the last day of the Absolutely Interdisciplinary conference on May 8. Coming in at just over 500 pages, the Stanford AI Index Report is a behemoth that documents the recent trends in AI. The Stanford Institute’s mission is to produce research that is rigorously vetted, broadly sourced and quantitative. People talk, but numbers talk louder.

The report was put together in 2023 and published on April 15. Given how fast AI develops, maybe my human eyes prevented me from catching some developments since then. So, when you’re reading this, if you find yourself thinking that surely the figures must’ve changed, I encourage you to find out. Now that we have our bearings, here are the key insights from the report that Perrault discussed.

AI performance on benchmarks

To measure the technical capabilities of an AI model, researchers have come up with clever benchmarks that are essentially pop quizzes for AI. Researchers grade the performance of AI models on these different benchmarks.

While it is true that AI outperforms humans on many benchmarks, it is still lagging on others, like competition-level mathematics or visual common sense reasoning. ‘Visual common sense reasoning’ refers to the ability to understand the relationships between different objects in an image. 

Perrault explained that researchers introduce a benchmark and, after a couple of years, AI models catch up to human performance on that benchmark: their progress reaches saturation. Therefore, researchers are always coming up with new benchmarks and testing new capabilities of AI. So the Sisyphean struggle of AI development carries on. 

Previously, AI could only answer prompts in one medium at a time — through text or image. As new benchmarks have shown, AI can process different mediums simultaneously. The researchers tested this through multiple choice questions in different subjects involving both text and diagrams. The performance of humans on this test was around 80 per cent on average while the best AI models give correct answers around 60 per cent of the time. The models are ahead in some benchmarks and are slowly catching up in others. 

Corporate investment in AI

Corporate investment in every industry has been shrinking since 2021, so investment in AI has been naturally shrinking too. However, the percentage investment in generative AI compared to other types of AI has been on the rise. The common public sentiment against AI can be distilled into one pressing question: “Is AI going to steal my job?” 

The news is not all bleak though, as the number of open or publicly available AI models are growing. These models are made to be transparent: you can pry them open, see the inner workings, and tweak them

 

Is AI the bullet in the chamber that kills job security? Perrault weighed in.

A report from consulting firm McKinsey & Company has found that while businesses’ AI adoption shot up from 2017 to 2019, it has remained stable since then. Despite all the attention that ChatGPT and other AI models garnered, behind the smoke and the mirrors, these industries didn’t immediately translate to workflow changes. Of course, the incentive to use AI in the workforce is very much still present. The report reveals that adopting AI in businesses can reduce costs and boost revenue.

But, for now, AI is not going to take your job.

Research and development of AI

For those weary of the monopoly in the AI industry, there’s dire news: most large tech companies have monopolies in emerging AI technology. Google is producing the largest number of AI models, with 40 models since 2019. Most AI models are coming from private companies and not from academic campuses. The private technology companies developing AI are also low on transparency and developing the frontier models of new AI is getting more expensive — with the largest ones costing well over 100 million USD.

The news is not all bleak though, as the number of open or publicly available AI models are growing. These models are made to be transparent: you can pry them open, see the inner workings, and tweak them. Open models allow other developers to learn and build on what already exists. However, the fact remains that closed models like those from private companies often outperform open models.

Governments are starting to heed the warnings that dystopian writers have been spouting for decades. We’re thankfully still far removed from the scorched-earth, post-apocalyptic world of Terminator, but the rapid march of AI development has raised eyebrows. Policymakers have identified a need to regulate AI to allow our infrastructure to catch up to it. In 2023, the US proposed 181 bills which aimed to regulate AI and most of the proposed regulations have been about constraining rather than expanding the use of AI.

AI continues to develop at the speed of a jet-plane and we are left watching the contrails behind. The Stanford AI Index Report gives us a peek into the direction we’re headed. With government policies, corporate investment, and human skepticism nudging the future of AI, Perrault’s report demystified some of these forces and prepared us as we wait for the contrails to dissipate into the atmosphere of our society. Hopefully, now we are a little more ready for an impending sonic boom.