3 days ago
3 AI Adoption Metrics That Really Matter
Coworkers using AI together in the office.
'Using AI effectively is now a fundamental expectation of everyone at Shopify,' Tobi Lütke, CEO of Shopify, said in his now viral AI memo on April 7th. But Shopify isn't just encouraging employees to use AI. They are measuring employee AI usage in performance reviews as well. 'We will add AI usage questions to our performance and peer review questionnaire,' Lütke wrote. Pushing for AI adoption in companies isn't brand new, although measuring employee performance based on AI usage is relatively new, and not universally defined yet.
Defining how to measure AI usage in a meaningful way is critical to avoid AI performance theater in an era where leaders from startups to Fortune 500 companies are trying to accelerate AI adoption. In Fall 2024, McKinsey surveyed a group of 238 executives, and 46% said that one of the biggest challenges in employee AI adoption is dealing with AI skill gaps. Many want to understand what that means in terms of the outcomes in their companies, and how to power them with AI. Setting performance and usage goals around AI is a balance between understanding AI's capabilities and defining metrics that aren't performative.
First, organizations need to define their goals and get a clearer picture of why they want their employees to adopt AI. Consulting companies such as McKinsey and PricewaterhouseCoopers have estimated productivity gains in the trillions based on use cases they evaluated, with PwC estimating up to '14% higher in 2030 as a result of AI – the equivalent of an additional $15.7 trillion.' But what this number means, how real it is, and how to get there is still pretty unclear. Defining how to get meaningful productivity increases from AI is something each company needs to define for themselves. A big thing to consider is that productivity gains from using AI don't show up as immediate ROI on the balance sheet. Being clear on the biggest area of impact AI can have on your business, and building goals around those is a good starting point to create expectations around using AI.
Once expectations are set, defining how to measure employee AI usage is next. Stephen Weber, a Director of Engineering at an AI narrative intelligence startup, uses AI frequently and manages a team of engineers. He shared with me during an interview that he thinks about performance with AI through a lens of outcomes, process, and improvement over time.
'If someone is using AI to finish tasks faster without sacrificing quality, that's a good sign. If they're solving harder problems, automating repetitive tasks, or generating new ideas with AI, that's another strong signal,'he said.
The double-edge of this AI sword is that some performance metrics can easily be manipulated. These performance metrics don't create value for the business or its customers. Weber explained that 'measuring output volume alone, like how many emails or reports they generate with AI.'
Those kinds of performance metrics also can push employees to be overreliant on the use of AI, which according to joint research between Microsoft and Carnegie Melon University, 'can inhibit critical engagement with work and can potentially lead to long-term overreliance on the tool and diminished skill for independent problem-solving.'
'These numbers don't show whether the AI is being used well. They just show activity, not value,' Weber agreed.
It's crucial to define expectations around AI usage that reflect its true capabilities while encouraging employees to push the limits of AI in their role. Reddit user Kevinlevin-11 complained that stakeholders in their organization had 'ridiculous' demands for using AI in software testing: 'One person says they don't want to maintain any test cases or code, and AI should be able to write, execute, and fix any failures on its own. Another person wants AI to decide on its own [how to test all business scenarios]
.' It's important to have realistic expectations that all software requires some level of human oversight and occasional troubleshooting.
Beyond understanding good and bad ways to measure AI usage, it's important to remember you can't always tell who is using AI, how much, or how they are using it without speaking to employees directly.
When your organization is creating standards around AI adoption and usage, consider these heuristics.
Good AI usage heuristics are:
AI usage anti-heuristics are:
As your organization considers measuring AI usage as part of employee performance, envisioning the desired value is foundational. Define how to measure it, ensure it's obtainable, mold it to fit different disciplines, and be specific while leaving room for creativity. Be intentional with AI performance measurements, because you will get exactly what you ask for, good or bad.
