Key Takeaways
- A study posted to arXiv on November 10, 2026 by Carnegie Mellon and JetBrains researchers linked five weeks of coding-agent token use to the commits of 404 JetBrains engineers, and surveyed 412 employees.
- Within each weekly-commit quintile, the 90th-percentile engineer-week used 39 to 240 times the tokens of the 10th percentile. Engineers with similar output had very different spend.
- Within an engineer, doubling token spend was associated with about 10% more commits. Commits, lines changed, bug-linked commits and commit-days explained 38% of spend differences between engineers and 13% of week-to-week changes.
- 89% of technical staff's coding-agent evaluations named at least one non-coding use, such as data analysis (52%) or brainstorming (49%).
- The authors' conclusion, published November 10, 2026: "We recommend using token spend at most for resource accounting," and companies "should not evaluate engineers' output or impact by their spend."
The AI token spend productivity study from Carnegie Mellon and JetBrains finds that how many tokens an engineer burns says little about how much they ship. It links agent telemetry to individual engineers' commits, and it arrives as employers argue over "tokenmaxxing." This post sets its numbers against the AI-use metrics employers have named and turns them into what to say in your next review.
What happened
On November 10, 2026, Shyam Agarwal, Olga Lvova, Agnia Sergeyuk, Anastasiia Serova, Timofey Bryksin, Thomas Zimmermann and Bogdan Vasilescu posted "The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures". JetBrains, which had about 2,600 employees, logs every coding-agent call through an internal gateway. The researchers matched five weeks of those logs, June 29 to August 2, 2026, to the commits of 404 engineers on a main IDE product, and added an anonymous survey of 412 employees.
The title gives the argument away. It borrows the old argument against counting lines of code and asks whether token spend is the same kind of number. The paper's answer: yes, if you use it to judge people.
The details that matter to engineers
Spend follows the person, not the work
Two agents carried nearly all the tokens, with a 69% and 31% split. Most of those tokens are re-sent context: "the median call sends about 400 times as many tokens as it receives," the authors write. That alone makes token counts sensitive to how an engineer works, how long their sessions run and which model they pick.
The spread is wide. Over the five weeks, the engineer at the 90th percentile used 71 times as many tokens as the one at the 10th. Holding output roughly constant doesn't close the gap: within each weekly-commit quintile, the 90th-percentile spend was 39 to 240 times the 10th. Even after accounting for commits, lines changed, bug-linked commits and commit-days, the 90th-percentile person-week still used 58 times the 10th.
Who the engineer is accounts for 79% of the deviance in spend. Role, manager status and team predicted held-out engineers' spend worse than simply using the weekly average.
More spend, a little more output
Spend and output do move together, just not one-to-one. "Within an engineer, doubling spend was associated with about 10% more commits," the paper says. Run the other way, a week with twice an engineer's usual commits used about 20% more tokens. The authors stress that "all our estimates are associations," not causal effects, since engineers chose how much to use the agents.
Some spend isn't tied to commits at all. "A week with no commit still costs 77% of the engineer's average week." The authors' preferred estimates put the share of tokens not tied to output at 39% to 42%. It was 59% for interns and 50% for QA engineers. When a new model shipped partway through the study, in its fourth analysis week, it took 45% of spend within two weeks.
Where the tokens go
The survey explains a lot of the gap. Coding agents get used for much more than code.
Respondents who used agents for two or more non-coding activities reported a median of 10 hours saved in the past week, against 3 hours for coding only. In 91% of the coding-agent evaluations, respondents reported some time saved, and 88% rated the effect on their overall productivity as positive. None of that shows up as a commit. The survey isn't linked to the telemetry, and the hours are self-reported.
The costs were concrete too: wrong or low-quality output (49%), tooling friction (24%), prompting and steering (21%) and verifying output (18%). And 55% of 299 respondents said they handled colleagues' material they suspected was AI-generated, at a mean of 1.6 hours a week.
AI token spend productivity study vs how employers count AI use
Several employers have already said how they count AI use. The table sets each common metric against what the JetBrains data says about it. The employer column comes from our earlier reporting; the study column is our reading of the paper.
| Metric | Where it has come up | What the JetBrains data shows | Verdict for reviews |
|---|---|---|---|
| Raw token spend or usage leaderboards | Amazon's October 2026 engineering memo names token usage as an input metric and says "tokenmaxxing is at odds with frugality." Microsoft's Jay Parikh wrote in an August email, as reported by 404 Media and The Register: "Tokenmaxxing is not what we are optimizing for." | 39–240x spread at similar weekly commits. Doubling spend goes with about 10% more commits. | Don't use it to compare people |
| Tokens per commit or per PR | Not named in the employer statements above. The paper tests it directly | The ratio falls as commits rise. Only 17 of 34 top-decile engineers in odd weeks stayed top-decile in even weeks. It shifts with model releases. | Unstable at the top. Don't rank on it |
| Adoption: who uses AI tools, and how often | Amazon's memo looks at teams at the 10th and 90th percentiles of AI adoption. | The authors suggest the share of engineers using agents on most working days, reported weekly, instead of token thresholds. | Usable for adoption, not for output |
| Task and outcome evidence | Amazon's manager bar: describe "a task where they helped and a task where they did not." | The authors recommend recording which task each agent session served and measuring what the work produces. | The defensible measure |
Amazon's memo calls token and code-volume metrics "deeply flawed" but still allows them "with care and judgement." The JetBrains paper is the first data we've seen on how flawed they are at the individual level.
What the study can't tell you
It's one company, and JetBrains builds AI coding tools, so some metered use may be product testing. Agent use there was voluntary, with no targets, which is exactly the condition where people don't game the number. Commits miss code quality, review and design work, and pull requests weren't available. Five weeks is short. Tokens aren't billed cost, and engineers' time isn't in the data. The survey had a 15.8% response rate.
The authors say spend still has a use: budgeting. Five-week totals ranked engineers' spend reliably (0.87), while single weeks rank-correlated at only 0.4 to 0.6. For context on why that matters, JetBrains said in August 2026 that its development-related AI spending rose roughly tenfold in six months, as InfoQ reported.
What this means for you
If your employer tracks AI usage
- Keep a task log, not a token log. For each agent session that mattered, note the task, the outcome and whether the work would have happened otherwise. The study's own advice to companies is to record which task each session served.
- Count non-coding work. Data analysis, planning and research don't produce commits. If 89% of evaluations included them at JetBrains, they're probably part of your week too. Say so in your review.
- Have a "helped / didn't help" pair ready. It's Amazon's bar for managers, and it's the answer a usage number can't give.
- Don't chase the leaderboard. Doubling spend went with about 10% more commits. Burning tokens to look busy buys little measurable output.
If you manage engineers
- Drop tokens-per-commit rankings. The top of the ranking reshuffles week to week.
- Budget on multi-week totals and revisit after model releases. A new model took 45% of spend within two weeks at JetBrains.
- Cut cost per call before capping people. The authors point to context management, caching and routing routine calls to cheaper models, and note that per-person caps mostly hit heavy users.
How this connects
Our Amazon tokenmaxxing memo post covered an employer saying token metrics are flawed. This study measures how flawed they are. It fits the pattern in the Harvard AI coding agents study, where agents raised code volume without a matching rise in delivered work. For the bigger question of what AI means for engineering jobs, see Will AI Replace Software Engineers? What the 2026 Data Actually Says.
The same question now comes up in hiring. Interviewers ask how you use AI tools, and the answer that lands is a task and an outcome, not a usage number. Our guide to coding interviews in the age of AI covers the new formats, and Mastering the Tech Interview covers how to tell that kind of story in behavioral and coding rounds.
FAQ
Does spending more on AI tokens make engineers more productive?
Only slightly, by the measure in this study. Among 404 JetBrains engineers, doubling an engineer's token spend was associated with about 10% more commits. The authors call these associations, not causal effects, and note that commits miss review, design and non-coding work.
What is tokenmaxxing?
Tokenmaxxing is using as many AI tokens as possible to look productive on usage dashboards or leaderboards. Amazon's October 2026 engineering memo said "tokenmaxxing is at odds with frugality," and Microsoft's Jay Parikh wrote in an August 2026 email, as reported by The Register, that it "is not what we are optimizing for."
Should companies use token spend in performance reviews?
The study's authors say no: companies "should not evaluate engineers' output or impact by their spend." They recommend using token spend at most for resource accounting, tracking adoption by the share of engineers using agents on most working days, and judging work by tasks and outcomes.
What do engineers use coding agents for besides coding?
In JetBrains' survey, technical staff named processing and analyzing data (52%), brainstorming (49%), research (44%), planning (39%), learning (28%) and writing (23%). 89% of coding-agent evaluations included at least one non-coding use.
Further Reading
- Agarwal et al., "The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures": the paper (arXiv, November 10, 2026), with full methods and supplementary results.
- Amazon, "An update from Amazon engineering leadership on the role of the developer in the age of AI": the October 2026 memo on token and code-volume metrics.
- The Register, "Microsoft tells engineers to curb their token-burning enthusiasm": Jay Parikh's August 2026 email.
- InfoQ, "JetBrains Details its First Steps to Bring Rapidly Growing AI Spend under Control": JetBrains' AI spend growth and controls.
