New editorial: Selling Shovels in the LLM Gold Rush (JOT)

Selling Shovels in the LLM Gold Rush: Why Software Engineering Research Risks Missing the Real Transformation is out as an editorial in the Journal of Object Technology (JOT), volume 25, number 2.

The title is not decoration, and the editorial says so plainly:

During the gold rush, the most reliable profits were not made by gold seekers, but by those selling picks, shovels, and wheelbarrows. Today, the analogous risk is the proliferation of conceptual tooling (taxonomies, maturity models, evaluation frameworks) without a proportional increase in knowledge about how LLMs actually behave when used and integrated as part of real software systems.

The variables that decide how this plays out in practice are scale, cost, duration of use and organizational impact, and they are precisely the ones the literature tends to leave implicit. Research that keeps that distance ends up describing the transformation instead of shaping it.

This is not a detour from jPipe. It is the position the project rests on, and the one its PI keeps returning to: claims have to answer to evidence. An argument that never says what it asserts, on what grounds, and under what conditions those grounds stop holding is not an argument you can check, whether it appears in a research paper or in the safety documentation shipped with a model. Justification models exist to force exactly that, and the editorial is the case for why it matters at the scale of a whole research community.

Abstract

The rapid adoption of Large Language Models (LLMs) is reshaping software engineering practice at an unprecedented pace. While research on LLM-based software engineering is flourishing, much of it emphasizes frameworks, roadmaps, and conceptual models that are easy to articulate and evaluate, yet increasingly detached from the realities of sustained, large-scale use. This editorial argues that a growing gap is emerging between how LLMs are studied in software engineering research and how they are actually engineered and operated in industrial settings. Key distinctions related to scale, cost, duration of use, and organizational impact are often left implicit, leading to a focus on tactical performance improvements while overlooking the strategic transformation of software production. We discuss the risks of accompanying and proxy research, examine their implications for research credibility, industrial relevance, and education, and propose concrete steps toward greater proximity between research discourse and engineering reality. Without such proximity, software engineering research risks talking about the transformation rather than helping to shape it.

Reference

Önder Babur, Sébastien Mosser, Alfonso Pierantonio. Selling Shovels in the LLM Gold Rush: Why Software Engineering Research Risks Missing the Real Transformation. In Journal of Object Technology, Volume 25, no. 2 (2026), pp. 1-6. [JOT]