Researchers have discovered that as businesses increasingly adopt AI for code generation, human developers might lack the crucial abilities to verify and fix AI-produced code, particularly if their skill development was hindered from the outset by relying on AI.
Much excitement surrounds AI coding assistants and the significant improvements they offer developers in terms of speed and precision. However, a question arises: are developers inadvertently delegating cognitive processes to AI when utilizing these tools as co-pilots?
This phenomenon was recently investigated by Anthropic researchers. They conducted an experiment to observe the rate at which software developers acquired a novel proficiency (mastering a new Python library) both with and without the aid of AI. Crucially, they also sought to establish if AI usage diminished developers’ genuine comprehension of the code they had just generated.
Their findings revealed a paradox: while developers using AI successfully completed unfamiliar tasks, they simultaneously failed to develop new skills.
This outcome doesn’t come as a shock to practicing software engineers. As Wyatt Mayham from Northwest AI Consulting explained, “AI coding assistants serve not as a quick path to proficiency, but as potent instruments demanding heightened discipline.”
Developers relying on AI demonstrated significantly lower understanding of coding principles.
During a randomized, controlled study, 52 “predominantly junior” developers were divided into two cohorts: one encouraged to leverage AI, the other prohibited from doing so. Both groups undertook a brief exercise involving the relatively modern asynchronous Python Trio library, which introduced concepts beyond mere Python proficiency. The participating engineers were adept in Python and familiar with AI coding assistants but had no prior experience with the Trio library.
The findings were conclusive: the cohort utilizing AI achieved a score 17 percentage points lower on the assessment than the manually coding control group – specifically, 50% versus 67%, which is roughly equivalent to a two-grade difference. This deficiency occurred even though the quiz addressed concepts they had engaged with only moments earlier.
Significantly, the most pronounced deficits in proficiency were observed in debugging code and understanding why and when code malfunctions. This presents a serious concern, as it implies that humans might lack the essential capabilities to scrutinize and troubleshoot AI-generated code, especially “if their initial skill development was hampered by early reliance on AI,” as the study authors highlighted.
Detailed overview of the study
The 70-minute experimental session was structured as a self-paced tutorial. Participants were given a problem statement, initial code, and a concise explanation of the Trio principles needed for its resolution. They were allotted 10 minutes for tool familiarization, followed by 35 minutes to implement two distinct features using Trio. The final 25 minutes were allocated for an assessment.
Participants were prompted to complete their work efficiently on an online coding platform. The AI group had access to an integrated sidebar AI assistant capable of modifying code and generating correct solutions upon request. Researchers recorded screen activity to monitor the time spent on coding or query composition, the nature of questions posed, and the errors encountered by participants.
Intriguingly, AI utilization did not inherently lead to a poorer score; instead, the critical factor influencing skill and concept retention was how developers engaged with the AI.
Members of the AI cohort dedicated as much as 30% of their allocated time (11 minutes) to crafting up to 15 queries. Conversely, the non-AI group encountered a higher number of errors, primarily related to syntax and Trio concepts, compared to their AI-assisted counterparts. Nevertheless, the researchers theorized that by independently resolving these errors, the non-AI group “likely enhanced their debugging capabilities.”
Participants from the AI group were categorized according to their extent and approach to AI use. Individuals scoring below 40% on the quiz demonstrated significant reliance on AI, exhibiting “reduced independent thought and increased cognitive offloading.” This segment was further subdivided into:
- AI delegators: These developers showed “complete dependence” on AI, finishing tasks most rapidly and encountering minimal or no errors;
- ‘Progressive’ AI users: Initially proactive with some questions, they eventually transitioned to total reliance on AI;
- Iterative AI debuggers: While starting with more inquiries, they ultimately entrusted AI with debugging and validating their code, rather than deepening their own comprehension.
The alternative user category, comprising those with quiz scores of 65% or higher, utilized AI for both code generation and conceptual questions. This group was subsequently divided into:
- Participants who generated code, manually integrated it into their workflows, and then posed subsequent questions. These individuals ultimately displayed a “superior level of understanding” on the assessment.
- Participants who formulated “hybrid queries” requesting both code and accompanying explanations. While often more time-consuming, this approach enhanced their comprehension.
- Participants who posed conceptual questions and then leveraged their own understanding to complete the assignment. They faced “numerous errors” throughout the process but successfully resolved them independently.
“The crucial factor is not if a developer employs AI, but rather the manner in which it is used,” Mayham asserted, noting these conclusions resonate with his personal observations. “Developers who prevented skill erosion were those who actively engaged their intellect instead of merely adopting the AI’s suggestions.”
Notably, developers participating in the study were conscious of their own approaches. Non-AI users described the task as “enjoyable” and reported gaining an understanding of Trio, whereas AI-using participants expressed regret, wishing they had focused more on the specifics of the Trio library, either through reviewing generated code or requesting more detailed clarifications.
“To be precise, [AI-using] participants confessed to feeling ‘lazy’ and acknowledged that ‘significant gaps persist in (their) comprehension,’” the researchers elaborated.
Strategies for developers to continuously sharpen their expertise
Numerous studies, including those by Anthropic, have demonstrated that AI can accelerate certain tasks by up to 80%. However, this latest investigation suggests that such speed sometimes comes without a corresponding increase in quality. The researchers highlighted that junior developers, feeling compelled to work at maximum pace, are jeopardizing their own skill growth.
“Productivity boosted by AI is not a bypass to true proficiency,” they stated, adding that the “rapid” integration of AI into professional environments can adversely affect employees who fail to maintain cognitive engagement. The researchers underscored that humans continue to require the expertise to identify AI errors, direct its output, and ensure proper supervision.
“Mental exertion — and even the experience of struggling deeply — is crucial for cultivating expertise,” they asserted.
The researchers recommended that managers adopt a “deliberate” approach when implementing AI tools, ensuring that engineers can sustain their learning trajectories while working. Prominent LLM providers offer supportive learning environments, like Anthropic’s Claude Code Learning and Explanatory modes, or OpenAI’s ChatGPT Study Mode.
According to Mayham, developers can counteract skill degradation by:
- Utilizing AI as a pedagogical instrument: Request both code and its explanations. Pose conceptual questions. He recommended, “Employ it to grasp the ‘rationale’ behind the code, rather than solely the ‘solution.’”
- Validating and refining: “Never accept AI-produced code without scrutiny.” Always dedicate time to reviewing, comprehending, and testing it. Frequently, the most profound learning arises from debugging or enhancing AI-supplied code.
- Preserving autonomous thinking: Employ AI to enhance workflow, rather than to supersede the cognitive process. Mayham stated, “The objective is to retain the role of solution architect, with AI serving as a remarkably efficient aid.”
He observed that AI-powered productivity cannot replace “authentic proficiency,” particularly in critical, safety-sensitive systems. Developers are required to be deliberate and disciplined in their tool adoption, ensuring continuous skill development “instead of degradation.” Successful developers will not merely delegate tasks to AI; they will leverage it to formulate superior inquiries, investigate novel concepts, and push the boundaries of their comprehension.
“While the danger of skill decay is undeniable, it is not predetermined. It is a decision,” Mayham concluded. “The developers destined to excel are those who regard AI as a Socratic partner in their learning journey, rather than an opaque system for task offloading.”
