<
https://www.theregister.com/2025/04/12/ai_code_suggestions_sabotage_supply_chain/>
'The rise of LLM-powered code generation tools is reshaping how developers
write software - and introducing new risks to the software supply chain in the
process.
These AI coding assistants, like large language models in general, have a habit
of hallucinating. They suggest code that incorporates software packages that
don't exist.
As we noted in March and September last year, security and academic researchers
have found that AI code assistants invent package names. In a recent study,
researchers found that about 5.2 percent of package suggestions from commercial
models didn't exist, compared to 21.7 percent from open source models.
Running that code should result in an error when importing a non-existent
package. But miscreants have realized that they can hijack the hallucination
for their own benefit.
All that's required is to create a malicious software package under a
hallucinated package name and then upload the bad package to a package registry
or index like PyPI or npm for distribution. Thereafter, when an AI code
assistant re-hallucinates the co-opted name, the process of installing
dependencies and executing the code will run the malware.
The recurrence appears to follow a bimodal pattern - some hallucinated names
show up repeatedly when prompts are re-run, while others vanish entirely -
suggesting certain prompts reliably produce the same phantom packages.
As noted by security firm Socket recently, the academic researchers who
explored the subject last year found that re-running the same
hallucination-triggering prompt ten times resulted in 43 percent of
hallucinated packages being repeated every time and 39 percent never
reappearing.
Exploiting hallucinated package names represents a form of typosquatting, where
variations or misspellings of common terms are used to dupe people. Seth
Michael Larson, security developer-in-residence at the Python Software
Foundation, has dubbed it "slopsquatting" – "slop" being a common pejorative
for AI model output.
"We're in the very early days looking at this problem from an ecosystem level,"
Larson told
The Register. "It's difficult, and likely impossible, to quantify
how many attempted installs are happening because of LLM hallucinations without
more transparency from LLM providers. Users of LLM generated code, packages,
and information should be double-checking LLM outputs against reality before
putting any of that information into operation, otherwise there can be
real-world consequences."'
Cheers,
*** Xanni ***
--
mailto:xanni@xanadu.net Andrew Pam
http://xanadu.com.au/ Chief Scientist, Xanadu
https://glasswings.com.au/ Partner, Glass Wings
https://sericyb.com.au/ Manager, Serious Cybernetics
