Asadullah (Asad) Rathore, Head of Professional Services – cyber security and AI consulting, Excite Cyber (ASX:EXT)
As Large language models (LLMs) are increasingly being embedded into software development, organisations must ensure they understand the cyber security risks that come with it. Developer tools that have generative AI capability integrated to assist developers when writing code are on the rise. But the LLMs they rely on could be altered by threat actors.
Without the right safeguards, LLMs can be manipulated by malicious actors, potentially injecting malware or misinformation into otherwise legitimate applications. In 2015, malicious actors distributed a tainted version of Apple’s Xcode developer tools and without realising developers began to create and distribute malware through the App Store.
Australian organisations must implement strict guardrails, including code validation layers, access controls, and continuous monitoring to protect their systems and ensure AI-driven development doesn’t become a new attack vector.
Organisations must put steps in place to ensure all the code they create and all outputs from generative AI tools are verified. Human oversight to ensure this is paramount. Software testing has always been important. But in the age of AI, it becomes even more critical as the developer writing your code might be an LLM that your organisations have not vetted. Outdated, incorrect or tainted data can be assimilated into a LLM just as easily as valid information.
The World Economic Forum has reported that AI-driven misinformation and disinformation are among the most significant risks facing the world. LLMs are created by scraping data from innumerable sources and can present false, misleading or even malicious information. As more LLMs are integrated into software development, organisations must understand the associated cyber security risks.
New ‘vibe coding’ is emerging where a developer provides plain language prompts to an app and a Gen AI tool writes all the code. It is a relatively small hop, step and jump now for threat actors to create coding tools that mimic legitimate tools to generate malware.
Since ChatGPT’s launch, the market has been flooded with countless generative AI tools, with many rushing to capitalise on the hype. While established companies offer some reliable options, an alarming number have emerged from shadowy sources with questionable credentials and potentially dangerous data practices.
‘Trust but verify’ is the basis of zero trust strategies where all activities are checked to ensure they are legitimate. The same principle must be applied to the use of generative AI systems and the LLMs that underpin them. While generative AI tools have the power to vastly increase productivity and enable people to move away from repetitive tasks and do higher value work, there is an increased responsibility to ensure those tools are not manipulated.
Independent and peer software reviews will become more important than ever before. In a world where applications are more interconnected, a LLM could either intentionally or accidentally connect an application through a malicious API that scrapes data or create backdoors into your systems. Services such as penetration testing, vulnerability scanning and red teaming will become increasingly critical as more software code is written by or sourced from third parties that have not earned your trust.
Generative AI and LLMs pose a new type of third-party security risk that must be acknowledged, understood and monitored. Corrupted or malicious LLMs have the potential to misinform or produce applications that steal data and leave your systems more vulnerable to attack.
