Cybercriminals are using advanced tools to attack systems that lack human oversight. The virtual corridors down which cybercriminals lurk to launch their attacks have become more complex, darker, and more sinister. While some of the old types of attacks remain, they have now taken on a more sophisticated and harder-to-detect complexion thanks to the use of AI, despite the same technology being brought to bear by enterprises. Attackers target customer relationship management (CRM), human resources (HR), and financial accounting platforms. They exploit automation where algorithms cannot detect anomalies or flag issues for human intervention. The defensive environment is shifting as a result, forcing a reversal that requires more human involvement in automated processes like payments and money transfers.
Automating Fraud with Artificial Intelligence
Know-your-customer (KYC) processes are a primary target. Automation is meant to make these processes faster and easier for staff, but it creates verification blind spots. Attackers use generative AI to fool these systems. They build “Frankenstein Identities” by combining stolen identity details with made-up information. These fake identities allow attackers to open accounts and build credit ratings over time.
Institutions often do not notice the fraud until after the attackers access loans or credit cards and disappear. Deep fakes – in the form of photographs, audio, and videos – are becoming fairly common ways to subvert KYC systems, especially where staff must physically verify account holder details via phone or video call. Fraudsters also generate credible-looking documents, such as title deeds, invoices, and identity documents, to bypass automated approval systems.
Attacking the Algorithms
Adversarial AI refers to the weaponization of machine learning systems. Attackers manipulate the data used to train these models rather than breaching firewalls or delivering malware. Data poisoning is a common technique where bad data is injected into a model. This causes the system to overlook false credentials, approve fake documents, and generally allow a process to proceed to the next stage. Model inversion tricks a model into leaking sensitive training data, which attackers then use to devise workarounds. Prompt injection is another vector where attackers use crafted text inputs to bypass safeguards or extract personally identifiable information.
Putting Humans Back into the Loop
Frank Herbert, in his 1965 book, Dune, wrote that “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” In many ways, what was once considered speculative in 1965, can now be said to be rather prescient.
The same AI that is being used to secure information systems, identities, and data is also being used to steal data, impersonate individuals, sabotage critical systems, and commit fraud. The continued digitization of systems and the resultant adoption of AI to secure these systems was meant to reduce the cognitive and administrative load on humans. However, this has now created an over-dependence on what can be said to be a single point of failure.
Africa ICT Analyst, Connecting Africa, is the author of this report. Francis currently works independently undertaking ICT research and consulting projects across Africa. Immediately prior to this, between 2007 and 2013, and again between 2017 and 2022, he headed the International Data Corporation (IDC) East Africa regional office located in Nairobi, Kenya. Between September 2013 and February 2015, Francis was the GSM Association’s (GSMA’s) Director for Africa, based at its sub-Saharan Africa office in Nairobi.
Francis has an extensive research background that spans more than 20 countries and experience of over 25 years. He has managed or been actively involved in numerous projects across Africa and has participated at numerous ICT events as a speaker, moderator, and content creator. Francis’ background is in journalism. He is also a published author of a children’s book and regularly contributes to various online ICT publications.
Leave a Reply