Simplifying Evidence Analysis with the Help of AI

Texas State University associate professor and ACFE Hubbard Award winner Zach Kelley was excited to talk about a partnership that connected his students with regional district attorneys — all with the help of artificial intelligence (AI). He shared more about this unique partnership in his session at the 36th Annual ACFE Global Fraud Conference, “From Images to Insights: University Students Advancing AI-Based Evidence Analysis.” 

As AI exploded in popularity and usage these past few years, Kelley determined that he wanted to quickly get an AI usage course on the schedule for students in the Department of Information Systems and Analytics. He pondered how to use his students’ AI development skills for a useful purpose. Thanks to his connections in law enforcement and the judicial system, Kelley soon discovered a specific need. 

Determining a Purpose 

A district attorney in Williamson County, Texas explained to Kelley that the state had changed the law of child pornography. Now, incident counts would determine an offender’s sentencing. In the case of child pornography, “incidents” referred to the number of pornographic photos and videos offenders possessed, so investigators previously had the painstaking task of going through disturbing images to find 500 distinct images to get a maximum charge for offenders. 

Kelley was hesitant about taking on this data, but his students came up with innovative ways to use AI to tackle the job. 

“We duplicated some of the stuff on some of the commercial packages,” said Kelley. “But what they've been able to do with AI integration put us about 20% to 22% more accurate to most of the stuff in the field.” 

Continued Kelley: “We could take images from BMPs and JPGs to anything else. If you get on your phone, dump it in a big folder and let the machine [work] on it for a while. Then when you went to the folder that had the bucketed images, it put a blurred one on there, so nobody actually had to look directly at it.” 

Kelley thought this task would take his students the length of a full semester, about four months, but they were able to get it done within four weeks. That’s when Kelley asked the Williamson County District Attorney’s Office if they needed help with anything else. 

Expanding Features 

Next, the District Attorney’s Office requested help with analyzing audio evidence. Students took recordings and transcripts of prison phone calls, which are known for being heavy with slang and secret phrasing. Large language models (LLMs) helped the students parcel certain words and phrases into categories and then analyzed them for anomalies. 

Students were now able to use technology to bucket images, read transcripts and audio, categorize everything, analyze data and create a report. Kelley shared this technology with fraud investigators who also shared it and began using it in their own cases. 

The Challenges of Using AI and How to Remedy Them 

The innovative nature of AI technology is sometimes clouded by some hiccups, but Kelley had some ideas on tweaks practitioners could make to remedy these concerns. 

You may have heard the term “AI hallucinations.” That is essentially when LLMs churn out responses with nonsensical or inaccurate information. Kelley said you can avoid these types of responses by not using what he calls “loose terminology.” 

“If you have people that know about prompt engineering well enough to talk [the LLMs] through it, you can get rid of most of the hallucinations,” said Kelley. 

There is also the issue of confidentiality and anonymity when it comes to AI tools. 

“The system was designed to maintain anonymity on information sets,” reassured Kelley. “If it's trying to parse that audio, it splits it up to two or three words at a time, pitches it so it sounds like The Chipmunks. That way, if somebody does get a hold of it and they don't know the source, they're not going to know who it was.” 

Kelley, however, cautioned everyone about using ChatGPT. He said the technology leaks and shares information. To get around this, he recommends hosting your own AI by getting a gaming computer with a good video card, loading a Lambda model and having someone attach some datasets to it. 

Future Goals 

Kelley is hoping more organizations will use the technology his students developed so the systems can be fed more data it can then learn from. 

“We have a functional analytics package that uses the language model structures to greatly speed up criminal investigations,” said Kelley. “Honestly, at this point, anything with the transcript or anything with an audio would work for them to be able to expand on.” 

Kelley is excited to welcome a new group of students to his course in an upcoming semester with the hope they can execute emotional analysis on transcripts. He said his students are open to hearing new ideas and trying to develop technology for it.