Structured Analytical Techniques and Large Language Models for Decision Support

• In Event: Intelligence in Democracies: AI, Film, and the Smog of War

Abstract

Large Language Models (LLMs) have captured the attention of the government, businesses and individual alike. LLMs are a branch of Artificial Intelligence (AI) research that generate human like responses to prompts based on very large datasets. These models can perform a number of remarkable tasks such as passing the legal and medical exams, writing computer code, and language proficiency tests. Customized LLMs are employed to simulate human activity in psychology, health care, and business (customer service). In this paper we propose training and employing a LLM for a complicated human task: Intelligence analysis.

We will present the results of our experimentation with the Large Language Model (LLM) ChatGPT as an analytical engine. We find it has the native ability to conduct Structured Analytical Techniques (SATs) as detailed in Heuer’s book Psychology of Intelligence Analysis as it was included in the training data. We develop a custom version of trained on proprietary materials used by intelligence analysts. We then provide an intelligence question and a data set and ask the LLM to perform the SAT using prompts. We compare the results with the same SAT training and process for two student groups. Our research provides additional data for the broad study of custom analytical LLMs providing relevant analytical functionality to support human analysis.

Authors

• Christopher Linebarger, University of Massachusetts Lowell

• John Borek, University of New Hampshire

• Bridget Rose Nolan, University of New Hampshire

• Jennifer Rose, Wesleyan University

• Taylor Nydam, University of New Hampshire

• Janet Baker

• Mark Bailey, National Intelligence University

• Andrew Macpherson, University of New Hampshire

• Frederic Baron, National Intelligence University