A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools

Josephine Marie Bakka; Tobias Worm Bøgedal; Andreas Kjeldgaard Brandhøj; Johannes Bjerva

doi:10.5220/0015000400004103

A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

Abstract

AI adoption has accelerated with the rise of LLMs, and people within the field of security are increasingly exploring their practical value. Threat modelling is central to secure system development, yet it remains largely manual and the value of LLM-driven tools is unclear. Even when LLMs prove useful, selecting the right one can be more challenging than using it. This paper introduces a systematic evaluation framework for LLM-driven threat modelling tools to support tool selection, observing the general LLM-integration, governance risks, and allowing for comparison of tool output. Using Goal-Question-Metric, we derive evaluation metrics and show the value of the framework on a set of state-of-the-art LLM-driven threat modelling tools. The results show our metrics distinguish both performance and governance risks, providing a basis for organisations to ensure automation strengthens rather than burdens their threat modelling process.

Original language	English
Title of host publication	Proceedings of the 23rd International Conference on Security and Cryptography
Editors	Sabrina De Capitani Di Vimercati, Pierangela Samarati
Number of pages	12
Volume	1: SECRYPT
Place of Publication	SciTePress
Publication date	Jul 2026
Pages	689-700
ISBN (Print)	978-989-758-858-7
DOIs	https://doi.org/10.5220/0015000400004103
Publication status	Published - Jul 2026
Event	23rd International Conference on Security and Cryptography: Volume 1 - Porto, Portugal, Porto, Portugal Duration: 16 Jul 2026 → 18 Jul 2026 Conference number: 23

Conference

Conference	23rd International Conference on Security and Cryptography
Number	23
Location	Porto, Portugal
Country/Territory	Portugal
City	Porto
Period	16/07/2026 → 18/07/2026

Keywords

Automated Threat Analysis
GQM Framework
Governance
LLMs
Risk Assessment
Secure Software Development
Security Evaluation Framework
Threat Modelling

Access to Document

10.5220/0015000400004103Licence: CC BY-NC-ND 4.0

AUB Link

Search for the material in Aalborg University Library's search engine

Cite this

@inproceedings{3d3a848e2c3b4c8c969352e5dc3827bd,

title = "A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools",

abstract = "AI adoption has accelerated with the rise of LLMs, and people within the field of security are increasingly exploring their practical value. Threat modelling is central to secure system development, yet it remains largely manual and the value of LLM-driven tools is unclear. Even when LLMs prove useful, selecting the right one can be more challenging than using it. This paper introduces a systematic evaluation framework for LLM-driven threat modelling tools to support tool selection, observing the general LLM-integration, governance risks, and allowing for comparison of tool output. Using Goal-Question-Metric, we derive evaluation metrics and show the value of the framework on a set of state-of-the-art LLM-driven threat modelling tools. The results show our metrics distinguish both performance and governance risks, providing a basis for organisations to ensure automation strengthens rather than burdens their threat modelling process.",

keywords = "Automated Threat Analysis, GQM Framework, Governance, LLMs, Risk Assessment, Secure Software Development, Security Evaluation Framework, Threat Modelling",

author = "Bakka, \{Josephine Marie\} and B{\o}gedal, \{Tobias Worm\} and Brandh{\o}j, \{Andreas Kjeldgaard\} and Johannes Bjerva",

year = "2026",

month = jul,

doi = "10.5220/0015000400004103",

language = "English",

isbn = "978-989-758-858-7",

volume = "1: SECRYPT",

pages = "689--700",

editor = "\{De Capitani Di Vimercati\}, Sabrina and Pierangela Samarati",

booktitle = "Proceedings of the 23rd International Conference on Security and Cryptography",

note = "23rd International Conference on Security and Cryptography : Volume 1, SECRYPT ; Conference date: 16-07-2026 Through 18-07-2026",

}

Bakka, JM , Bøgedal, TW , Brandhøj, AK & Bjerva, J 2026, A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools. in S De Capitani Di Vimercati & P Samarati (eds), Proceedings of the 23rd International Conference on Security and Cryptography. vol. 1: SECRYPT, SciTePress, pp. 689-700, 23rd International Conference on Security and Cryptography, Porto, Portugal, 16/07/2026. https://doi.org/10.5220/0015000400004103

A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools. / Bakka, Josephine Marie ; Bøgedal, Tobias Worm ; Brandhøj, Andreas Kjeldgaard et al.
Proceedings of the 23rd International Conference on Security and Cryptography. ed. / Sabrina De Capitani Di Vimercati; Pierangela Samarati. Vol. 1: SECRYPT SciTePress, 2026. p. 689-700.

Research output: Contribution to book/anthology/report/conference proceeding › Article in proceeding › Research › peer-review

TY - GEN

T1 - A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools

AU - Bakka, Josephine Marie

AU - Bøgedal, Tobias Worm

AU - Brandhøj, Andreas Kjeldgaard

AU - Bjerva, Johannes

N1 - Conference code: 23

PY - 2026/7

Y1 - 2026/7

N2 - AI adoption has accelerated with the rise of LLMs, and people within the field of security are increasingly exploring their practical value. Threat modelling is central to secure system development, yet it remains largely manual and the value of LLM-driven tools is unclear. Even when LLMs prove useful, selecting the right one can be more challenging than using it. This paper introduces a systematic evaluation framework for LLM-driven threat modelling tools to support tool selection, observing the general LLM-integration, governance risks, and allowing for comparison of tool output. Using Goal-Question-Metric, we derive evaluation metrics and show the value of the framework on a set of state-of-the-art LLM-driven threat modelling tools. The results show our metrics distinguish both performance and governance risks, providing a basis for organisations to ensure automation strengthens rather than burdens their threat modelling process.

AB - AI adoption has accelerated with the rise of LLMs, and people within the field of security are increasingly exploring their practical value. Threat modelling is central to secure system development, yet it remains largely manual and the value of LLM-driven tools is unclear. Even when LLMs prove useful, selecting the right one can be more challenging than using it. This paper introduces a systematic evaluation framework for LLM-driven threat modelling tools to support tool selection, observing the general LLM-integration, governance risks, and allowing for comparison of tool output. Using Goal-Question-Metric, we derive evaluation metrics and show the value of the framework on a set of state-of-the-art LLM-driven threat modelling tools. The results show our metrics distinguish both performance and governance risks, providing a basis for organisations to ensure automation strengthens rather than burdens their threat modelling process.

KW - Automated Threat Analysis

KW - GQM Framework

KW - Governance

KW - LLMs

KW - Risk Assessment

KW - Secure Software Development

KW - Security Evaluation Framework

KW - Threat Modelling

UR - https://www.scopus.com/pages/publications/105048657250

U2 - 10.5220/0015000400004103

DO - 10.5220/0015000400004103

M3 - Article in proceeding

SN - 978-989-758-858-7

VL - 1: SECRYPT

SP - 689

EP - 700

BT - Proceedings of the 23rd International Conference on Security and Cryptography

A2 - De Capitani Di Vimercati, Sabrina

A2 - Samarati, Pierangela

CY - SciTePress

T2 - 23rd International Conference on Security and Cryptography

Y2 - 16 July 2026 through 18 July 2026

ER -

A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools

Abstract

Conference

Keywords

Access to Document

AUB Link

Other files and links

Fingerprint

AI:SECURITY

Cite this

A Hierarchical Evaluation Framework for LLM-driven Threat Modelling Tools

Abstract

Conference

Keywords

Access to Document

AUB Link

Other files and links

Fingerprint

Projects

AI:SECURITY

Cite this