EU AI Act: risk classification for crowd safety AI tools

Prohibited

Examples

Real-time remote biometric surveillance; social scoring; manipulative AI systems

Obligation

Banned outright under Art. 5 — do not deploy under any circumstance

High-risk

Examples in crowd safety

Emergency call classification; dispatching of first responders; crowd triage systems (Annex III 5d)

Key obligations (Title III)

Conformity assessment; risk management system; bias-tested training data; human oversight; logging; transparency

GPAI / LLM

Examples

LLM-based reporting assistants; situational awareness tools; foundation model integrations

Key obligations (Title VIII)

Transparency; technical documentation; disclosure of capabilities and limitations (Art. 52)

Limited risk

Examples

Chatbots or AI interfaces interacting with members of the public during an incident

Key obligations

Inform users they are interacting with an AI system (transparency obligation)

Minimal risk

Examples

Logistics optimisation; spam filters; non-critical background analytics

Key obligations

No specific obligations; voluntary codes of practice encouraged

Classification decision logic for new AI tools

Does the tool classify emergency calls, dispatch first responders, or perform triage?

YES HIGH-RISK (Annex III 5d) — full Title III compliance required

Does the tool process personal data (geolocation, biometrics, social media)?

YES GDPR obligations apply in full — lawful basis, DPIA, data minimisation

Is the tool based on a GPAI or large language model foundation?

YES Title VIII obligations apply — transparency, documentation, capability disclosure

Does the tool operate without human oversight or make fully autonomous decisions?

YES PROHIBITED for high-risk applications under Art. 14 — human-out-of-the-loop is banned

Key regulatory obligations by instrument

EU AI Act (2024/1689)

Art. 5 — Prohibited practices: biometric surveillance, social scoring

Art. 6 & Annex III 5d — High-risk classification for emergency dispatching

Art. 9–15 — Risk management, data quality, documentation, oversight, accuracy, conformity

Art. 14 — Human oversight mandatory; human-out-of-the-loop prohibited

Art. 52 — GPAI transparency and capability disclosure

Art. 2(3) — National security exemption (civil protection NOT exempt)

Art. 6(1)(d)+(e) & Recital 46 — Vital interests and public task as lawful bases in crises

Art. 5 — Data minimisation, purpose limitation, storage limitation, accuracy

Art. 22 — Right not to be subject solely to automated decisions; human review required

Art. 25 — Privacy by design and by default from the outset

Art. 32 — Security of processing: encryption, access control, breach response

Art. 35 — DPIA required for high-risk data processing (biometrics, geolocation)

Compliance gaps in crisis AI tools (SAPEA 2025) and recommended remediations

AI Act requirement	Observed gap in crisis tools	Recommended remediation
Risk management system	No continuous risk assessments; post-deployment monitoring absent	Establish lifecycle risk register; quarterly post-deployment audits
Data quality & representativeness	Training data incomplete, biased, or undocumented	Bias audits; enforce demographic representativeness in datasets
Transparency & explainability	Black-box models (LLMs, deep learning) with limited auditability	Implement XAI; require model rationale in operator interfaces
Human oversight	Human-in-the-loop not consistently implemented or stress-tested	Mandate override mechanisms; stress-test under operational scenarios
Technical documentation	Informal or proprietary; no standardised documentation	Adopt standardised protocols aligned with Art. 11 AI Act
Conformity assessment	Not in place for most civil protection tools	Pre-deployment assessments; pilot a Crisis AI Compliance Toolkit

Compliance obligations by lifecycle stage

Stage 1

Design & procurement

Classify tool (risk tier)
Conformity assessment
Bias audit
DPIA
Technical documentation

Stage 2

Training & pilots

Pre-define decision rules
Stress-test oversight
Validate data quality
Operator training on overrides

Stage 3

Live deployment

Human oversight active
Logging all AI outputs
Data minimisation
Human review for individual-affecting decisions

Stage 4

Post-event review

Post-deployment audit
Red-teaming
Incident documentation
Data deletion / anonymisation
Annual compliance review

Key principle (SAPEA 2025): AI systems must operate within pre-established decision thresholds and prioritisation rules — rules defined before deployment, not by the AI. This safeguards legal defensibility, human oversight, and accountability even when outcomes are imperfect. The L'Aquila earthquake trial demonstrated the legal risks of failing to distinguish uncertain AI forecasts from human decision protocols.

Legal Compliance Frameworkfor Crowd Safety AI

Legal Compliance Framework
for Crowd Safety AI