Portrait of Marco Simoni

LLM Alignment & Reinforcement Learning

Marco Simoni

I work on making large language models reason more reliably — through reinforcement learning methods that stay stable under pressure, and by testing them where mistakes actually cost something: cybersecurity.

Researcher, Scuola Superiore Sant'Anna, Pisa PhD, Sapienza University of Rome

  • 232Citations
  • 6h-index
  • 4i10-index
  • 16Publications
Get in touch Read the papers Download CV

About

What I work on, and why.

I am a researcher at Scuola Superiore Sant'Anna in Pisa, in the Department of Excellence in Robotics and AI, where I work on reinforcement learning for post-training and alignment of large language and multimodal foundation models — policy optimisation, reward design, training stability. I hold a PhD in Artificial Intelligence from Sapienza University of Rome.

Most RL post-training methods optimise a reward and hope reasoning follows. My work looks at what actually happens to the gradient: with GTPO, now accepted at TACL, I showed that group-relative objectives push conflicting updates through tokens shared by rewarded and penalised completions, and that fixing this removes the need for KL regularisation entirely. At ACL 2026 we traced the hidden objective biases that group-based RL introduces without anyone asking for them.

I test these ideas in cybersecurity — vulnerability detection, threat intelligence, malware analysis — because it is a domain where a plausible-sounding wrong answer has a cost, which makes it an honest benchmark for reasoning. Alongside this I build open-source research infrastructure: DantinoX, GTPO, TITAN and MoRSE are all public.

LLM Alignment GRPO / GTPO Knowledge Graphs AI for Cybersecurity JAX / PyTorch

Experience

Where the work has happened.

  • June 2026 — present

    Researcher

    Scuola Superiore Sant'Anna, Pisa · Department of Excellence in Robotics and AI

    Reinforcement learning for post-training and alignment of large language and multimodal foundation models: policy optimisation, reward design, and training stability.

  • Oct 2025 — Sep 2026

    AI Researcher — Horus, a predictive cybersecurity foundation model

    NetGroup

    • Built a hierarchical Transformer (~10M parameters) from scratch that predicts the next attack from an attack sequence and autoregressively generates the continuation of the campaign.
    • Trained on 80,000 samples across four features — country, threat group, industry, time — reaching F1 = 0.60 on next-attack prediction.
  • May — Dec 2025 · Mar — Apr 2026

    AI Researcher — natural language to XACML policies

    CNR-IIT

    • A two-step LLM framework turning unstructured natural-language commands into compliant U-XACML access and usage control policies, via an intermediate structured JSON representation.
    • 93% accuracy on policy generation, 91% on ambiguous or noisy input, and 98% agreement with expert-defined policies — generalising across smart home, smart office and healthcare settings.
    • Hybrid LLM + dedicated-library design, validated for reliable on-device inference on constrained hardware.

European projects

Task leader

TURING — Trustworthy Unified Robust Intelligent Generative Systems

Sep 2025 — present

Led the development of adversarially robust training pipelines for multimodal foundation models applied to complex physical-system simulations.

Contributor

SIFIS-HOME — Secure Interoperable Full-Stack IoT for Smart Homes

Nov 2022 — Nov 2024 · H2020

Implemented core modules of the Data Analytics Toolbox: the System Protection Manager, the Application Manager and the Intrusion Detection System.

Project site

Education

  • PhD in Artificial Intelligence Excellence Sapienza University of Rome · Cybersecurity curriculum · 2022–2026
  • MSc in Artificial Intelligence and Data Engineering 110/110 cum laude University of Pisa · 2020–2022
  • BSc in Electronic Engineering University of Pisa · 2017–2020

Papers

Synced from Google Scholar. Last updated 21 September 2026.

Projects

Open-source research code — all of it public and runnable.

Language modelling framework

DantinoX — one backbone, three generation paradigms

JAX Flax NNX autoregressive masked diffusion flow-matching MHA / GQA / MLA MoE LoRA

Comparing autoregressive decoding, masked diffusion and continuous flow-matching is usually unfair: each paradigm lives in its own codebase, so any measured difference may come from the tokenizer or the training loop rather than the paradigm itself. DantinoX removes that confound.

The model backbone is fully separated from the generation method, so switching between AR, LLaDA-style masked diffusion and ELF flow-matching is a single configuration field — the weights, tokenizer, trainer and streaming generator stay byte-for-byte identical:

Attention (MHA/GQA/MLA, Flash, sliding-window, differential), feed-forward (dense, MoE, LatentMoE), positional encoding, optimizer, LoRA and DP×TP sharding are all flags on the same two dataclasses — thousands of valid combinations, zero code changes. It ships with a paradigm-agnostic trainer, a benchmarking suite with zero-execution FLOP counting, and a 14-subcommand CLI.

2-minute demo GitHub Documentation pip install dantinox

Reinforcement learning

GTPO — group-relative trajectory-based policy optimization

The official implementation of GTPO, a method for stable policy optimization in LLMs. It targets two failure modes of GRPO:

  1. Token-level gradient conflicts — tokens shared across positively and negatively rewarded completions receive inconsistent updates, which often ends up penalising essential formatting tokens.
  2. Policy collapse — negatively rewarded completions destabilise training, flattening the output distribution and degrading performance.

GTPO introduces conflict-aware gradient corrections and entropy-based regularization, which makes training stable without KL-divergence regularization and without a reference model at all.

Paper GitHub Explainer

Cyber threat intelligence

TITAN — typed bidirectional knowledge graph for CTI reasoning

TITAN is a typed, bidirectional knowledge graph framework for Cyber Threat Intelligence reasoning and question answering. It ingests MITRE ATT&CK STIX bundles, builds the TITAN ontology, generates both reasoning (CoT) and non-reasoning (NoCoT) datasets, and provides an end-to-end pipeline for training, evaluation and graph execution.

TITAN answering with Chain of Thought 2:40 · 6 MB
With Chain of Thought
TITAN answering without Chain of Thought, example 1 1:51 · 4 MB
Without Chain of Thought (1)
TITAN answering without Chain of Thought, example 2 2:01 · 4 MB
Without Chain of Thought (2)
TITAN used as a tool by a cybersecurity agent 2:22 · 5 MB
As a tool for a cybersecurity agent
Paper GitHub

Retrieval augmented generation

MoRSE — mixture of RAG security experts

MoRSE is the first specialised AI assistant for cybersecurity. It runs two Retrieval Augmented Generation systems designed to give structured, verifiable answers to security questions.

Unlike LLMs that answer from parametric knowledge alone, MoRSE retrieves from non-parametric knowledge bases and grounds its answer in what it found — which both improves accuracy and makes the answer checkable. Because the knowledge bases update in real time, MoRSE keeps learning new threats without retraining.

Paper (ACM SAC 2025) Demo

Writing

Shorter, less formal notes on what I am reading and building.