RTG Seminar by Msc. Luigi Di Marino and MSc. Rosy Caliri

Msc. Luigi Di Marino and MSc. Rosy Caliri
Title: Foundation models and Transformers
24. April 2025 - Physik West SE 22.00.0177 - 2:15 p.m.

The Transformer is the first transduction model to rely entirely on self-attention for computing input and output representations, without using sequence-aligned recurrent neural networks (RNNs). Its architecture has since become the building block of foundation models, one of the latest frontiers in artificial intelligence. Thanks to the large amount of training data and self-supervision, these models are highly efficient, demonstrating remarkable versatility and performance across a wide range of tasks. For this reason, they are becoming appealing in physics research nowadays.

In this talk, we will first present the core components of the Transformer—the encoder and decoder—as introduced in the original paper, with a particular focus on their attention mechanisms. We will then introduce the concept of foundation models, highlighting their main characteristics and capabilities.