Schedule

All seminars are happening on Thursday 4-5pm (CET), unless the description specifies a different day/time.

April 18th - Julien Siems and Riccardo Grazzi

Title: Is Mamba Capable of In-Context Learning?

Abstract:

State of the art foundation models like GPT-4 perform surprisingly well at in-context learning (ICL), a variant of meta-learning concerning the learned ability to solve tasks during a neural network forward pass, exploiting contextual information provided as input to the model. This useful ability emerges as a side product of the foundation model’s massive pretraining. While transformer models are currently the state of the art in ICL, this work provides empirical evidence that Mamba, a newly proposed state space model which scales better than transformers w.r.t. the input sequence length, has similar ICL capabilities. We evaluated Mamba on tasks involving simple function approximation as well as more complex natural language processing problems. Our results demonstrate that, across both categories of tasks, Mamba closely matches the performance of transformer models for ICL. Further analysis reveals that, like transformers, Mamba appears to solve ICL problems by incrementally optimizing its internal representations. Overall, our work suggests that Mamba can be an efficient alternative to transformers for ICL tasks involving long input sequences. This is an exciting finding in meta-learning and may enable generalizations of in-context learned AutoML algorithms (like TabPFN or Optformer) to long input sequences.

May 2nd - Łukasz Dudziak

Title: TBA

Abstract:

TBA

May 9th - Ganesh Jawahar

Title: TBA

Abstract:

TBA