Diffusion

Trajectory forecasting models aim to predict the future positions of agents given past observations. However, many real-world applications such as sports analytics require not just prediction, but controllable simulation of plausible futures under hypothetical scenarios. In this post, I investigate diffusion-based models for trajectory generation, and show how high-level “sketches” of plays can guide multi-agent dynamics. I have open sourced the code base for the model and canvas app. Sketched trajectory conditioning Simulated gameplay 1. The idea of sketch-guided trajectory simulation During NBA games, coaches often have to plan attacking plays to break through a defensive setup within a short period of time. They may sketch out instructions for a few players on a whiteboard, and rely on their mental model to project how both teammates and opponents might move in response. Prior work has explored data-driven approaches for this problem, using imitation learning1 and GANs2 to generate sketch-conditioned gameplay simulations. In this post, I explore whether diffusion models3 4 can be used for this task as well. If you are unfamiliar with diffusion, please check out this post. ...

This post is an introduction on diffusion written for my younger self. If you are new to diffusion and already familiar with VAE, this may be a good entry point for you. Diffusion has blown up over the past few years, with a huge amount of literature produced, but many differ in how they frame this modeling paradigm1. I do not intend to approach this via the most standardized or unified theory. I am not the best person to do that, nor do I think that angle helped me the most when approaching the subject (see my previous post). While many explanations derive diffusion from first principles, I found it useful to start by understanding how the objective is shaped into something we can actually train. ...