Diffusion for sketch-guided trajectory simulation

Tue, 28 Apr 2026 00:00:00 +0000

Trajectory forecasting models aim to predict the future positions of agents given past observations. However, many real-world applications such as sports analytics require not just prediction, but controllable simulation of plausible futures under hypothetical scenarios. In this post, I investigate diffusion-based models for trajectory generation, and show how high-level “sketches” of plays can guide multi-agent dynamics. I have open sourced the code base for the model and canvas app.

Sketched trajectory conditioning

Simulated gameplay

1. The idea of sketch-guided trajectory simulation

During NBA games, coaches often have to plan attacking plays to break through a defensive setup within a short period of time. They may sketch out instructions for a few players on a whiteboard, and rely on their mental model to project how both teammates and opponents might move in response. Prior work has explored data-driven approaches for this problem, using imitation learning¹ and GANs² to generate sketch-conditioned gameplay simulations. In this post, I explore whether diffusion models³ ⁴ can be used for this task as well. If you are unfamiliar with diffusion, please check out this post.

Sports Analytics on Some days I delve

Diffusion for sketch-guided trajectory simulation

1. The idea of sketch-guided trajectory simulation