Data Filtering for Generative Video Pre-training | Field Notes by Linum

Image and video models have gotten a lot better over the last few years, even though the internals of these models haven't changed much since Stable Diffusion 3.Of course, there have been small variants like the auto-regressive diffusion that GPT-Image popularized. But at a high level, it's pretty much all flow matching with a transformer backbone and a v-prediction objective. In our experience, most of the gains are directly attributable to 3 flavors of data improvements:RL has driven some improvement, but it's only started truly working for image and video in the past few months. Data Filtering & Rebalancing: Remove noisy data and resample your data strategically so your model learns more effectively Data Annotation: Gather better annotations like richer captions, bounding boxes, and font details so that it's easier for your model to disambiguate visual conceptsLLMs have gotten infinitely better at image captioning in the last 12 months. Less so for video, but that's for another time. Synthetic Data Generation: Finetune an ensemble of existing generative models to create training data for which there is little-to-no naturally occurring data (e.g. image editing / reference-conditioning for Nano-Banana style models)Often a collection of brittle LoRAs trained for very specific tasks and an LLM finetuned to filter out bad synthetic data. A couple of years ago, the prevailing wisdom across all generative models (be it text, image, audio) was to aggregate as much data as humanly possible for pre-training. Luckily, the field has gotten a lot smarter about this. If you throw a bunch of low-quality data (e.g. heavily compressed JPEGs) into pre-training, your model is going to waste a significant amount of its capacity learning how to mimic this slice of data. If you filter your dataset well, your model will have a lot easier time learning what you want it to learn. We know this sounds obvious, but it's a lot harder to do in practice.Then again, all good advice should seem obvious in retrospect. Today we're going to walk you through how our approach to data filtering has evolved since 2024. And, hopefully we'll save you from a couple of headaches if you end up training your own generative models down the line. 2024Old-school CV on CPUsCPUPySceneDetectshot detection via heuristicsEAST Detector (OCR)smaller, less accurate OCR modelH.264 motion vectorsfilter out low-motion videos using L2-norm of codec’s motion vectorsHaar cascadesidentify and subsample talking head videosEarly 2025Finetuned LLMs on GPUsGPUAutoShot + TransNetV2neural networks for better shot detectionPaddleOCRlarger, better OCR model served at scale with TensorRTQwen-2-VL-2B, SFTcategorical filters on content (e.g. watermark, poorly lit)Late 2025Reinforcement LearningGPUWAFTneural networks for optical flow prediction; filter long-tail of low-motion videosDataset rebalancinguse captions as tags to subsample overrepresented categories within datasetQwen-2.5-VL-3B, RLVRfine-grained aesthetic filtering (i.e. score 1-4)keptthrown outthrown out by mistakekept but should be thrown outRL rubric [2024] Filtering on a budget — Traditional CV on CPUs On the first go around, we decided to push our raw dataset through old-school computer vision algorithms. This way we could get away with a cluster of cheap CPU instances instead of an unholy number of GPUs running a multimodal LLM.Or, spending several million in GPT-4 tokens. Scene detection We need to filter down tens of billions of images and videos to create our pre-training dataset. Images don't really require any specific pre-processing, but raw videos do.When you train a generative video model, you first need to pre-train on image generation. If the model learns nouns before it learns verbs, it tends to converge better (and faster). Next time you watch a television show or movie, track how often the camera cuts. If you're watching something made in the last twenty years, more likely than not you'll see a cut every 5 seconds. When to cut and how to cut is an authorial decision, not something a generative video model should do arbitrarily. So, we need to slice n' dice our videos on shot boundaries into video clips before we can filter them down. With our cheapskate CPU-only agenda, we picked up PySceneDetect. At a high level it maintains a rolling window of K-frames and if the K+1 frame has significantly different image statistics, it categorizes the frame as a cut. There's no underlying machine learning model. It runs really fast but struggles with common transitions like dissolves, fades, and jitter cuts (which low key is a huge issue). Getting to know your data Whenever you get new data, you should spend a few days reviewing random samples, listing what you'd like to keep and what you'd like to throw out. Ideally, you take the time to draft an ontology of categories within "good" and "bad" and track the relative sizes of these categories. At some point during the data filtering process, your engineer brain will take over, and you'll spend way too much time tuning the knobs of your heuristics (or LLMs), chasing that "perfect" decision boundary. These notes are going to save you from yourself down the line. They'll give you the facts you'll need to talk yourself out of trying "one more idea", when the answer is clearly "no". Plus, understanding the shape of the data distribution will really help with dataset rebalancing. Certain categories are overrepresented in the natural distribution of all videos. We need to subsample and suppress this signal, otherwise it will dominate training and our model will struggle to learn the long-tail of people/places/things/actions that we need in order to generate anything. Sieving out the un-captionable Generative video models are primarily limited by what we can describe correctly and consistently in words.Or more precisely, what LLMs can annotate for us cheaply. Text provides a pretty good scaffold to understand the visual world, but it's by no means the correct conditioning mechanism for all aspects of video generation. Details like camera trajectories in space-time and the nuances of an actor's performance are simply indescribable in natural language.Reference-driven video generation is currently in-vogue. It's the first articulation of a control for these ineffable properties of videos, but we don't think it's a very good control per se.It's sufficient for autogenerated and remixed content like memes and brainrot, but it doesn't allow for the level of fine-grained direction necessary to convey authorial intent. It doesn't enable us to create the compelling stories that this technology should herald. For now, we need to filter out clips where the primary "thing" that makes the video clip interesting is un-captionable. Without a crystal clear text description, it's just noise to our text-to-video model. Text-heavy For example, we want to filter out text-heavy videos. It's still hard for LLMs to caption motion graphics that are constantly changing on screen.Reading the screen isn't the hard part, it's establishing a clear description of when and how the text changes on screen. We don't want to waste capacity in our 2B parameter model learning motion graphics when it could be allocated instead to learning actions.Zooming out, our goal is creative tools for animation. Text and motion graphics aren't a priority right now. To do this, we sampled frames from each video and ran a tiny EAST Detector to extract bounding boxes for text. From there, we filtered out text heavy videos based on the percentage of the frames that had text and the percentage of each frame covered in text. Using a CNN for this task was a good idea, but the specific choice was wrong. In order to run tens of billions of frames on CPUs, we had to resize the frames aggressively. So, a lot of text-heavy samples with small fonts fell through the cracks. EAST is a pretty old model from 2017. It's small and far from the state of the art on text detection. Getting it to run efficiently on CPUs without cache-thrash and thread oversubscription was a challenge. Even after performance optimizations, it was still the largest bottleneck for this version of the data pipeline.It took us weeks to run EAST on our dataset, across a massive cluster of CPUs. Indescribable actions When there's not much happening on the screen (e.g. close-up on a person's face), it's hard to describe the specific action taking place. If there's too much happening (e.g. extremely shaky camera, a soccer match with a bunch of folks moving across the pitch at once), LLMs struggle to caption the clip correctly. We lumped these categories of videos together as "indescribable action" clips to be thrown out. Videos are typically serialized on disk in a compressed format. Codecs like H.264 reduce file size by storing keyframes and motion vectors that describe how the keyframes change over time, rather than RGB values for each pixel over time.We're eliding a bunch of specifics here. In H.264, the encoder computes the residual between two frames (the frame difference) and then applies the DCT (Discrete Cosine Transform) to the residual. This way we get a heavily quantized motion vector that gives us lossy frame reconstructions in exchange for a smaller file size on disk. We used the motion vectors stored within the mp4 files themselves to isolate and filter out the "indescribable action" videos.In 2024, we also tried running traditional optical flow algorithms like Lucas-Kanade on CPUs. They're extremely brittle and didn't transfer well to open world video. Specifically, we used mv-extractor and computed two heuristics per clip: average_frame_energy: L2-norm of all motion vectors averaged across the video min(sub_clip_average_frame_energy): Split each clip into a variable number of chunks depending on the video's length, calculate average frame energy for each chunk, and take the minimum across these L2-norms Then came the decision tree: