Flow Matching, Normalization, and Custom Losses
11/11/2026
Flow-Matching is a simple and powerful way of solving multi-modal regression problems - but its simplicity also makes it easy to abuse. There are certain things you can do with Flow Matching that might seem legitimate but break down the baked-in mathematical assumptions it's built on. In this post we're going to talk about a couple of those things, and some better ways to approach them, starting with normalization.
In my opinion, normalization is probably the most misunderstand, ignored and frequently-got-wrong aspects of doing ML on animation data - so let's start with what most people understand by normalization and see why that isn't quite right.
Ask many ML researchers working on animation what normalization is and they'll probably point you to a line in their code that looks like this:
X = (X - X.mean(axis=0)) / X.std(axis=0)
Basically, subtracting the mean and dividing by the standard deviation on each dimension of the pose feature vectors.
The reason this is done (as many of you may know) is because multiple aspects of Neural Networks rely on the mathematical assumption that the input data is roughly centered at zero with a standard deviation of roughly one. But what actually are these assumptions? Well, there are basically two main assumptions and neither implies we need to do things exactly like that:
- Network Weight Initialization scales weights under the assumption that inputs are roughly zero centered with a scale of roughly
-1to+1. The idea being that as the input values propagate through the network you want to keep their scale roughly the same. Inputs which are too large or too small may cause the activations on deeper layers (and eventually the output) to be huge or tiny. - Certain losses such as the l2 loss assume differences roughly in the range
-1to+1or they make training unstable due to the resulting very large numbers (i.e. from squaring the error) causing large gradients.
That's all true. The issue is that when we scale the dimensions of our feature vectors there are other knock-on effects that people forget, but must also be accounted for.
- The scale of each dimension going into a network will roughly govern how much the network takes that feature into account when computing the output. All other things equal, features with small magnitudes tend to be used less by networks than those with large magnitudes.
- The scale of each dimension in the loss function will govern how much the network focuses on getting that particular dimension correct compared to others.
Those might seem like minor side-effects, but they can end up being very bad for network training if not accounted for.
The common example is this: let's consider the case where one of the features in our feature vector is almost always close to some constant value with some tiny floating point noise left over. In this case, this dimension will have a standard deviation very close to zero, which will mean its values get scaled up massively during the naive normalization above.
If we put these scaled up numbers into a network we're effectively giving a large importance to a dimension which is at best random noise, and at worst may have some kind of strange or hard to understand correlations with other dimensions in the input or output.
If we put these numbers into a loss function we have issues too: we're effectively forcing the network to try and predict with high importance what is (again) at best random noise, and at worst some number with bizarre correlations to other aspects of our input or output.
This isn't just a theoretical edge case either. If you have a feature vector containing the rotation of (for example) a knee joint, it's highly likely that two axes of that rotation encoding are almost always close to or exactly a constant value. So the reality is this situation happens pretty much any time we try to encode a pose as a flat vector.
So what's the solution? Well, we can still center the data using the mean of each dimension. This was never an issue:
X = X - X.mean(dim=0)
But if we're either inputting this vector into a network, or into a loss function, we need to make sure that the scale of each of the dimensions in the vector makes sense.
One way to do this is to instead scale each of the different parts of the vector by a single computed standard deviation. For example, we can take the mean of the standard deviation across the parts of the vector that represent either joint translations, or joint rotations, and use those single numbers to scale all the values together.
X[:,pos] = X[:,pos] / (X[:,pos].std(dim=0).mean() + eps)
X[:,rot] = X[:,rot] / (X[:,rot].std(dim=0).mean() + eps)
This scales inputs into roughly the right range, while maintaining the relative importance of all entries of the same type, both as input to the network, and in the loss function. That allows us to account correctly for joints that barely rotate vs joints which have massive ranges of motion.
To help with the ambiguity around all of this terminology I've seen people refer to this as "scaling" your feature vectors rather than "normalizing" them, which I like.
There is one final caveat to this story - and that's about dealing with what networks output. When we have an untrained network, assuming the input is "scaled" correctly as discussed above, the network will still output a vector which is roughly in the range -1 to +1 on each dimension. That means while the network input is "scaled", the output is "normalized" (in the senses discussed above). We can turn this "normalized" vector into a "scaled" vector by multiplying it by the per-dimension standard deviation of our "scaled" vectors. This gives the network a massive boost at the beginning of training since it doesn't have to learn the per-dimension scales itself.
That all probably sounds confusing, but I think some example code makes it clearer. Here is how we might deal with all of this for something like an auto-encoder. (This is just illustrative pseudo-code, not how you would actually do a training loop)
# Raw feature Vectors
X = ...
# Mean of raw feature vectors used for centering
Xmean = X.mean(dim=0)
# Centered feature vectors
Xcenter = X - Xmean
# Per-type scaling values
Xscale_pos = X[:,pos].std(dim=0).mean() + eps
Xscale_rot = X[:,rot].std(dim=0).mean() + eps
# Scaled feature vectors
Xscaled = Xcenter.clone()
Xscaled[:,pos] /= Xscale_pos
Xscaled[:,rot] /= Xscale_rot
# Standard deviation of our scaled feature vectors
Xscaled_std = Xscaled.std(dim=0)
# When we pass this through a network, our output will be "normalized"
# in the sense it should be centered at zero with std ~1 on each dimension
Xout_normed = network_autoencoder(Xscaled)
# We can re-scale this to match our "scaled" feature vectors
Xout_scaled = Xout_normed * Xscaled_std
# We should compute l2 losses in scaled space
loss = torch.mean(torch.square(Xout_scaled - Xscaled))
# If we want to convert the output from scaled to raw space we invert things
Xout_raw = Xout_scaled.clone()
Xout_raw[:,pos] *= Xscale_pos
Xout_raw[:,rot] *= Xscale_rot
Xout_raw = Xout_raw + Xmean
# For some more complex losses we might need to do them on the raw data
loss_fk = fk_loss(Xout_raw, X)
Where does Flow-Matching come into this? Well with Flow-Matching we have another very annoying normalization related difficulty: the initial noise distribution we sample from is effectively the same space in which we compute our loss over the network velocities:
# Sample from Unit Gaussian distribution.
Xtil = torch.randn_like(X)
# Sample uniform time value t
t = torch.rand([batch_size, 1])
# Compute linear interpolation
Xbar = (1 - t) * Xtil + t * X
# Compute velocity
Xvel = denoiser_network(torch.cat([Xbar, t], dim=-1))
# Compute l2 loss
loss = torch.mean(torch.square(Xvel - (X - Xtil)))
If our vectors are "normalized" so that each dimension has standard deviation one, we don't have an issue. But as we mentioned above, both for the network input and the L2 loss we don't really want each dimension normalized individually - as it means dimensions are not weighted correctly. So how can we use Flow-Matching with "scaled" feature vectors instead?
The trick is to swap our initial distribution from the unit gaussian to samples from our "scaled" feature vector distribution, using Xscaled_std to scale the initial sampling:
# Sample from Scaled Gaussian distribution.
Xtil = Xscaled_std * torch.randn_like(Xscaled)
# Sample uniform time value t
t = torch.rand([batch_size, 1])
# Compute linear interpolation
Xbar = (1 - t) * Xtil + t * Xscaled
# Compute velocity scaled by std
Xvel = Xscaled_std * denoiser_network(torch.cat([Xbar, t], dim=-1))
# Compute l2 loss
loss = torch.mean(torch.square(Xvel - (Xscaled - Xtil)))
Now our flow can work in the "scaled" space, and our loss accounts for things correctly. Of course, we must also sample from the same scaled distribution during inference.
Unfortunately, Flow-Matching has another limitation, which is that our loss must be an L2-like loss. I think this is best explained in Section 4.2 of this paper, but effectively the mathematics of training relies on the merging of two expectations, which only works if the loss on the velocities predicted by the network is an L2 loss.
Sadly this means things like forward kinematics, foot-sliding, or other "geometric" losses shouldn't really be used. In the same paper, there is a good evaluation related to this in Figure 9, showing what happens if you swap an L2 loss for an L1 loss. In reality this is unlikely to break things entirely, but it almost certainly introduces bad bias and most likely reduces the diversity of the sampling.
So what can we do? Well we can still use any losses which are basically L2 in flavor (see Section 4.5 of this for a more formal definition of what that means). For example, while we can't use a Forward Kinematics loss, we can still incorporate per dimension joint weights without breaking things, to counter the error propagation down the joint chain:
# Compute weighted l2 loss
loss = torch.mean(Xweights * torch.square(Xvel - (Xscaled - Xtil)))
We can also use losses which penalize per-frame differences to try and reduce jitter or noise (a common problem for Flow-Matching-based animation generation). Here is how we might do it assuming we have a shape [batchsize, 2, features] for our vectors.
# Compute velocity scaled by std
Xvel = Xscaled_std * denoiser_network(torch.cat([Xbar, t], dim=-1))
# Compute target velocity
Xtrg = Xscaled - Xtil
# Compute standard l2 loss
loss_frame = torch.mean(torch.square(Xvel - Xtrg))
# Compute frame delta l2 loss (assuming frames on dim 1)
loss_delta = torch.mean(torch.square((Xvel[:,1] - Xvel[:,0]) - (Xtrg[:,1] - Xtrg[:,0])))
# Combined loss
loss = loss_frame + loss_delta
It took me a while to intuit why this loss is allowed. If we re-write it as follows, then (to me at least) it becomes a little clearer why it works:
# Compute velocity scaled by std
Xvel = Xscaled_std * denoiser_network(torch.cat([Xbar, t], dim=-1))
# Flow matching error
Xerr = Xvel - (Xscaled - Xtil)
# Compute standard l2 loss
loss_frame = torch.mean(torch.square(Xerr))
# Compute frame delta l2 loss (assuming frames on dim 1)
loss_delta = torch.mean(torch.square(Xerr[:,1] - Xerr[:,0]))
# Combined loss
loss = loss_frame + loss_delta
This computes exactly the same thing but here the loss is saying "if you're going to make an error, make the same kind of error for consecutive frames", which for me more intuitively keeps the same average velocity target if we consider multiple samples of t and Xscaled.
The final limitation is that losses shouldn't depend on the predicted target itself (rather than the velocity). Again, this is explained in the same paper again in Section 4.1. While this doesn't break the mathematical framework exactly, it does produce instability if you are not careful.
So there we go - I hope that has shed some light on how we might want to approach scaling or normalization when we're doing Flow Matching - as well as planted some ideas for how we might adapt some existing losses to fit into the Flow-Matching framework. And that's basically all I've got for you today. I hope that has been clear and helpful! As always, thanks for reading.