With the unit bearing removed, the stub axle just "floats" in the seal. So "movement" in that case is expected.
With the unit bearing installed, the stub rides on the needle bearing portion of the unit bearing assembly. And...if I'm not totally mistaken, there is slight wiggle room in that.
With the hub installed in the unit bearing, and hub locked, there should be close to zero movement.
With both sides together and hubs installed but not locked, I would assume if you grabbed the u joint and moved it, you could possibly feel more movement in the drivers side. The axle shaft on that side is quite a bit shorter which would require less force to wiggle.
If the needle bearing on the side in question is good and the stub is not worn out the only reason to replace the whole assembly is if the unit bearing itself is bad.
If the needle bearing is worn out there is a write up on how to replace just it. Google "guzzle" and some other related words. You'll find it.
If the stub axle is worn or out of round it will need to be replaced.
Hope this helps.