I'd imagine once a ladder has been pushed down it is flat on the ground BUT still touching the wall. So there is no need to move any further, simply get 2 or more warriors to pick the ladder up. It may then be raised immediately.
If you push a ladder down in your move phase for instance, the enemy can pick it up in the same turn! This is so that turns don't drag on. Remember warriors cannot climb it in the same turn.
It basically plays that a siege ladder once it reaches the wall, will be in place at the start of every turn should there be enough carriers to raise it when it does get pushed down.
If you have a Uruk Siege Ballista then warriors can ride to the top in the same turn!
Why Mordor forces don't have one at their disposal is beyond me!