Reinforcement Learning for Quadruped Walking
A walking policy for a Unitree Go1 that stays low and symmetric on flat ground, and still clears rough terrain, trained mostly by the shape of the ground rather than extra gait rewards.
UMass course project with Trung Dang
On the real robot: Unitree Go1
The same policy, transferred to a physical Unitree Go1 Edu.
Training
Isaac Lab, one L40S GPU
- parallel robots
- 3,000
- training iterations
- 10,000
- wall-clock time
- 20h
The policy usually settled between 5,000 and 7,000 iterations. The body-height target is 31 cm.
Goal, approach and outcome
- Goal
- Walk with long, low, symmetric strides, and still handle rough ground, without a separate reward for the gait itself.
- Approach
- Train in Isaac Lab on terrain that punishes bouncing and one-sided gaits. A depth camera is added for stairs and obstacles. Observations are normalized so the left and right sides stay balanced.
- Outcome
- Stable walking on flat and rough terrain in simulation, then the same controller on a real Unitree Go1.
Method
A policy that can climb stairs often stomps on flat ground. A policy tuned for a quiet walk often cannot clear the next step. The training setup is meant to get both from one controller.
Terrain instead of a gait reward
On a perfectly flat floor, almost any gait that moves the robot in the commanded direction gets the same reward. The policy then settles on a pronk or a canter: all the feet leave the ground together, or the steps lean to one side. Those gaits are noisy, and they do not look like an animal walking.
Rather than adding a foot-trajectory reward, the ground itself is given a few sparse obstacles. A bounce that used to be fine now trips the robot. The obstacles also set a useful foot height: once the foot clears them, lifting higher does not help. Slopes teach the body to stay level.
The blind policy trains on very rough ground, with spikes sampled from 4 to 20 cm. The vision policy does not. It can see 0.3 to 3 m ahead, so random spikes become something to memorize instead of a gait to learn. That version uses rails and waves, which are regular enough to step over and still irregular enough to punish a pronk.
What the network sees
Training is actor–critic in Isaac Lab, with an estimator for the body pose and the ground height. The actor does not get privileged state. It sees a 48-number proprioceptive vector: linear and angular velocity, the gravity direction in the body frame, the velocity command, joint positions and velocities, and the previous action. The action is a target position for each of the 12 joints, tracked by a PD controller.
When a depth camera is used, a small convolutional network compresses the image to 32 numbers and appends them, so the actor’s input grows from 48 to 80. The critic also sees an elevation map of 187 cells. Putting the camera in the same loop avoids the usual two-stage recipe, where a blind policy is trained first and vision is added later.
The scene is a grid. One axis changes the terrain type, the other raises the difficulty. Flat-ground training is a large flat tile. Rough training mixes pyramid slopes, horizontal rails, random rough patches, and stairs. The layout is mirrored left to right, and the stair side is balanced with flatter ground on the opposite side, so the policy cannot get away with using only three legs or leaping off one side.
Whatever asymmetry is left in the numbers is taken out by normalizing each observation: subtract the mean of the current rollout and divide by the standard deviation, with a small amount of noise on that deviation.
Rewards
The reward is the usual locomotion set, not a library of gait terms. It pays for tracking the commanded forward and yaw velocity, and it penalizes vertical bounce, roll and pitch rate, tilted orientation, joint acceleration, joint power, collisions, and jerky actions. Base height is pulled toward 31 cm.
The blind policy uses weights from prior quadruped work. The vision policy triples the velocity-tracking weights. The larger observation makes early returns very negative, and without that change the robot stops trying before it walks.
- Track commanded linear velocity and yaw rate.
- Keep the body at 31 cm and level, without hopping.
- Prefer smooth joint motion and few collisions.
Results
The comparison is qualitative: the gait without terrain shaping, the gait with it, and the same idea on flat ground, rough ground, and with a depth camera.
In simulation (Isaac Lab)
The policy these clips come from, trained with many parallel robots on GPU.
What the terrain changes
Without terrain shaping
Irregular jumping. The feet leave the ground together instead of stepping.
With terrain shaping
One foot at a time, with a longer and lower stride.
Flat, rough, and with depth
Rough terrain, blind policy
No depth image. The gait stays symmetric across the rough patch.
Baseline walk with joint and IMU readings only.
Rails, rough patches, and slopes, with the camera in the loop.
The same camera policy, back on flat ground, still walking rather than bouncing.
Against the usual training setup
Compared with the default Isaac Lab configuration and the reward weights used by earlier blind quadruped policies, the terrain-shaped runs did three things differently:
- Fewer bad gaits. Pronking and exaggerated foot lift, common when the only job is to track velocity, show up much less often.
- A more symmetric step. Mirrored terrain and normalized observations stopped the three-legged and one-sided leaps.
- A camera policy that actually uses the image. It clears obstacles the proprioception-only policy cannot see.
The controller then runs on the physical Go1. The clip at the top of the page is that transfer.
