I was thinking the same thing. Years ago, I had a vehicle which would do the same and even sometimes pull the opposite direction when the brake was released. One of the front flexible brake hoses had broken down inside. The hose would baloon when pressure was applied and then collapse when pressure was released.