hold does active PID to try to maintain the position: the higher the error the more power it tries to apply to counter the error. “Hold” means exactly that, it will try to hold the motor there. It uses the built-in PID that has been tuned to work for as many different outputs as possible. If it does not work for you, you need to use a custom PID.
brake is passive. It actually shorts the leads of the motor causing it to generate it’s own power making it slow down. However, if you try to turn it, it will (with friction) turn and not try to come back (passive). It is a mechanical brake, slowing down the motor without applying extra power.
coast is passive with no shorting, making it smooth.