Physics engines and full robot simulators.
Real-time physics simulation with a popular Python API for RL.
Gazebo packages for simulating underwater vehicles and manipulators.