Blake,
I do not believe the worm will work with the Rack gear as your describe above. The worm itself is small enough in diameter that your assumption is correct; direct driving a worm with a Vex motor will cause the motor housing to interfere with the rack.
You can however position the worm such that the motor is clear of the rack (utilizing additional gearing/shaft/whatever).
It should be noted that the worm does NOT mate very “cleanly” with the rack. Your mileage may vary.