I’m not a lawyer (and if I were a lawyer, I wouldn’t trust myself as far as I could throw myself), so take this with a grain of salt:
First, I don’t think there’s anything dramatically special about Vex motors or motor controllers. You can find similar items in hobby shops, on-line DIY electronics stores, etc. So I’m guessing if you wanted to patent something, you could describe the motor and controller in a very generic fashion. In fact, I’m guessing you would want to describe such a motor generically so somebody can’t bust your patent because you were overly specific.
Having said that, I think Vex components are made by these guys: https://www.innovationfirst.com/about
And I don’t think they are non-profit. They would probably be happy to see their items go into commercial products so they could make more money, but that’s just my guess.
On the other hand, if you were to try to patent an educational robotics platform using Vex components, you might find yourself getting slapped by the REC, et al, who probably have agreements on who uses the Vex components for education and so forth.
Just my opinion, though. Somebody please correct me if I’m wrong on that.
This is done all the time. Buy parts from another manufacturer, combine them together somehow to create a new product, profit. We would’t usually use VEX motors but I don’t see any difference between using them and any other commercially available part.