I am currently in the process of developing what is known as the “Platinum Document.” This document will be posted later on over time once the document is polished enough.
In this screenshot, the camera knows where it is in 3D space, and also where the turning point elements are in 3D space by integrating machine learning with matrix mathematics. Because I am lazy, for now the machine learning is using a pre-trained library of YOLOv5 for the sake of testing. In blue of the image you can see that the tracked balls are the x, y, and z of the ball in world space. That is, whereever the camera is in 3D space, as long as it sees the balls the X, Y, and Z of the balls would be seen as static unless the balls themselves move.
As for being able to accomplish what is being done, I may release snippets of code however I don’t believe I am willing to release the github code (explained later). It is highly likely that this system will be a staple of VEXU position tracking due to its capability of permanence and relying on the environment instead of GPS field strips (we do not wish to flip a coin and hope that the field was setup correctly for the GPS system). Therefore we plan to provide steps on how to create a similar system that guides teams on how we approached this system.
The use case is to put the camera on the V5 robot and the V5 robot is capable of knowing where it is, and also where the objects are, without any additional idler odometry wheels.
When complete, the document will include:
- Pose estimation that has permanence, at a rate of ~30-60Hz, depending on the computational device. (X, Y, Z, pitch, yaw, roll in space)
- X, Y, Z object tracking relative to the camera (in meters)
- Object detection
When complete, the document will not include:
- X, Y, Z of tracked objects in world space (as shown in the picture). This is done in my own project with a large amount of matrix math and therefore is not something I can simply add, and neither am I comfortable sharing the raw code over github and flat-out giving code to individuals who don’t know how it works.
In development:
- Object permanence. This is the capability of identifying an object and giving it a unique identifier by recognizing where it is in 3D space. If the camera does not see the object by turning away from it, the camera can still keep note that the object still exists in 3D space despite not actually seeing it. This allows the robot to drive around and keep note and drive to objects across the field without actively needing to see it all the time. And if the camera looks back and sees the object disappear, then it removes the object from its index.
- Adequate communication library. I have already established a line of communication with the V5 Brain, however I need to establish communication in a manner that is modular and allows additional lines to be established over a single port.
The document will be released as a reply below once the document is polished enough. It ultimately adds many new features and also removes software (such as Docker) that are no longer a requirement to use.
