How to do UnitTests with PROS?

Programming Support
,
#1

Hey all,

I mentor a team and am looking to start teaching them about Test Driven Development, UnitTests, and the like. In prep I want to set up an environment so I can point them in the right direction. I am running into issues though.

For context, they use Visual Studio as their IDE and use PROS CLI for building. They use PROS proper, not OKAPILIB.

For my tests I thought to have multiple projects - a ‘core-lib’ that has all the code that would be shared between the skills and the competition programs (using the PROS template system). This would be the main target for unit testing, as the skills and competition would be integration and system tests done manually on the robot. The unit tests would be done through a separate project that would also have the core-lib applied to it (so whatever unit test library and code doesn’t end up in the core-lib and thus in the skills and competition code).

I am having trouble though - when I use the Google Unit Test or the built in Microsoft Unit Test frameworks, I get linking errors using the imported core-lib:

|Severity|Code|Description|Project|File|Line|Suppression State|
|---|---|---|---|---|---|---|
|Error|LNK2019|unresolved external symbol public: static bool __cdecl AnalogValue::check_range(double) (?check_range@AnalogValue@@SA_NN@Z) referenced in function public: void __cdecl corelibtests::corelibtests::TestMethod1(void) (?TestMethod1@corelibtests@1@QAAXXZ)|core-lib-tests|C:\Projects\VexVRC\2022 Summer\core-lib-tests\core-lib-tests.obj|1||

There seems to be many things that can cause this - the .a file not being found for example. I do believe I have the .a file being located appropriately, so I don’t think that is it (the libraries and library directories are set to point to the firmware directory where the .a gets copied to when the template is installed).

Another possible cause is that there is a mismatch between the architecture: the PROS application is written for ARM, and the test libraries are written for x86 or x64. This, I think, is the problem.

My Question
Given all that context: how does one go about unit testing PROS code? Do you use any unit testing frameworks? Do you write the tests in the same project as the application? Do you need to run the tests on the v5 brain?

1 Like
#2

I haven’t heard of anyone doing unit tests for their programs in vex robotics. This is probably because it isn’t really viable to do without a physics simulation for your entire robot, if you want to test most of the functionality of the code. Instead you could run the tests on your actual robot, and see if it did the right thing. You could also potentially use some sort of position tracking like with the GPS sensor to make sure your robot ended up in the right spot.

I would assume you couldn’t get the testing libraries to work because of the architecture mismatch, as I don’t see too much of a reason it wouldn’t be able to compile other then the operating system being different. You probably need to use a ARM linux version of the library. Despite all this, I think unit tests could definitely be useful to test what broke once you get to a competition, because it always breaks at competitions for some reason.

#3

Given you can’t compile PROS without being part of the PROS team and under NDA, etc. you are probably best off looking at how OkapiLib has set up their unit tests. Generally, I’ve been able to run these out of WSL / WSL2 rather than from the windows command line.

1 Like
#4

Thanks iseua395.

There is a lot of logic that can go into the robot control that can be tested without hardware: are your limits being handled properly, does your ‘wrap around’ for setting angles behave as expected, does your PID algorithm produce the expected results, and so many more things.

You can also unit test with mock hardware to produce known / expected sequences of values from sensors, to ensure expected values are being sent to the motors, etc… Then given some known inputs, you can have the mocks record commands they get and compare them to expectations. This I don’t know if it is worth getting in to yet, but it is on my mind.

Ideally, you pass these tests every time(ish) you compile the code, and definitely before you put it on the final robot (again, ideally, realistically is a different story).

I would call the tests you are talking about - that require hardware - integration tests (they test actually doing an action in the environment it was meant to be done in) and are another important part of good testing behaviors. With the final level of tests being system tests which, for Vex I guess, would be running your Auton routines (or portions thereof) and checking results.

So, in my mind, there are three levels of tests:

  1. Unit Tests test the logic itself, isolated from the hardware
  2. Integration Tests test an action on the hardware (keep it as small and concise an action as possible)
  3. System Tests test the entire workflow being worked on
2 Likes
#5

Sorry, I don’t actually want to unit test PROS, (if that is what you meant) I want to unit test code written against the PROS library for V5. But I don’t think that changes the rest of your response:

Thank you for the pointer! I will look to see how they unit test. Brief look at it looks like they rebuild the GoogleTest framework with GCC (same cross compiler used by PROS) so they are on the same platform. I will post back with findings.

2 Likes
#6

Honestly, I’d seriously recommend using OkapiLib over raw PROS. There’s little glory in re-inventing the wheel…

1 Like
#7

I think the best person you can talk to in this case is @theol0403 (you might be able to find him at the VEX Discord) He managed to do some proper Unit Testing on PROS by interfacing with the okapi unit testing framework

Maybe @somewhereoutinspace can help you too since I remember seeing him ask about unit testing on PROS earlier as well.

Here’s the source on the okapi unit testing framework if you’re any interested.

I’d honestly recommend using okapi for anyone that uses pros and has prior experiences with programming. Really isn’t any shame in doing that. It has an abstraction layer for hardwares like motors and sensors. There are also classes like chassis models which can simplify programming too. It is not necessary for you to use the motion algorithms provided in okapi. Okapi itself is already an excellence framework to build your code upon…

5 Likes
#8

Unfortunately I’ve never worked with PROS unit testing, but i would like to second the notion of using Okapi. QUnits and other tools provided by the lib will go a long way towards keeping you sane.

1 Like
#9

Thanks all. I will ask the team to re-look at Okapi again - they started this past year with it but gave it up part way through the season. I will make sure they revisit it to make sure they make an informed choice. As @somewhereoutinspace said - having units-types for parameters is invaluable and something I was going to teach the kids this summer.

Still, the process of unittests is sort of independent of that, I need to figure out the logistics. I’ve been hacking away at getting the okapilib unittests running on Windows and on WSL2 for Windows with no luck. But I basically understand their process, I think:

  1. Compile the code using native (not arm) architecture for testing
  2. UnitTest on native platform
  3. Compile for arm architecture
  4. Put on the brain

If that sequence is correct, then I think I can mimic it on their preferred platform. If I misunderstand I hope someone can correct me.

  • If I find it necessary, I will open a different thread to troubleshoot running the okapilib tests, but for now I don’t think it belongs here.
#10

From what I recall (and it’s been a while):

mkdir cmake-build-debug
cd cmake-build-debug
cmake -DCMAKE_BUILD_TYPE=Debug -G "CodeBlocks - Unix Makefiles" ..
cmake --build . --target OkapiLibV5 -- -j 2
chmod +x ./OkapiLibV5
./OkapiLibV5

Where OkapiLibV5 is a binary that executes the unit tests

#11

Yeah, those are the commands, but the cmake -DCMAKE_BUILD_TYPE=Debug -G "CodeBlocks - Unix Makefiles" .. step fails when trying to build the googletest suite. The process of getting to that step also seems onerous for a Windows user (took over 6 hours to get everything downloaded and installed). So rather than try to troubleshoot at the moment I am going to abandon doing exactly that and instead do a lighter version if I can manage it.

Plan:

  • Given:
    • CMake is already a binary for Windows.
    • Google Test Suite is shipped with VS
    • VS can use CMake to do the building
  • Have a custom CMake script with a test recipe that:
    • compiles core-lib as an x86 application (??)
    • adds it to the test project
    • Triggers the VS Test Runner to get the visuals on which tests pass and which don’t

I have doubts about this plan, I may not be able to build the core-lib as a sharable library to consume by tests, so may need to bite the bullet and do tests directly in the core-lib like okapilib does.

#12

yeah judging from the error this seems to be correct. the error is looking for something from the standard lib that it thinks should be defined using the CDECL calling convention, where libpros.a is compiled for Arm and uses the APCS calling convention… But I digress.

As others have suggested okapilib is pretty much the only project (aside from Theo’s stuff) that’s using unit testing. This is accomplished by creating an API layer around platform-specific functions (like threads, devices, etc) and having those conditionally compiled/linked against the proper libraries for the platform.

I think you can solve this by doing it the other way around: have the test application consume core-lib, but specifically a core-lib that’s been compiled and linked for running on the host machine

4 Likes