Making ConVEX a bit more like robotC

My team has been experimenting around between ConVEX and robotC, and we decided that we like robotC better for it’s fast download times and easy debugging. However, I personally prefer ConVEX for all of those real C features that make it easier for me to program, and which would eventually help my teammate too. Anyway, I want to make downloading ConVEX faster, so I had an idea:

What if I make it so that I only download the user ConVEX code to the robot through VEXnet, and then redirect competition function calls to the user code stored in flash?

I’ve got a few questions on this:

  • Does the Cortex map flash memory to global (or whichever correct term) memory space?
  • Would I have to specify exact address(es) in flash to write the user code to (like from the cortexFlash utility)?
  • Would running the code be as simple as calling the function from a pointer?
  • Could I just store some pointer offsets at the beginning of my new code, then transfer execution to the address specified by the respective offset?
  • Would I have to write a protocol for ConVEX to receive and write the user code, or could I just modify the cortexFlash utility?
  • For compiling this user code, could I simply remove all unnecessary source files in the Makefile?
  • How would I get the offsets of the different functions in this compiled user code, so that I can store them somewhere?
    • Otherwise, could I force each function to begin at a specific offset, or should I compile each function separately and flash them in different offsets?
  • Did jpearman not add this functionality because it was impossible without a lot of work?

Also, here is some information that might be important:

  • We are running Windows 10, and compiling via gcc-arm-none-eabi on Bash on Ubuntu on Windows.
  • We are not using Eclipse, we are using Atom and some batch files.

What your asking for is significantly over the top.

However I’ll bite (ish), save a copy of the last bin file, maybe even hash it and store the hash somewhere in flash outside the program space on the Cortex. Then modify Jame’s stm32flashCortex in such a way that it:

  1. Reads the stored hash, checks it against the OLD copy of the bin file you compiled and uploaded last time
  2. If it matches, do a diff on the new bin vs the old one
  3. Write the difference, should be faster

Question is, will all that work actually benefit you in the long run?

Relevant XKCD:

As for the time spent, in my opinion it is both a learning experience and something that would make ConVEX a more viable option. When programming with ConVEX, my team has had to flash using the USB A to A cable, which is difficult to plug in because of it’s position, and even then ConVEX takes twice as long as robotC going wireless through the controller (via the programming cable). This wouldn’t just help me, but anyone who uses ConVEX if and when I get this working.

To clarify, you suggested that I modify the cortexFlash utility to only flash when it is given a different file (therefore different hash), correct? Wouldn’t this just prevent extraneous re-flashing? Even so, it might be useful to add it (with an override option, of course).

I intend to compile and flash only the vex user code, while leaving ChibiOS on the cortex untouched. Of course, if ChibiOS isn’t on the cortex, I’ll have to flash it on anyway, but only once. This is in contrast to how ConVEX works currently, where it re-flashes ChibiOS and the vex user code every time, making it tedious as mentioned above. If I’m not mistaken this is similar to how robotC works as well.

Also, that XKCD comic was a nice touch.

Do you know how PROS flashes?

I know how PROS flashes: https://1drv.ms/w/s!Al7ukSGwH9Brj_NBRqHxoRASt7sCGg (disclaiminer: this isn’t VEX-scope authoritative/official and there’s likely some minor semantic inaccuracies)

And the actual implementation: https://github.com/purduesigbots/pros-cli/tree/master/prosflasher

To contribute to your original question however:

Yes, running code can be as simple as running the function given a pointer. Most of your other questions are the fundamental question that prevents flashing code in this manner. It’s much simpler and safer to just compile your code, link the OS (which is very small), and flash the whole binary than to do the partial linking as you described.

What’s being described is the use of a jump table, an area at know absolute memory address with pointers to system functions. ConVEX could be reorganized to work this way, my estimate would be user code size reduced to < 2k, you would place the vex system functions and ChibiOS functions in high memory and modify the flash download code to only erase flash where the user code would be placed.

The other thing you might want to look at is changing some of the compile and linker options.
have a look at using -fdata-sections -ffunction-sections as compiler flags and then --gc-sections as a linker options, this may (I have not tried it with ConVEX) make the final binary smaller.

Hmm, I’ll look into that. Right now I am trying to figure out how to do a partial (or rather, shared?) link with gcc.

So I have successfully compiled the ChibiOS and ConVEX code as a shared library (.so), but when trying to link just the user code to it I keep getting an error telling me that


ld

(gcc linker) had a segmentation fault. I’ve read that this is supposed to be a bug in gcc, so I am trying to get a true Linux machine running for testing (instead of bash on ubuntu on windows). Any ideas?

Exact steps:

  • Create files vexplaceholder.c and vexplaceholder.h (with placeholder code)
  • Duplicate test project into “OS” and “User” projects
  • Modify the “OS” Makefile and replace

$(VEXUSERSRC)

with


vexplaceholder.c

  • Compile “OS” as normal (make)
  • In bash,

cd

to /bin/obj, then run


arm-none-eabi-gcc *.o -shared -o ../../../libconvex.so

  • Modify the Makefile in “User” and do the following:
    • replace the

CSRC

line to be


CSRC = vexuser.c

    • add

$(CONVEX)

to the


ULIBDIR

line

    • add

-lconvex

to the


ULIBS

line

  • Compile “User” as normal (make)

I wouldn’t have thought a shared library would work, however, I will give you some other suggestions later when I’m back at my computer

How about a flasher program which caches the previous build and flashes only the differences? To my knowledge this would work with both ConVEX and PROS, and would still make the download process much faster. The linker script would have to be modified to put all of the vex user functions at the end of the binary, or the OS be put at the end of the binary and the user code at the beginning, that way modified user code only offsets the rest of the user code after it (keeping flashing time quick). Also, there would have to be an override option in the flasher program to flash the entire binary.

We could take this a step further, and designate (or use predefined) flash storage to store a checksum of the previous program, and possibly other information (like checksums of the OS code and user code [and vex driver code?] separately, say if someone wants to modify the OS), that way the flasher program knows exactly what to rewrite (might have to parse the .map files for it though). I was personally thinking it would be best to embed it into the code, by using a static variable to allocate space (see below), then modify that area’s binary data with whatever information is necessary.


static char __attribute__ ((section(".data.__checksumAndOptions"))) __checksumAndOptions[16];

void vexUserSetup() {
  __asm__ ("" : : "r" (__checksumAndOptions));
  // Other setup code...
}

Explanation on how the above code works (for everyone):


static

doesn’t do much, really


char

defines the variable as a byte (or array of bytes)


__attribute ((section(".data.__checksumAndOptions")))

tells the compiler to store the variable in the


.data

section, along with the other similar global variables (I suppose it might not be necessary)


__checksumAndOptions[16];

defines the variable name and size (16 ‘chars’ or bytes)


__asm__ ("" : : "r" (__checksumAndOptions));

inserts empty assembly code, but tells the compiler that the (empty) code needs to read the variable


__checksumAndOptions

, thereby forcing it into the output file

This approach has an issue however. Although very minor, the assembly call actually adds 4 bytes of machine code to the


vexUserSetup

function, presumably to load the variable into a register or something. I tried replacing the “r” with just “”, but then the


__checksumAndOptions

wasn’t included in the compiled binary according to the output.map file.

Yeah… no. There is a better way: modify the linker script. We can easily add padding somewhere and store the fancy checksums and other information there. In this case I think it might be easiest to slightly decrease the size of the flash memory that can store code, then store the meta information at the end, past the new definition of the flash memory. This way, the extra data does not interfere with the code, and is always at a predictable location.

Maybe my ‘old’ solution would work better for PROS, because otherwise it could interfere with the filesystem? I don’t know. I guess the PROS filesystem could be easily reprogrammed to avoid this chunk of memory if there would be collisions.

Does anyone see any problems with this approach?

I am working on programming a new flasher program now, by using jpearman’s code for CortexFlash as a base. The code is posted on github, but keep in mind that it isn’t finished yet. (I am working on understanding/modifying the hex parser code)

I just finished a prototype program and will test it when I meet with my team again. However, I still need to program the progress bars and operation time reports.

Also, looking back at Cody’s post, I realize this is pretty much what he meant all along. I apologize for my stupidity.

It works!!! Again, my code is posted on GitHub, and can be built with gcc. Progress indicators and operation timing are still missing, but it now works reliably. The only downside is that the cortex’s flash memory is erased in 2048-byte pages, requiring all of those bytes to be rewritten instead of overwriting just the changed bytes. (When the flash is written to, it only switches 1’s to 0’s, so to switch a 0 to a 1 the page needs to be ‘erased’)

However, I could use some help with the linker script. I was able to put the user code and variables at the end of memory (maybe not at the end of the binary though), but changes in the code (not just a variable value) end up requiring almost all of the pages to be rewritten because of a few changed bytes. I assume this is because the user function/variable locations changed, and all references to those functions/variables need to be updated. Is there a good way around this?

Sounds good, I will take a look at what you did this week.