ROBOTC file system

About a year ago I wrote some code that would allow blocks of memory to be written as a file in the ROBOTC file system. This would allow, for example, an array or structure containing information that the robot had obtained to be saved in flash and later downloaded to the PC for analysis. I decided not to release the code at that time, as we know there are problems in using flash when the robot is functioning, however, because of the recent interest in “rerun” type programs I’m going to throw it out with the understanding that there will possibly be certain limitations.

The ROBOTC file system uses two areas of flash memory, a table of contents with two long integer entries per file, the address (as an offset from the beginning of the file system) of the file in flash and the size of each file. The files themselves can be stored anywhere, ROBOTC places then after the TOC, I place them in higher memory so that there is always some room for the ROBOTC program to be downloaded into what is called “slot 0”.

Each file has a header followed by the actual file data, the header could change with different versions of ROBOTC but has been stable at 22 bytes for some time (in the 3.6x versions, not checked V4.X yet). The header, as a structure, would be something like this.

typedef struct _flash_file {
    unsigned char name[16];                ///< file name
    unsigned char type;                    ///< file type, wav, exe etc.
    unsigned char time[4];                 ///< file creation time
    unsigned char unknown;                 ///< unknown flags
    } flash_file;

A 16 character (max) file name
A file type, these are defined in the RobotCIntrinsics.h file as follows.

typedef enum short
{
    ftNone              = 0,
    ftExecutable        = 1,

    ftSound             = 2,
    ftWAV               = 3,

    ftText              = 4,
    ftData              = 5,
    ftIcon              = 6,
    ftMIDI              = 7,
    ftLast
} TFileExtensionTypes;

There is a 4 byte time stamp and a final byte that I haven’t bothered to decode yet as it seems unused for data files.

I add some internal variables to the data structure as well but these are not written to flash.

Code that would print all the file names in the file system may look something like this (this is one of the library functions, the user would just call FileReadVTOC() to print the directory in the debug window).

/*-----------------------------------------------------------------------------*/
/** @brief     Read flash file table of contents and print in debug window     */
/*-----------------------------------------------------------------------------*/

void
FileReadVTOC()
{
    long *toc = (long *)(baseaddr + VTOC_OFFSET);
    long  addr;
    long  size;
    short slot;

    flash_file  f;

    // more than kMaxNumbofFlashFiles files we have an error
    for(slot=0;slot<kMaxNumbofFlashFiles;slot++)
        {
        // Read file address
        addr = *toc++;
        // read file size
        size = *toc++;

        // Good file ?
        if( addr == (-1) )
            return;

        f.addr       = baseaddr + addr;
        f.data      = (unsigned short *)(f.addr + FLASH_FILE_HEADER_SIZE);
        f.datalength = size;

        // Read header
        FileReadHeader( &f );       
        // Display
        FileDebug( &f );
        }
}

Code to save data to a file like this.

static short tmpData[1024];

#include <FlashLib.h>

task main()
{
    int   i;

    // create some test data
    for(i=0;i<1024;i++)
        tmpData* = i;

    // Write 200 bytes (100 short ints) to a file called test
    FileAddFile( &tmpData[0], 100, "test" );

    // Done
    while(1)
        wait1Msec(500);
}

I’m not following normal file access conventions, there is no fopen, fwrite, fclose etc. All you can do is dump memory to a new file that is added to the file system. When done, you can pull up the “File Management” window like this.

fileManagement.jpg

I still want to do a little testing so will probably post the code tomorrow or Thursday.*

I’m very excited for this :stuck_out_tongue: Looks promising.

[Crosses fingers for ConVEX version]

I’m not planning a ConVEX version in the short term, sorry. If/when Purdue release their file system I may piggyback on that, but ROBOTC has a few advantages in doing this compared to ConVEX or PROS.

  1. ROBOTC already has the File Management dialog, I’m already overloaded with work and other projects, I don’t really want to create a new GUI and protocol to access a file system in ConVEX.

  2. ROBOTC does not erase the file system when you download a new program. This is because of the way the VM works, when the VM is downloaded to the cortex all flash is erased, this is the equivalent of downloading a PROS or ConVEX project. When you download the ROBOTC program, the VM has the necessary smarts to be able to erase a small amount of flash and allow reprogramming. The program download process looks at the file system table of contents and is able to decide where the program should go based on how flash is currently assigned. I’m placing files in high memory (the top 192K) but that still leaves 96k for “normal” ROBOTC programs and 96K for the VM.

I already did 90% of the work on this about a year ago, it just needs a little cleaning and I will release it, but I’ve had enough for tonight and didn’t get that finished.

As I said, I love you forever. <3

Somebody is definitely toeing the creepiness line…

Thank you so much.

You know that you are a nerd when an announcement like this is the high point of your day

I posted the first release of this to github.

https://github.com/jpearman/rcfs

This does not work like a normal file system. The idea is that blocks of memory can be saved to files, a file is not opened and closed in the normal way, it would be bad to lose power with a partially written file and the VTOC would need to be constantly updated if a file were slowly added to. A more sophisticated file system could possibly work in this way, but it would not be a good idea for what ROBOTC has implemented.

The key functions are.

Add file with default (incrementing) names.

RCFS_AddFile( unsigned char *data, int length );
Add a file with a given name

RCFS_AddFile( unsigned char *data, int length, char *name )

Files can have duplicate names, it’s up to the user to keep names organized.

Get a pointer to the data in a named file (ie. open the file for reading)

RCFS_GetFile( char *name, unsigned char **data, int *length )

As the files are in flash, to read you can just access the data using a pointer.

All file size parameters are now given in bytes rather than words (which the original code used).

When writing a file** interrupts will be disrupted for up to 70uS**. This means that the ROBOTC debugger will briefly lose connection******, however, the user processor doesn’t seem to lose communication with the master processor. I would only use these functions when the robot is not in a competition. For a “rerun” type of program, record the data to an array in memory, then hit a button to cause that memory to be saved as a file.


Just a little more explanation of this. Although I don’t know exactly what causes the comms dropout, my suspicion is that receive characters are probably lost from the message going to the cortex from the PC. At 230k baud (the master to user processor baud rate), each character will arrive approximately every 43uS. There is no fifo in the STM32 uart so, if interrupts are disabled because of flash write for 70uS, at least one character is probably lost therefore causing a bad message. Due to bad message ROBOTC does not reply, hence the timeout on the PC when no reply is received.

I have put a limit of 8192 bytes as the maximum file size, this is an artificial limit so you can change this to whatever you feel comfortable with. I have not tested with large files so it may not work well. I have also not put any code to detect the end of the file system, so be aware of what you are writing and how much. You can do this in your own code like this.

#define   MAX_FLASH_FILE_SIZE   10000
#include <FlashLib.h>

If MAX_FLASH_FILE_SIZE is defined before the library is included and it will override the default.

You cannot delete files with this code. I suggest that individual files are not deleted using the “file management” dialog, to clean everything just reload the ROBOTC firmware which will erase all flash (and therefore files). Some older versions of ROBOTC left some junk in high flash memory, V3.62 looks OK in this respect but, again, I have not tested this to an extreme.

There are no functions that erase flash as part of the normal file system code, this means you will not damage the cortex by writing too much, you will just crash it and need to reload ROBOTC. There are some functions left in my stm32_flash library (which is called by the file system library) that can erase flash, just leave that alone and you will not damage anything.

See the flashFileDemo.c file to get started, please be very vareful with this code and if you are at all unsure of how it works then don’t use it!

Awesome, I will try this Saturday, Thanks for making it available.

Cheers Kb

I pushed a small update that checks for enough space before writing a file and also returns success or error from the RCFS_AddFile functions.

A couple of limitations that apply to this hack.

Don’t use the “file delete” or “all file delete” functions in the file management dialog. ROBOTC modifies the table of contents for the file system but data is not erased. As the code I posted does not erase flash, any data you write will be incorrect, remember that flash can only be programmed to a ‘0’, that is, when flash is erased all bits are set to a ‘1’, when we program the flash we change some bits to a ‘0’. The problem with erase is that it takes 20mS per page, that doesn’t seem like a long time but it causes all interrupts to stop and I have no idea how ROBOTC would react to this when done from user code. To erase all flash and start over just reload the ROBOTC firmware, this erases everything. Of course, if you only use the code for reading files that were loaded using the file management dialog then the above does not apply.

I would also set all motors to 0 before adding files, I don’t think anything bad will happen if you don’t but writing 10K of data to flash takes about a second and you will most likely have no motor control during this time.

First I would like to thank you for creating this library, however I am having issues with “writing” the data to the file. When I open the file that I just wrote to, it seems that the data is all garbage (ie. Random characters). I know the data is garbage due to the fact that I set all the elements of an array to be written to the same value. I opened the file and it was just a list of random characters that were not the same. I am confused as to why this is happening. I have been reviewing the library of code that was written for this file accessing system, and yet cannot find why this is happening. Does anyone have any insight as to why this is happening? Thanks!

Which version of ROBOTC are you using? As you can see this library was written some time ago and may not be compatible with the latest V4.26.

Edit: Yes, as expected it’s broken under 4.26, I will get it fixed and update the repo. It actuals causes a reboot of my cortex so surprised you could write anything if you are on this version.

Thanks man. Can’t wait for the updated version of this, bummer that it is not compatible for the current version. For future updates, what exactly is causing this error to happen? Is it the “address assignments?”

I pushed an update to add compatibility with ROBOTC V4.26

https://github.com/jpearman/rcfs.

I tested the demo code with both V3.63 and V4.26, due to the way this code works it may not be compatible with other versions of ROBOTC. Let me know if this fixes your issue, if not, send me the code that’s not working and I will take a look.

The code uses knowledge of the table of contents and layout of the ROBOTC file system, unfortunately, this layout can (and does) change with different versions, it was not intended to be accessed by users. V4.26 moved the table of contents and also changed the file header size.

Where do I find the information for the current changes for the table of contents and layout of the ROBOTC file system? I would like to thank you for your help so far.

Not sure what you want to know, there is no official information for any of the ROBOTC side of this, the code uses information I learned from reverse engineering. You can see the diff on github that shows what I changed in this revision.

Understood. Well thanks for all your help. I believe I have a better understanding of all that is going on!

One thing that I am curious about is how did you reverse engineer it? What information did you use to reverse engineer it, and where can I find the information I need to reverse engineer it myself (when the next version comes out)? Thanks for creating an updated version of the Flash Library by the way. Sorry about all the questions, this topic is truly interesting.

Still having issues with garbage being written to the file. The code I am using is the demo code you wrote (this will be the code I reference to from now on, since I know it has been successfully tested), and I am running on robotc v. 3.63 (most current robotc 3.X) however my team is looking into purchasing robotc 4.X and updating it to v. 4.26. I’ll keep you updated.