Storage Backed Queue

This is the Storage Backed Queue implementation with fixed size of user initialization.
The main buffer structure variable and the internal data buffer both should be declared on the application side and the pointers of the variables are used in the module.

Usage

All API functions have the first argument as the buffer instance.

Internal data buffer size and pointer are initialized with module init function.

Module read and module write functions are used to read/write data bytes.

Other API functions to return the status of buffer whether empty or full.

Return codes of API functions are defined in header file.

Storage backed Queue calls
Function	Description
rs_sb_queue_open	Function to initialize the storage backed queue
rs_sb_queue_close	Function to de initialize the storage backed queue
rs_sb_queue_set_config	Function to sets the configuration for the storage backed queue
rs_sb_queue_set_sector_config	Function to sets the sector configuration for the storage backed queue
rs_sb_queue_push_data	Function to push the data into the storage backed queue
rs_sb_queue_pull_data	Function to pull the data from the storage backed queue
rs_sb_queue_check_available_file_count	Function to get available files count
rs_sb_queue_is_available	Function to check if data in queue is available or not
rs_sb_queue_flush	Function to reset the elements of the storage backed queue structure
rs_sb_queue_discard	Function to disacrd the elements of the storage backed queue structure

Error Code

Every API’s for the message queue module returns some success or failure values. Please refer below section,

RAPIDSEA ERROR CODES

Example Demo

Please refer below section,

Storage Backed Queue Demo

Message Queue Header Details

Documentation from the relevant header as follows:

Storage Backed Queue Specific APIs.

Defines macros and functions specific to Storage Backed Queue functionality

Author: Embien RAPIDSEA Team
Copyright: Embien Technologies India Pvt. Ltd.

Defines

RS_SB_QUEUE_FILE_DATA_BLOCK_SIZE

<SBQ Data block size



SBQueue

The intention of the Storage Backed Queue is to store the incoming data in a message queue and back it up in a persistent medium such as SD card or NOR flash or a standard file system. Performance and no loss of data are important considerations.



Design

There are two message queues – one for reading (read_data_buf) and one for writing(write_data_buf). To avoid costly memory move operations, they will be used as circular FIFOs and managed by 4 variables - push_write_offset, pull_write_offset, push_read_offset and pull_read_offset.

They will form the gate for the storage medium.

| |

—————-— | Storage | ——————

| (write_data_buf) |— > | | — > | (read_data_buf)|

| | | | | |

—————-— | | ————–—

                      |              |

                      |              |

                      ----------------

When a data comes in, it will be written in the write buffer. Once it overflows, it will be stored in the storage medium.

Similarly, when a data is requested, the data from read buffer will be given. But when no data is there, it will read from the storage medium to read buffer and then served from there.

Data Record Format

In coming data will be treated as a single bunch and stored as a data record. Each data record will be aligned to 4 bytes boundary

The data record is written in following format

u8 header

u24 data size

u8 [] data

u8 [0-4] for adjustment

u8 footer

u24 zeros

SBQueue supports multiple storage mediums and each of them need different handling.

Raw SD Card Storage

Data Block Format

Multiple records are collected and written together as a block either as a file or raw sector.

Each block is written in following format

u16 header_sign

u16 data_size in bytes

u8 record_validity (0 - valid, 1 - full list read complete. 2 - Skip the next n records given in the first data position)

u24 numeric_seq to indicate order

This is used primarily in the case where the performance is important while the SD card is accessed over a limited SPI interface. As each SD sector read/write operation takes a lot of time, the operations need to be minimized. So, data block size must be aligned to multiple sectors (say 4K size aligned to 8 sectors).

All the blocks in the SD Card are treated as a circular FIFO and managed by two variables:

file_index_first – Index to the next block to be read

file_index_last - Index to the next block to be written

The sequence number will be used to find the last oldest and latest written blocks in the SD card. As there could be a million blocks, binary search algorithm is used to search the lowest and highest of them.

Whenever a write happens to the SD card, the data block is written to it and s32_file_index_last updated.

whenever a read happens, the data block is read from it and s32_file_index_first updated. But there will be no way to know about this read. For that, a special flush operation is needed which will mark the lowest sequenced block to skip the blocks read so far.

To maintain this, another variable called s32_first_valid_block is used. This will be initialized to the lowest sequenced block index on initialization. Whenever the write(s32_file_index_last) overflows, this variable will be moved to always point to the lowest sequenced block (which in turn is s32_file_index_last). So, whenever a flush happens, it can be used to update the skip block count.



Validation

Assume the number of blocks in the SD card is 8 and each push occupies one block. Let us see the flow.

first_valid_block – FVB

file_index_first – RI (Read Index)

file_index_last – WI (Write Index)

FIFO Operation Table Description



Operation FVB WI RI Remarks

Initialization 0 0 0 - Restart 0 0 0 - Push 0 1 0 - Restart 0 1 0 - Push 0 2 0 - Pull 0 2 1 - Pull 0 2 2 - Restart 0 2 0 Flush not done Push 0 3 0 - Pull 0 3 1 - Flush 0 3 1 - Restart 0 3 1 - Push 0 4 1 - Pull 0 4 2 - Push 0 5 2 - Flush 0 5 2 - Restart 0 5 2 - Push 0 6 2 - Push 0 7 2 - Restart 0 7 2 - Push 0 0 2 - …

Notes:

During write, increment FVS if equal to WI and RI is greater than WI.
During write, increment FVS if equal to WI and there is space in the FIFO and it is not empty.
For restart, consider using memset-0 operation on the handle interface structure.

Raw NOR Flash Storage

Data Block Format

Multiple records are collected and written together as a raw sector

Each block is written in following format

u8 record status [0xAA and 0X00]

u8 reserved [zero]

u16 data_size in bytes

u8 [] data

NOR flash is used in a wide array of applications that require fast code execution and high reliability over a SPI interface. Data block size must be aligned to multiple sectors (say 4K size aligned to 1 sector). All the sectors in the NOR Flash are treated as a circular FIFO and managed by two variables:

file_index_first – Index to the next block to be read

file_index_last - Index to the next block to be written

The record status will be used to find the read and written blocks in the NOR Flash.

Whenever a write happens to the NOR Flash, write sector will be erased, the data block is written to it along with the record status as 0XAA and s32_file_index_last updated.

whenever a read happens, the data block is read from it, 4 bytes of data along with record status as 0X00 is written and s32_file_index_first updated.



Validation

Assume the number of sectors in the NOR flash is 2048 (i.e 4K) and each push occupies one sector. Let us see the flow. file_index_first – RI (Read Index)

file_index_last – WI (Write Index)

FIFO Operation Table Description



Operation WI RI Remarks

Initialization 0 0 Restart 0 0 Push 1 0 Restart 1 0 Push 2 0 Pull 2 1 Pull 2 2 Restart 0 0 Push 1 0 Pull 1 1 Flush 2 2 Restart 0 0 Push 1 0 Pull 1 1 Push 2 1 Push 3 1 Push 4 1 Flush 4 1 Restart 4 1 Push 5 1 Push 6 1 Restart 6 1 Push 7 1 Restart 7 1 Push 8 1 Restart 8 1 Pull 8 2 Pull 8 3 Restart 8 3 Pull 8 4 Pull 8 5 Push 9 5 Push 10 5 Restart 10 5 Pull 10 6 Pull 10 7 Restart 10 7 Pull 10 8 Pull 10 9 Pull 10 10 Flush 11 11 Restart 0 0 Pull 0 0 Pull 0 0 Push 1 0 Push 2 0 Restart 2 0 Pull 2 1 Pull 2 2 Pull 2 2 Flush 3 3 Flush 4 4 Restart 0 0 Push 1 0 Push 2 0 Push 3 0 Push 4 0 Discard 5 5 Restart 0 0 Push 1 0 Discard 2 2 Restart 0 0

…

RS_SB_QUEUE_SD_CARD_INTERFACE

RS_SB_QUEUE_FLASH_INTERFACE

RS_SB_QUEUE_USB_DIR: sb_queue max file name*/

RS_SB_QUEUE_MAX_PATH_SIZE: sb_queue max file name*/

RS_SB_QUEUE_MAX_FILE_NAME: sb_queue file size*///EIR should be dynamic?

RS_SB_QUEUE_FILE_SIZE

RS_SB_QUEUE_FILE_APPEND_MODE

RS_SB_QUEUE_FILE_INDEX_OFFSET

RS_SB_QUEUE_OVERWRITE_DISABLE

RS_SB_QUEUE_OVERWRITE_ENABLE

RS_SB_QUEUE_BLOCK_HEADER_SIZE

< Block header - 4 bytes Sign + 4 bytes-size

Block data start location

RS_SB_QUEUE_BLOCK_DATA_START: Prepare block header.

RS_SB_QUEUE_BLOCK_HEADER_WITH_SIZE(h, s): Extract sign from block header.

RS_SB_QUEUE_BLOCK_HEADER_WORD(x): Extract size from block header.

RS_SB_QUEUE_BLOCK_HEADER_DATA_SIZE(x): Prepare sequence number.

RS_SB_QUEUE_BLOCK_STATUS_WITH_SEQ_NUM(s, n): Extract sequnce number.

RS_SB_QUEUE_BLOCK_SEQ_NUM(x): Extract block status.

RS_SB_QUEUE_BLOCK_STATUS(x): Block state information Valid data Block

RS_SB_QUEUE_BLOCK_STATE_VALID: Discard all the blocks till this one.

RS_SB_QUEUE_BLOCK_STATE_DISCARD_TILL_THIS: Discard the next n number of blocks given by first dword of data.

RS_SB_QUEUE_BLOCK_STATE_DISCARD_NEXT

Typedefs

typedef struct tag_rs_sb_queue_config rs_sb_queue_config_t

typedef rs_ret_val_t *sbq_backup_data(struct tag_rs_sb_queue_instance *ptr_sb_queue)

typedef rs_ret_val_t *sbq_restore_data(struct tag_rs_sb_queue_instance *ptr_sb_queue)

typedef rs_ret_val_t *ptr_fn_read_seq(struct tag_rs_sb_queue_instance *ptr_sb_queue, uint32_t u32_file_index, uint32_t *ptr_seq_num, uint32_t *ptr_total_skip)

typedef rs_ret_val_t *ptr_fn_write_seq(struct tag_rs_sb_queue_instance *ptr_sb_queue, uint32_t u32_file_index, uint32_t u32_seq_num, uint32_t u32_num_block_skip)

typedef struct tag_rs_sb_queue_instance rs_sb_queue_instance_t: Structure representing a queue instance for a ring buffer with file-based storage.

Functions

rs_ret_val_t rs_sb_queue_set_config(rs_sb_queue_config_t *ptr_sb_queue_config, uint8_t u8_interface_type, uint8_t *ptr_path, uint8_t *ptr_file_prefix, uint32_t u32_file_count, uint32_t u32_file_max_size, uint8_t u8_overwrite_enable)

rs_ret_val_t rs_sb_queue_set_sector_config(rs_sb_queue_config_t *ptr_sb_queue_config, uint16_t u16_physical_sector_size, uint16_t u16_logical_sector_size, uint32_t u32_total_sector_count, uint32_t u32_sector_start_count)

rs_ret_val_t rs_sb_queue_open(rs_sb_queue_instance_t *ptr_sb_queue_instance, rs_sb_queue_config_t *ptr_sb_queue_config)

rs_ret_val_t rs_sb_queue_close(rs_handle_t handle)

Closes an SB queue instance and performs necessary cleanup.

This function retrieves the SB queue instance associated with the given handle, checks if there is any data that needs to be backed up, and then marks the queue as uninitialized. Finally, it releases the handle associated with the SB queue.

Parameters:: handle – [in] - The handle of the SB queue instance to be closed.
Returns:: Possible return values include RS_ERR_OK for success, or error codes

rs_ret_val_t rs_sb_queue_push_data(rs_handle_t handle, void *ptr_data, uint32_t u32_data_size)

Pushes data into the SB queue associated with the given handle.

This function adds data to the write buffer of the SB queue. It manages the write offsets and ensures that the data conforms to the expected format with headers and footers. If the write buffer reaches its maximum size, the data is backed up to a file. It also handles cases where the buffer needs to be shifted to accommodate new data.

Parameters:

handle – [in] - The handle of the SB queue instance where the data will be pushed.
ptr_data – [in] Pointer - to the data to be pushed into the queue.
u32_data_size – [in] - The size of the data to be pushed.

Returns:

Returns the status of the push operation, or RS_ERR

rs_ret_val_t rs_sb_queue_pull_data(rs_handle_t handle, void *ptr_data, uint32_t *ptr_max_data_size)

rs_ret_val_t rs_sb_queue_check_available_file_count(rs_handle_t handle)

Gets file count.

This function returns the number of files count

Parameters:: handle – [in] - The handle of the SB queue instance from which to peek data.
Returns:: 0 on success or error code on failure

rs_ret_val_t rs_sb_queue_is_available(rs_handle_t handle)

Checks if there is available data in the SB queue.

It checks the initialized state of the queue and the indices to ascertain the availability of data.

Parameters:

handle – [in] - The handle of the SB queue instance to check for availability.

Returns:

- 1 if data is available in the queue,

0 if no data is available,
RS_ERR_NULL_POINTER if the provided handle is invalid or NULL,
RS_ERR if the queue has not been initialized.

rs_ret_val_t rs_sb_queue_flush(rs_handle_t handle)

Flush all the elements in the queue.

This function flushes all the elements in the queue

Parameters:: handle – [in] - The handle of the SB queue instance to check for availability.
Returns:: 0 on success or error code on failure

rs_ret_val_t rs_sb_queue_discard(rs_handle_t handle)

Discard all the data in the queue.

This function discard all the elements in the queue

Parameters:: handle – [in] - The handle of the SB queue instance to check for availability.
Returns:: 0 on success or error code on failure

struct tag_rs_sb_queue_config

Public Members

uint8_t sb_interface_type: indicate the interface type

uint32_t file_count: File count

uint32_t file_max_size: File maximum size

uint32_t file_block_size: File block size

uint8_t overwrite_on_overflow: When set overwrites old data on buffer overflow

uint16_t physical_sector_size: Physical sector size

uint16_t logical_sector_size: Logical sector size

uint32_t total_sector_count: Totla number of logical sectors

uint32_t sector_start_count: Logical sector offset

rs_sd_card_info_t *ptr_sd_card_info

struct tag_rs_sb_queue_instance

#include <rs_sb_queue.h>

Structure representing a queue instance for a ring buffer with file-based storage.

Public Members

rs_sb_queue_config_t *ptr_sb_queue_config: pointer to the sb queue

int32_t push_write_offset: Push write offset

int32_t pull_write_offset: Pull write offset

int32_t push_read_offset: Push read offset

int32_t pull_read_offset: Push read offset

uint8_t write_data_buf[RS_SB_QUEUE_FILE_SIZE]: Write data buffer

uint8_t read_data_buf[RS_SB_QUEUE_FILE_SIZE]: Read data buffer

int32_t file_index_first: File index to read data from next

int32_t first_valid_seq_index: This holds the block that has the lowest seq num

int32_t file_index_last: File index that should be written next

uint8_t discard_fvs_update: Once all sectors has been written and Write index overflowed to FVS, it is no longer necessary

uint32_t sub_write_data_len: write data length

uint8_t *ptr_sub_read_data_len: read data length

uint32_t read_file_size: Size of the file to be read

rs_ret_val_t ext_flash_handle: NOR flash handle

uint32_t performance_hit: Performance hit metric for the queue operation

sbq_backup_data *ptr_fn_backup_data: Pointer to backup data function for the queue

sbq_restore_data *ptr_fn_restore_data: Pointer to restore data function for the queue

ptr_fn_read_seq *ptr_fn_read_seq: Pointer to read sequence function for the queue

ptr_fn_write_seq *ptr_fn_write_seq: Pointer to write sequence function for the queue

uint16_t num_sectors_per_block: Number of sectors in each block

uint16_t sb_queue_status: sb queue storage status

uint32_t seq_counter: Sequence counter for managing queue operations