Best ways to store binary data in OI

angelo_melo

I am looking for tips and suggestions on the best way to store and serve binary data on a table..

I have a DOCUMENT table that stores metadata about attachments, while the PDF files themselves are stored on the file system.

Whenever our web-based frontend needs to retrieve a PDF, it makes an HTTP call to our OI backend. The backend performs an `OSBRead`, then uses `SRP_Encode` to Base64-encode the PDF, wraps the encoded data in JSON, and sends it back to the frontend.

I have noticed that with larger PDFs, the time required to read and encode the files becomes noticeable. I have experimented with increasing the number of bytes read by `OSBRead`, which has improved performance, but the overall process can still be slow.

Would it make sense to store the Base64-encoded PDF data directly in the `DOCUMENT` record instead?

Are there any concerns or limitations I should be aware of regarding record size, performance, or storing large amounts of encoded binary data directly in an OI record?

BarryStevens

If OI10, then you should not have a 64k limit issue.
But store as a seperate record, not in a field in case binary values create field markers (IMHO)

DonBakke

I'm going to side step the original question for now and ask if you have run any benchmarks to determine if either the OSBRead call or the Base64 encoding takes up most of the processing. I'm asking because if it is really just one issue, rather than two, perhaps there are things we can suggest to resolve that single bottleneck.

angelo_melo

OSBread is the true bottleneck

BarryStevens

Hang on, why OSBREAD and not OSREAD. I would assume that OSBREAD would do some sort of string thingy (my tech term 🤣)

angelo_melo

Good question... I assumed I had to use OSB for binary files. But I just tried OSRead and it worked just fine, and it seems to be faster because i don't have to loop to read the next chunk of data.

Maybe my problem is solved

BarryStevens

Cool. the B probably stands for [B]lock

BarryStevens

... or Buffer or Bloody confusing!!!