Skip to Content (Press Enter)
How is your experience with the OutSystems Community?
2021-08-04 12-19-54
Mayank Dharmpurikar
119Views
5Comments
ODC: Merging Chunks of Large Binary Files Exceeding Input Payload Limit
Question
Reactive
ODC
Application Type
Reactive

I am working on an application where I need to handle large Excel files (up to 40 MB). Here’s how I’m processing the files:

  1. Upload and Chunking:

    • The file is uploaded via the browser.
    • It is split into 5 MB chunks and saved into the database.

  2. Processing with Timer:

    • A Timer fetches these chunks from the database and processes them asynchronously.
    • The Timer calls a .NET extension to merge the chunks back into a single binary file using a method similar to the BinaryConcat function in the Binary Concat Forge component.

  3. Issue:

    • When I pass all the chunks to the .NET extension method, I receive the following error:

      name: 'OutSystems.Application.ErrorHandling.ExtensionException', message: 'Input payload is too large (33.33MB), maximum allowed is 5.5MB.', stack: 'OutSystems.Application.ErrorHandling.ExtensionException'

    • This indicates that the input payload exceeds the OutSystems limit of 5.5 MB.

Questions:

  1. Is there any way to increase the maximum allowed payload size in OutSystems ODC for server-side processing?
  2. If not, what is the best approach to handle this requirement? My main goal is to reassemble the file efficiently to process it asynchronously.
  3. Is there another recommended method in OutSystems to handle such large files?

Additional Details:

  • The chunks are stored in the database as binary data.
  • The final merged binary file needs to be processed further (parsing the Excel data).
  • This process is triggered and executed entirely on the server-side (via a Timer).

Looking forward to suggestions and best practices for managing this scenario. Thank you!

0
0
09 Dec 2024
Copy Link
2024-07-31 11-32-34
ndy

Dear @Mayank Dharmpurikar .

Implement an external File Processing microservice to handle file merging and processing.

1. OutSystems uploads the chunks to a cloud storage service (e.g., AWS S3, Azure Blob Storage).

2. Metadata for chunks (e.g., file URLs) is sent to a microservice.

3. The microservice merges and processes the file, returning the result to OutSystems.

0
0
09 Dec 2024
Copy Link
2019-01-07 16-04-16
Siya
 
MVP

Please have a look at the "Use with large binary files" of the documentation at https://success.outsystems.com/documentation/outsystems_developer_cloud/building_apps/extend_your_apps_with_external_logic_using_custom_code/external_libraries_sdk_readme/ . This talks about the issue you are facing and some work arounds.

0
0
09 Dec 2024
Copy Link
2025-06-28 01-52-12
Marco Mateo

Hi,

I am also experiencing the same issue using AWS SDK for multi part upload in S3 bucket.

We are already slicing the binary into 5MB (5242880) but when we pass it to server action (external logic), it throws an error Input payload is too large (6.67MB), maximum allowed is 5.50MB

Any idea for workaround?

At least any document or info on how ODC compute for the Input Payload?

Thank you!

FYI. The minimum file that AWS accepts is 5MB. Hence the reason why I am chunking it to 5MB. Ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html

0
0
02 Feb
Copy Link
2021-01-01 09-23-30
Michael de Guzman
Champion

Hi Marco,


I'm not sure if this should be in another post but I hope this still helps.

Even if your file is 5MB, the "payload too large" error is thrown because of overhead as explained in the latter part of this link:  https://success.outsystems.com/documentation/outsystems_developer_cloud/building_apps/extend_your_apps_with_custom_code/best_practices_for_using_external_libraries/

Try using pre-signed URL instead:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html


https://www.outsystems.com/forge/component-overview/20242/s3-presigned-file-upload-helper-odc

0
0
08 May
Copy Link
2021-01-01 09-23-30
Michael de Guzman
Champion

Hi @Mayank Dharmpurikar , 

This is super late reponse but I hope it still helps :)

The reason you are still seeing that 33.33MB error even after chunking the file is because you are passing the whole list of chunks at once. 

Instead of pushing your list of chunks to your .NET extension, perhaps expose those chunks via a REST API with security token. On your .NET extension, pull the chunks via the REST API then do the assembly. This should take care of your issue.  Or better yet, assemble the chunks in ODC before returning the whole file to your .NET extension via REST API.  It depends on your requirement of course.

You can see this pattern recommended in the official docs under the "Use with large binary files" section: https://success.outsystems.com/documentation/outsystems_developer_cloud/building_apps/extend_your_apps_with_custom_code/best_practices_for_using_external_libraries/

0
0
08 May
Copy Link
Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.