Data Mapper Streaming to DB

Streaming large files through DataMapper:

Streaming extra-large files through DataMapper without consuming tons of memory is possible.  Let me illustrate with an example.d1

The HTTP endpoint accepts a message – a large file – which it passes into a DataMapper. From Logger, message reaches a Foreach which wraps a DB endpoint. DataMapper must create “iteratable” objects from the file and so that the Foreach can process the items iteratively and push them into the database. Enable streaming to manage the processing of this large file

Step 1: To enable streaming, click to open the DataMapper Properties.

d2

Step 2: Check the box to enable streaming.

d3

Step 3: Save and start streaming!

d4

Leave a comment