Two-level concurrency in Node.js - worker threads and async pools for data integration pipelines

The problem Imagine we develop a SaaS platform that analyzes users' activity data - think fraud detection, behavioral analytics, compliance reporting. Our task is to create the integration pipeline that ingests and normalizes raw client data before it reaches those systems. Our clients collect data, store it in CSV files, and upload them to their S3 buckets. Then they create a mapping for their data in our application - select a datatype for each field, some of them can be serialized JSON objects, others parsed with complex regular expressions or base64 decoded, etc. Our task is to download these files, parse each row, apply custom mapping, find corrupted records, and store the converted data on our side. From a bird's-eye view the whole integration task is nothing else but: In more detail it works like this: Client stores CSV files in the S3 bucket. Each row represents one item Client gives us credentials for accessing these files Client creates mapping for rows Client schedules integ