Optimizing Large-Scale Data Ingestion into Relational Database

Eons ago I had a requirement to ingest large sets of data into our relational database (MySQL). This is how I approached the problem and optimized the solution, thought about sharing it in case someone needs something similiar. * *There are plenty of tools which do a much better job at loading data. Case Study - Optimizing Large-Scale MySQL Data Ingestion Ingest (Text/CSV)encrypted data file 18 MB to 100 MB large (around 80K to 1200K lines of data set) from a SFTP Server. The data format was predefined and positional. Poll and load data from the latest files from SFTP Load only the Delta from the files (new changes only) Update existing records with Unique key. The Tech stack was Java/Spring Boot, MySQL ** Initial Solution (Procedural) ** My initial solution was to write simple Java (with Distributed Transaction, Buffered Input, Local Caching etc...). Poll the SFTP for the latest file public void connect ( String host , String port , String username , String password ) throws JSchExcep