Lightweight ETL on AWS Lambda Using DuckDB and Snowflake Connector

Original Japanese article : AWS Lambda × DuckDB × Snowflake ConnectorによるETLの実装 Introduction I'm Aki, an AWS Community Builder ( @jitepengin ). In my previous article, I introduced how to connect to Snowflake from AWS Lambda using Key Pair authentication. Securely Implementing Snowflake AWS Lambda Integration with Key Pair Authentication + Secrets Manager This time, I would like to try the event-driven data ingestion approach that I introduced in the previous article. In this article, I will implement an event-driven ETL pipeline that uses DuckDB on AWS Lambda to perform lightweight transformations on Parquet files stored in Amazon S3 and then load the processed data into Snowflake. In addition, during the implementation process, I encountered an interesting limitation where write_pandas fails when writing to a Catalog-Linked Database. I will also summarize the root cause and the workaround. Why Snowpipe Is Not Enough Snowpipe is a very convenient feature for automatic data ingestion. H