DAY 13 - End-to-End Architecture Design

Day 13 of Phase 3: Performance & Production Thinking in the Databricks 14 Days AI Challenge – 2 (Advanced) focused on designing and documenting the end-to-end architecture of the system developed throughout the challenge. The first task involved creating an architecture diagram that represents the complete data and machine learning workflow. The architecture illustrates how raw e-commerce event data flows through a layered lakehouse design. Raw CSV data is ingested into the Bronze layer where it is stored as Delta tables. From there, feature engineering transforms event-level data into curated user-level features within the Silver layer. These features are used to construct the training dataset for machine learning models. Logistic Regression and Random Forest models are trained and evaluated, with experiments tracked using MLflow. The trained model is then used within a batch inference pipeline to score users and generate predictions that are stored in the Gold layer. In parallel, a c