Running Screaming Frog on GCP with Cloud Run Jobs

Running Screaming Frog on a VM means paying for idle time between crawls. Cloud Run Jobs let you spin up a container, run the crawl, and shut down - you only pay for actual compute time. At Precis , we built an internal service that runs Screaming Frog crawls at scale using GCP. This post walks through the core setup - a simplified version you can deploy in about 30 minutes. For this proof-of-concept, we're going to use Cloud Run Jobs, Cloud Storage, and a simple Dockerfile combined with a bash file used as its entrypoint. What we're building The setup is straightforward: Dockerfile that installs Screaming Frog Entrypoint script that runs the crawl Cloud Run Job that executes the container GCS bucket to store exports Prerequisites You'll need: Google Cloud Project with billing enabled gcloud CLI installed and configured Screaming Frog license Basic Docker knowledge Setting up GCP resources These next steps assume you have gcloud sdk installed and that you are somewhat familiar with GCP