1. AWS Step Functions Developer Guide: "AWS Step Functions is a serverless orchestration service that lets you combine AWS Lambda functions and other AWS services to build business-critical applications." It explicitly lists AWS Glue and AWS Lambda as supported service integrations.
Source: AWS Step Functions Developer Guide
"What is AWS Step Functions?" and "Supported AWS Service Integrations".
2. AWS Step Functions Developer Guide - Managing AWS Glue Jobs: "You can use Step Functions to start an AWS Glue job... Step Functions can wait for the job to complete
and then retrieve the results." This confirms the direct integration required by the scenario.
Source: AWS Step Functions Developer Guide
"Manage AWS Glue jobs with Step Functions".
3. AWS Glue Developer Guide - AWS Glue Workflows: "A workflow in AWS Glue is a container for a set of related jobs
crawlers
and triggers... You can use a workflow to create a complex multi-job extract
transform
and load (ETL) pipeline." The documentation focuses on orchestrating Glue-native components
not external services like Lambda as a primary step.
Source: AWS Glue Developer Guide
"Overview of workflows in AWS Glue".
4. AWS Whitepaper - Practicing Continuous Integration and Continuous Delivery on AWS: This paper discusses the operational burden of self-managed solutions. "When you manage your own... servers on Amazon EC2
you are responsible for scaling
managing
and maintaining the infrastructure... AWS-managed services... reduce your operational overhead." This principle supports choosing a managed service (Step Functions) over a self-hosted one (Airflow on EC2/EKS).
Source: AWS Whitepaper: "Practicing Continuous Integration and Continuous Delivery on AWS"
section on "Operational Responsibility".
