S3 Object Lambda is the ideal solution as it is purpose-built to modify data as it is retrieved from Amazon S3. By creating an S3 Object Lambda endpoint, a Lambda function can be invoked to process the data from any S3 GET request made through that endpoint. This function can contain logic to inspect the request context and dynamically redact Personally Identifiable Information (PII) before the data is returned to the calling application. This approach maintains a single, original copy of the dataset in S3, fulfilling the dynamic redaction requirement with the least operational overhead and without data duplication.

1. Amazon S3 User Guide. The official documentation explicitly lists PII redaction as a primary use case for S3 Object Lambda. It states

"With S3 Object Lambda

you can add your own code to S3 GET...requests to modify and process data as it is returned to an application." It further clarifies a use case: "Redacting personally identifiable information (PII) – You can redact PII data...for any application that does not need access to this data." This avoids creating derivative copies.

Source: AWS Documentation

Amazon S3 User Guide

"Transforming objects with S3 Object Lambda".

2. AWS Blog: "Filter and redact sensitive data from your data lake using Amazon S3 Object Lambda". This official AWS blog post provides a detailed architectural pattern for the exact scenario in the question. It demonstrates using S3 Object Lambda to "dynamically filter and redact data based on the user’s permissions" without creating multiple copies of the dataset

highlighting it as a low-overhead solution.

Source: AWS Big Data Blog

November 15

2021.

3. AWS Whitepaper: "Best Practices for Organizing

Training

and Securing Your Data Lake". While discussing data governance and security

such whitepapers advocate for mechanisms that can apply policies dynamically at the time of access. S3 Object Lambda is a direct implementation of this principle

allowing for fine-grained

dynamic data masking without the overhead of pre-processing and storing multiple data versions.

Source: AWS Whitepaper

"Best Practices for Organizing

Training

and Securing Your Data Lake"

section on "Data Governance and Security".