Integration, data & analytics
Analytics and data lakes
Modern analytics on AWS centres on a data lake in S3, with serverless and managed engines on top. The exam tests which service to use for each step: catalog, transform, query, govern, visualise.
The data lake pattern
Ingest (Firehose, DataSync, DMS, AppFlow, uploads)
▼
S3 data lake — raw zone → cleaned zone → curated zone (Parquet, partitioned)
▼
AWS Glue Data Catalog (tables & schemas) ◄── Lake Formation (permissions, governance)
▼
Query & process: Athena · Redshift (Spectrum) · EMR · SageMaker
▼
Visualise: QuickSight
Services by role
| Service | Role | Cue words |
|---|---|---|
| AWS Glue | Serverless ETL (Spark), crawlers that infer schemas, the Data Catalog (central metadata), Glue DataBrew for visual data prep | "transform", "catalog", "CSV to Parquet", "serverless ETL" |
| Amazon Athena | Serverless SQL directly on S3; pay per data scanned; federated queries to other sources | "ad-hoc queries on S3", "no infrastructure" |
| AWS Lake Formation | Build and govern data lakes — central, fine-grained (table, column, row) permissions across Athena, Redshift Spectrum, EMR | "secure the data lake", "column-level access" |
| Amazon EMR | Managed Hadoop/Spark/Hive/Presto clusters (or EMR Serverless) | "big data processing", "existing Spark/Hadoop jobs" |
| Amazon Redshift | Columnar, massively parallel data warehouse; Spectrum queries S3 data; Serverless option; concurrency scaling | "data warehouse", "BI", "complex analytical queries" |
| Amazon QuickSight | Serverless BI dashboards and visualisation | "dashboards", "visualise" |
| Amazon OpenSearch Service | Search and log analytics with dashboards | "full-text search", "log analytics" |
| AWS Data Exchange | Find and subscribe to third-party datasets | "buy/subscribe to external data" |
| AWS Data Pipeline | Legacy data workflow service (in maintenance mode) | Prefer Glue, Step Functions or MWAA |
Making analytics fast and cheap
- Columnar formats (Parquet, ORC) — read only needed columns; often far less data scanned than CSV/JSON.
- Compression (Snappy, GZIP).
- Partitioning by commonly filtered columns (e.g.
year=2026/month=08/) — Athena scans only relevant partitions. - Larger files rather than millions of tiny ones.
- Workgroups and query limits in Athena to control cost.
- Spot instances for EMR task nodes.
Key idea
"Convert CSV to Parquet and catalog it, serverless" → Glue. "Run SQL on that data in S3 with no infrastructure" → Athena. "Control who can see which columns" → Lake Formation. "Dashboards" → QuickSight.
Athena vs Redshift vs EMR
| Choose | When |
|---|---|
| Athena | Ad-hoc, intermittent queries on S3; no infrastructure; pay per query |
| Redshift | Frequent, complex BI queries with high concurrency and predictable performance on structured data |
| EMR | Custom big-data processing with Spark/Hadoop, ML pipelines, full control over frameworks |
Securing a data lake
- S3 encryption (SSE-KMS), Block Public Access, bucket policies.
- Lake Formation permissions and tag-based access control.
- VPC endpoints for private access; CloudTrail data events for auditing.
- Macie to discover sensitive data.
Exam patterns
- "Analysts want to query JSON logs in S3 occasionally, minimal cost and management" → Glue crawler + Athena (convert to Parquet to cut cost).
- "Daily ETL jobs written in Spark, serverless" → Glue jobs (or EMR Serverless).
- "Enterprise data warehouse for BI tools" → Redshift.
- "Give the finance team access only to certain columns of the data lake" → Lake Formation.
- "Interactive dashboards for executives" → QuickSight.
Written by Bipin Singh
Work with meSenior Full-Stack Engineer · AI & AWS. I design and run production systems on AWS — serverless, data and AI.