IAM Roles, Groups, and Policies for Data Access

IAM Roles, Groups, and Policies for Data Access – AWS Data Engineer Associate Guide

Why IAM Roles, Groups, and Policies for Data Access Matter

Identity and Access Management (IAM) is the backbone of security in AWS. For data engineers, controlling who can access what data — and how — is a critical responsibility. Misconfigured IAM policies can lead to data breaches, unauthorized access, compliance violations, and operational disruptions. On the AWS Data Engineer Associate exam, a solid understanding of IAM is essential because nearly every AWS data service (S3, Redshift, Glue, Athena, Kinesis, DynamoDB, etc.) relies on IAM for authorization and authentication.

What Are IAM Roles, Groups, and Policies?

IAM Users
An IAM user represents a person or application that interacts with AWS. Each user has unique credentials. However, managing permissions at the individual user level does not scale well — this is where groups and roles come in.

IAM Groups
A group is a collection of IAM users. You attach policies to a group, and every user in that group inherits those permissions. For example, you might create a DataEngineers group with permissions to access S3 buckets, Glue jobs, and Redshift clusters. Groups simplify permission management by allowing you to assign and revoke access at scale rather than per user.

Key points about groups:
- A user can belong to multiple groups.
- Groups cannot be nested (a group cannot contain another group).
- Groups are not identities — you cannot reference a group in a resource-based policy's Principal element.

IAM Roles
A role is an IAM identity with specific permissions that can be assumed by trusted entities — users, applications, or AWS services. Roles do not have long-term credentials (passwords or access keys). Instead, when a role is assumed, AWS provides temporary security credentials via AWS Security Token Service (STS).

Common use cases for roles in data engineering:
- Service Roles: An AWS Glue job assumes a role to read from S3 and write to Redshift. A Lambda function assumes a role to process records from a Kinesis stream.
- Cross-Account Roles: A data engineer in Account A assumes a role in Account B to access data stored in Account B's S3 buckets.
- EC2 Instance Profiles: An EC2 instance running an ETL script assumes a role via an instance profile to access DynamoDB tables.
- Federated Access: External users (e.g., from a corporate directory) assume roles via SAML or Web Identity Federation to access AWS data resources.

IAM Policies
Policies are JSON documents that define permissions. They specify which actions are allowed or denied on which resources under what conditions.

Types of policies:
- AWS Managed Policies: Pre-built policies maintained by AWS (e.g., AmazonS3ReadOnlyAccess, AWSGlueServiceRole).
- Customer Managed Policies: Custom policies you create and manage for your specific use cases.
- Inline Policies: Policies embedded directly into a single user, group, or role. Use sparingly — they are harder to manage at scale.
- Resource-Based Policies: Attached to resources like S3 buckets, SQS queues, or KMS keys. They specify who (which principal) can perform actions on that resource.
- Service Control Policies (SCPs): Used with AWS Organizations to set permission guardrails across accounts.
- Permission Boundaries: An advanced feature that sets the maximum permissions an IAM entity can have, even if broader policies are attached.

How IAM Works for Data Access

Policy Evaluation Logic
Understanding how AWS evaluates policies is critical:
1. By default, all requests are implicitly denied.
2. An explicit Allow in a policy overrides the implicit deny.
3. An explicit Deny always overrides any Allow — deny always wins.
4. If SCPs, permission boundaries, or session policies are in play, the effective permissions are the intersection of all applicable policy types.

Policy Structure (JSON)
A typical IAM policy contains:
- Version: Always use "2012-10-17".
- Statement: One or more permission statements, each containing:
  - Effect: "Allow" or "Deny"
  - Action: The API actions (e.g., "s3:GetObject", "glue:StartJobRun")
  - Resource: The ARN(s) of the resources the actions apply to
  - Condition: (Optional) Conditions under which the policy applies (e.g., IP address, MFA, encryption requirements, tags)

Example: Granting a Glue Job Access to Specific S3 Paths
You create a role called GlueETLRole with a policy that allows:
- s3:GetObject and s3:PutObject on arn:aws:s3:::my-data-lake/raw/* and arn:aws:s3:::my-data-lake/processed/*
- glue:* on relevant Glue resources
- logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents for CloudWatch logging

The Glue job's configuration specifies this role. When the job runs, it assumes the role and receives temporary credentials scoped to only those permissions.

Practical Scenarios for Data Engineers

1. Least Privilege for Data Pipelines
Always grant the minimum permissions needed. Instead of giving a Lambda function s3:* on all buckets, restrict it to the specific bucket and prefix it needs, and only the actions required (e.g., s3:GetObject).

2. Cross-Account Data Access
Account A owns an S3 bucket. Account B's Glue jobs need to read from it. Solution: Create a role in Account A with an S3 access policy and a trust policy that allows Account B to assume it. Alternatively, use S3 bucket policies to grant cross-account access directly.

3. Tag-Based Access Control (ABAC)
Use IAM policy conditions with resource tags. For example, allow data engineers to access only Glue jobs tagged with Environment=Production and Team=DataEngineering. This scales well as resources grow.

4. S3 Access Points
For large data lakes, S3 Access Points simplify access management by creating named network endpoints with dedicated access policies, reducing the complexity of a single monolithic bucket policy.

5. Lake Formation Integration
AWS Lake Formation provides a higher-level abstraction over IAM for data lake permissions. It uses IAM roles but adds table-level and column-level access controls for Glue Data Catalog resources. Understand that Lake Formation permissions work in conjunction with IAM — you need both to be correctly configured.

6. Redshift Spectrum and Athena
When Redshift Spectrum or Athena queries data in S3, they use IAM roles to authorize access. The role must have permissions to read the S3 data and access the Glue Data Catalog for metadata.

7. KMS Integration
If your data is encrypted with AWS KMS, the IAM role accessing the data must also have kms:Decrypt (and possibly kms:GenerateDataKey) permissions on the relevant KMS key. This is a common exam scenario — access denied errors often result from missing KMS permissions.

Key Differences to Remember

- Users vs. Roles: Users have long-term credentials; roles provide temporary credentials. For AWS services and cross-account access, always prefer roles.
- Identity-Based vs. Resource-Based Policies: Identity-based policies are attached to users, groups, or roles. Resource-based policies are attached to resources (S3 buckets, SQS queues, KMS keys, etc.). For cross-account access, resource-based policies allow the principal to access the resource without assuming a role (and the principal retains permissions from their own account).
- Permission Boundaries vs. SCPs: Permission boundaries limit the maximum permissions of a single IAM entity. SCPs limit the maximum permissions for an entire AWS account or organizational unit.

Exam Tips: Answering Questions on IAM Roles, Groups, and Policies for Data Access

1. Default Deny: Remember that everything is denied by default. If a question says a service cannot access a resource, check whether there is an Allow policy — and whether an explicit Deny is overriding it.

2. Explicit Deny Wins: If any applicable policy contains an explicit Deny, the action is denied regardless of any Allow statements. This is one of the most tested concepts.

3. Roles Over Users for Services: Whenever a question involves an AWS service (Glue, Lambda, EMR, Redshift, etc.) accessing another resource, the answer almost always involves an IAM role, not an IAM user or access keys. Never hardcode credentials.

4. Cross-Account Access Patterns: Know the two main approaches — (a) a role in the target account with a trust policy, or (b) a resource-based policy on the target resource granting access to the source account's principal. Understand when each is appropriate.

5. Least Privilege Principle: When choosing between multiple policy options, the correct answer is almost always the one that grants the narrowest set of permissions needed to accomplish the task.

6. Condition Keys: Be familiar with common condition keys such as aws:SourceIp, aws:PrincipalOrgID, s3:prefix, aws:RequestedRegion, kms:ViaService, and tag-based conditions. Questions may test whether you can restrict access based on conditions.

7. KMS + IAM Together: If data is encrypted, the role needs both the data service permissions (e.g., s3:GetObject) and the KMS permissions (e.g., kms:Decrypt). A missing KMS permission is a classic "access denied" trap in exam questions.

8. Service-Linked Roles: Some services (e.g., Redshift, EMR) use service-linked roles that AWS creates and manages. Know that these exist and that they cannot be modified arbitrarily.

9. Groups Cannot Be Principals: If a question asks about granting access via a resource-based policy to a "group," that is incorrect. Resource-based policies reference IAM users, roles, accounts, or federated users — not groups.

10. Lake Formation vs. IAM: If a question involves fine-grained access to data lake tables or columns, Lake Formation is likely the answer. If it involves service-to-service access or infrastructure-level permissions, IAM roles and policies are the answer.

11. Temporary Credentials and STS: Understand that AssumeRole, AssumeRoleWithSAML, and AssumeRoleWithWebIdentity are STS API calls. Temporary credentials have an expiration, which is a security benefit over long-term access keys.

12. Read the Policy Carefully: Exam questions may present a policy JSON and ask you to identify the problem. Look for incorrect ARNs, missing actions, overly broad wildcards, wrong Effect values, or missing Condition blocks.

13. PassRole Permission: When a user or service needs to assign (pass) a role to another service (e.g., assigning a role to a Glue job), the entity doing the passing needs iam:PassRole permission. This is a commonly tested edge case.

14. Eliminate Answers with Hardcoded Credentials: Any answer choice that involves embedding access keys, secret keys, or passwords in code, configuration files, or environment variables is almost always wrong. The correct approach uses IAM roles.

Summary

IAM Roles, Groups, and Policies form the foundation of data access control on AWS. As a data engineer, you must design secure, scalable access patterns using least privilege, temporary credentials, and the right combination of identity-based and resource-based policies. For the exam, focus on understanding policy evaluation logic, role-based access for services, cross-account patterns, KMS integration, and the principle of least privilege. These concepts appear repeatedly across multiple question types and are essential to passing the AWS Data Engineer Associate certification.