POSIX ACLs for Data Lake Storage Gen2: Complete Guide for DP-203
Why Are POSIX ACLs Important for Data Lake Storage Gen2?
In enterprise data environments, controlling who can access specific files and directories is critical for data governance, compliance, and security. Azure Data Lake Storage (ADLS) Gen2 implements a hierarchical namespace on top of Azure Blob Storage, and with it comes the ability to use POSIX-like Access Control Lists (ACLs). This is essential because:
- Fine-grained access control: Unlike container-level or account-level RBAC, POSIX ACLs allow you to set permissions at the individual file and directory level.
- Regulatory compliance: Many industries require strict data access controls, and POSIX ACLs help meet these requirements by restricting data access to only authorized identities.
- Defense in depth: POSIX ACLs work alongside Azure RBAC, providing an additional layer of security. RBAC is evaluated first; if RBAC grants access, ACLs are not evaluated. If RBAC does not grant access, ACLs are then evaluated.
- Hadoop ecosystem compatibility: POSIX ACLs are the standard permission model used across Hadoop ecosystems (HDFS), making ADLS Gen2 a natural fit for big data workloads.
What Are POSIX ACLs?
POSIX ACLs (Portable Operating System Interface Access Control Lists) are a permission mechanism that follows Unix/Linux-style access control conventions. In ADLS Gen2, every file and directory has an associated ACL that determines which users, groups, or service principals can perform specific operations.
Key Concepts:
1. Owning User: The user who created the file or directory. The owning user always has permissions that can be configured.
2. Owning Group: The group associated with the file or directory.
3. Named Users: Specific Azure AD users assigned explicit permissions.
4. Named Groups: Specific Azure AD groups assigned explicit permissions.
5. Other: All users who are not the owning user, owning group, named users, or named groups.
6. Mask: Limits the maximum permissions for named users, named groups, and the owning group. It acts as a ceiling on the effective permissions for these entries.
Permission Types:
POSIX ACLs use three standard permissions:
- Read (r): For files, allows reading the content. For directories, allows listing the contents.
- Write (w): For files, allows writing or appending. For directories, allows creating child items and deleting child items.
- Execute (x): For files, this has no meaning in ADLS Gen2 (it is not used). For directories, allows traversing (navigating through) the directory to reach child items.
Two Types of ACLs:
- Access ACLs: Control access to a specific object (file or directory). Every file and directory has an access ACL.
- Default ACLs: Templates of ACLs associated with a directory that determine the access ACLs for any new child items created under that directory. Files do not have default ACLs. Default ACLs are essentially an inheritance mechanism.
How Do POSIX ACLs Work in ADLS Gen2?
1. ACL Evaluation Order:
When a user (security principal) attempts to perform an operation on a file or directory:
- Azure RBAC is evaluated first. If an RBAC role assignment grants the required permission (e.g., Storage Blob Data Contributor), access is allowed and ACLs are not checked.
- If RBAC does not grant access, the POSIX ACL is evaluated.
- The ACL check proceeds in this order: Owning User → Named Users → Owning Group / Named Groups → Other.
- The mask is applied to named users, named groups, and the owning group to compute effective permissions.
2. The Super User:
The storage account owner and any identity with the Storage Blob Data Owner RBAC role is considered a super user and bypasses all ACL checks.
3. Inheritance via Default ACLs:
When a new child item (file or directory) is created under a parent directory:
- The child's access ACL is initialized from the parent's default ACL.
- If the child is a directory, it also inherits the parent's default ACL as its own default ACL.
- If the child is a file, it receives an access ACL from the parent's default ACL but does not receive a default ACL (files never have default ACLs).
Important: Default ACLs are NOT retroactively applied. If you change the default ACL on a directory, existing child items are not affected. Only newly created items inherit the updated default ACL. To update existing items, you must recursively update their access ACLs.
4. The Sticky Bit:
The sticky bit is an advanced feature that can be set on directories. When enabled, it ensures that a child item can only be deleted or renamed by:
- The owning user of the child item
- The owning user of the parent directory
- A super user
This is particularly important in shared directories (e.g., /tmp) where multiple users may write files, preventing users from deleting each other's files.
5. Umask:
The umask is a value set on the client application that controls how default permissions are applied when new files and directories are created. The umask subtracts permissions from the inherited ACL. In ADLS Gen2, the umask for files is typically 006 and for directories is 000, but this depends on the client.
6. Recursive ACL Operations:
ADLS Gen2 supports recursive ACL updates, which allow you to set, update, or remove ACLs across an entire directory tree. This is crucial for:
- Applying new security policies to existing data
- Onboarding new users or groups who need access to existing directory structures
- Revoking access from users or groups across large datasets
Recursive operations can be performed using Azure CLI, PowerShell, SDKs (.NET, Java, Python), or REST APIs. They support continuation tokens for resuming interrupted operations on large datasets.
Practical Example:
Consider a data lake with the following structure:
/data/finance/reports/2024/q1-report.csv
To allow a user analyst@company.com to read q1-report.csv, you need:
- Execute (x) on /data, /data/finance, /data/finance/reports, /data/finance/reports/2024 (all parent directories for traversal)
- Read (r) on q1-report.csv
This is a critical point: without execute permission on every parent directory in the path, the user cannot reach the file, even if they have read permission on the file itself.
Setting ACLs:
ACLs can be managed through:
- Azure Portal (Storage Explorer)
- Azure CLI (az storage fs access set)
- Azure PowerShell (Set-AzDataLakeGen2ItemAclObject)
- REST API
- SDKs (.NET, Java, Python, JavaScript)
- Azure Storage Explorer (desktop application)
ACL Entry Format:
ACL entries follow this format:
[scope:]type:id:permissions
- scope: "default" for default ACLs (omitted for access ACLs)
- type: user, group, mask, or other
- id: Azure AD object ID (omitted for owning user, owning group, and other)
- permissions: rwx (or a subset, using - for denied, e.g., r-x)
Example: user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r-x
Limits:
- A maximum of 32 ACL entries per access ACL and 32 per default ACL are recommended as best practice. The hard limit is 32 entries including the owning user, owning group, mask, and other.
- Named users and named groups share the 28 remaining slots (32 minus 4 mandatory entries).
RBAC vs. POSIX ACLs — When to Use What:
- Use Azure RBAC when you need broad access control (e.g., a service principal that needs access to all data in a storage account or container).
- Use POSIX ACLs when you need fine-grained access control at the directory or file level.
- Use a combination: RBAC for service accounts and data engineers, ACLs for restricting analyst access to specific datasets.
Common Pitfalls:
- Forgetting to set execute permission on all parent directories when granting access to a deeply nested file.
- Confusing default ACLs (which affect future children) with access ACLs (which affect the current item).
- Assuming that changing a default ACL retroactively updates existing children (it does not).
- Not accounting for the mask, which can limit effective permissions even when explicit ACL entries grant broader access.
- Forgetting that Storage Blob Data Owner role bypasses ACLs entirely.
Exam Tips: Answering Questions on POSIX ACLs for Data Lake Storage Gen2
1. Remember the RBAC-first evaluation: If a question asks whether a user can access a file and the user has been assigned the Storage Blob Data Contributor or Storage Blob Data Reader RBAC role, the answer is YES — ACLs are not even checked. Only the Storage Blob Data Owner role provides super user access that bypasses everything.
2. Execute permission on directories is key: Exam questions often test whether you understand that traversing directories requires execute (x) permission on each directory in the path. If a question describes a user with read on a file but no execute on a parent directory, the user cannot access the file.
3. Default ACLs vs. Access ACLs: If the question asks about setting permissions that should apply to future items created in a directory, the answer involves default ACLs. If the question is about current access, it involves access ACLs.
4. No retroactive inheritance: If a question mentions changing a default ACL and asks whether existing files are affected, the answer is no. You must use recursive ACL updates to apply changes to existing items.
5. Hierarchical namespace required: POSIX ACLs only work when the hierarchical namespace (HNS) is enabled on the storage account. If a question mentions Blob Storage without HNS, POSIX ACLs are not available.
6. Mask limits effective permissions: If an exam question provides ACL entries and a mask, calculate the effective permissions by performing a bitwise AND between the granted permissions and the mask. For example, if a named user has rwx but the mask is r-x, the effective permission is r-x.
7. Sticky bit for shared directories: If a question asks how to prevent users from deleting each other's files in a shared directory, the answer is the sticky bit.
8. 32 ACL entry limit: If a question asks about scaling access control to hundreds of users, the answer is to use Azure AD security groups rather than individual user entries, due to the 32-entry limit.
9. Know the tools: Recursive ACL operations can be performed using Azure CLI, PowerShell, and SDKs. The Azure Portal supports viewing and editing ACLs on individual items but is not ideal for bulk recursive operations.
10. Shared Key and SAS bypass ACLs: Access via Shared Key authentication or account-level SAS tokens bypasses POSIX ACL checks entirely. Exam questions may test this — only Azure AD (OAuth 2.0) authentication respects POSIX ACLs. If you want ACLs to be enforced, ensure clients authenticate with Azure AD.
11. Write permission on a directory: To create or delete files within a directory, you need write (w) AND execute (x) on that directory. Questions may test this combination.
12. Read on directories vs. files: Read (r) on a directory allows listing its contents. Read (r) on a file allows reading the file's data. These are frequently tested distinctions.
