Here's an article I dug up about how ACE masks work:
Access control in Windows 2000: Tying it all together
By Paul Cooke
15 Nov 2004 | Realtimepublishers.com
Get a glimpse inside Paul Cooke's e-book "The definitive guide to Windows 2000 security" with this series of book excerpts, courtesy of Realtimepublishers.com. This excerpt is from Chapter 5, "Configuring access control." Click for the book excerpt series or get the full e-book.
Tying it all together
I've now covered all the pieces that go into making an access control decision in Windows 2000. Remember, the single goal of an access control decision is to determine whether a security principal is authorized to perform some action on some object. Windows 2000 handles access control decisions by considering the following three pieces of information:
The security principal's access token.
The security principal's desired access mask.
The object's security descriptor.
I've talked very briefly about access masks as they relate to ACEs. Well, the security principal's desired access mask is pretty much the same thing. The desired access mask is a simple 32-bit flag structure in which bits are turned on for rights that the security principal wants and turned off for rights that the security principal doesn't want. The SRM manipulates this desired access mask to compute whether access will be allowed or denied. The requested access mask is used to generate a granted access mask.
Initially, the granted access mask is set to all zeros (no access granted). Each requested access right bit is evaluated; if access is allowed, the corresponding bit is set in the granted access mask. If access is allowed or denied, the bit is turned off in the requested access mask. Once all the desired access mask bits are turned off, the granting access mask is returned for use. The bits that are left on in the granted access mask determine which rights the security principal is actually authorized to use.
The complete process for determining a security principal's authorizations is summarized in the following five steps:
1. If there is no DACL in the object's security descriptor, the security principal receives all access to the object that the security principal has requested.
2. If no access bits are set in the desired access mask, the security principal receives no access to the object.
3. If the right to access the SACL is set in the desired access mask, the security principal's access token is consulted to see whether the Manage Auditing and Security Log privilege is present. If it is, the bit for the SACL is set in the granted access mask; otherwise, it's not set.
4. If the desired access mask indicates that the security principal wants access to Read Permissions, Change Permissions or Modify Owner, the security descriptor is used to compare the Owner SID of the object with one of the SIDs in the security principal's access token's User SID or Group SIDs fields. If a match is found, the security principal is granted access; otherwise, the bits in the granted access mask aren't set.
5. The object's DACL examines each ACE in the object's security descriptor using the following rules:
a. If the inheritance flags of the ACE are marked INHERIT_ONLY, the ACE is skipped.
b. If the SID in the subject's access token doesn't match the SID in the ACE, the ACE is skipped.
c. If the ACE type is access-denied, the rights in the security principal's desired access mask are compared with the ACE's access mask. If there are any matches, the security principal gets no access to the object.
d. If the ACE type is access-allowed, the rights in the security principal's desired access mask are compared with the ACE's access mask. If there are any matches, access in the granted access mask is turned on.
e. If there are any bits still turned on in the desired access mask, access checking continues with the next ACE.
f. If all the ACEs are processed and there are any bits remaining in the desired access mask that haven't been turned off, access is implicitly denied. As a result, all the bits in the granted access mask are turned off, and the security principal gets no access to the object.