Troubleshooting
Problem
A user is getting a 403 Forbidden response but ACLs on the object are correct. Policy seems to be correct on: /WebSEAL/isam9.level2.org-default/jct/hello.html and the ACL allows read. How can I diagnose this?
Symptom
Unexpected 403 Forbidden response.
Cause
Incorrect authorization rules.
Diagnosing The Problem
Check settings and collect traces if necessary.
Resolving The Problem
Before collecting any traces check the following:
##
## Make sure the correct objectspace is being used.
##
Check in the Reverse Proxy config file:
[server]
#server-name = isam9.level2.org-default
server-name = Other
At restart WebSEAL registers with the Policy Server to pull policy from /WebSEAL/<server-name>. In this case it will use /WebSEAL/Other.
Update <server-name> if not correct and restart.
It might also be the cause that /WebSEAL/Other is the desired objectspace. In this case the policy needs to be moved.
If <server-name> has been verified but a 403 is still returned continue.
##
## Make sure the Policy Server is replicating policy updates automatically.
##
Check the Policy Server conf file:
[ivmgrd]
auto-database-update-notify = no
If set to no then you will have to issue:
pdadmin sec_master> server replicate
Replicate the Policy and retest.
Regardless of setting check the Policy Server message log. The replication might fail due to changes in network or some other reason. You will see a message similar to:
2017-04-27-08:15:49.272-05:00I----- 0x13279384 pdmgrd NOTICE idb notifier IVNotifier.cpp 536 0x7f70ccdfa700
HPDDB0900I Could not send client notification (default-webseald-isam9.level2.org, 0x1354a426).
Can you ping isam9.level2.org from the Policy Server?
Check the WebSEAL message log to see if there is a problem downloading
the latest policy db.
Is this the latest copy as compared to the Policy Server's copy?
/var/PolicyDirector/db/master_authzn.db?
Check by date or just use the Linux sum command if in doubt.
Software locations are:
Policy Server: /var/PolicyDirector/db/master_authzn.db
WebSEAL: /var/pdweb/db/webseald-default.db
Appliance locations in the support file are:
Policy Server: /var/PolicyDirector/db/master_authzn.db
WebSEAL: /var/pdweb/default/db/webseald-default.db
If none of the above resolve the issue then continue on to traces.
##
## Collect traces
##
Enable pdweb.debug, pdweb.snoop, and pdweb.wan.azn traces. Steps are located here:
Appliance: http://www-01.ibm.com/support/docview.wss?uid=swg21663410
Software: https://www-304.ibm.com/support/docview.wss?uid=swg27013204
All three traces are required. Complete information from the above must be collected
Recreate the problem. In the pdweb.wan.azn trace you will see the access decision for the resource.
You can find by looking for search string "INPUTS". For example, debugging access for /jct/hello.html.
2017-04-27-10:00:22.624-06:00I----- thread(9) trace.pdweb.wan.azn:9 /pro
ject/amwebsec900/build/amwebsec900/src/pdwebrte/webcore/amw_azn.c:257:
[192.168.61.1] INPUTS -
protected_resource=/WebSEAL/isam9.level2.org-default/jct/hello.html,
operation=r
2017-04-27-10:00:22.624-06:00I----- thread(9) trace.pdweb.wan.azn:9 /pro
ject/amwebsec900/build/amwebsec900/src/pdwebrte/webcore/amw_azn.c:278:
[192.168.61.1] OUTPUT - permission=1
2017-04-27-10:00:22.654-05:00I----- thread(9) trace.pdweb.wan.azn:9 /project/amwebsec900/build/amwebsec900/src/pdwebrte/webcore/amw_azn.c:147: [192.168.61.1] Attr Name:AZN_PERMINFO_FAIL_REASON , Value: 268809264
The "permissions=1" means denied.
Check the reason code using:
pdadmin sec_master> errtext 268809264
HPDAC1072E The step-up authorization policy on the protected object has denied access.
We see in this case step-up policy has denied access. So, the ACL might be okay but a POP is also in affect.
*** NOTE ***
If the following error is seen:
pdadmin sec_master> errtext 268809260
HPDAC1068E Access to the protected object was denied by EAS override.
pdadmin sec_master>
This means a custom External Authentication Service denied access. This is normally Advanced Access Control Policy. Debugging the Deny for AAC is outside the scope of this document.
If the above error is seen then normal AAC debugging should be used.
##
## Check the object policy all the way to the top. For example,
##
pdadmin sec_master> object show
/WebSEAL/isam9.level2.org-default/jct/hello.html
pdadmin sec_master> object show /WebSEAL/isam9.level2.org-default/jct
pdadmin sec_master> object show /WebSEAL/isam9.level2.org-default
...
It is very import to go all the way to "/".
At each object show check the attached or effective ACL, POP, or AuthzRule.
Does the user have the 'T' perm all the way down to hello.html? If not, then access will stop as soon as there is no 'T' down the path.
##
## Is the Junction case sensitive?
##
Another common problem is the junction is case-insensitive. This means even if the access was /jct/HELLO.html the policy needs to be on /jct/hello.html. This is easy to overlook when debugging.
##
## Does the user have access?
##
Is the user in a group the gets denied? Even if the user is a member of a group that grants access it will get denied. Policy is not a union in cases such as this.
##
## Is the user coming from an IP that is restricted by the POP? We see in this case:
##
pdadmin sec_master> pop show ip-test
Protected object policy: ip-test
Description:
Warning: No
Audit level: none
Quality of protection: none
Time of day access: sun, mon, tue, wed, thu, fri, sat, :anytime:local
IP Endpoint Authentication Method Policy
Auth Level: Forbidden Network: Any Other Network
Auth Level: 1 Network: 192.168.100.0
Netmask: 255.255.255.0
Access is only allowed from the 192.168.100.0 network, but the above access came from 192.168.61.1.
There could also be a time-of-day restriction.
##
## AuthzRule check.
##
What is the AuthzRule checking? Retrieve the rule and examine:
pdadmin sec_master> authzrule show azn_engine_target_resource
Authorization Rule Name: azn_engine_target_resource
Description:
Rule Text: <xsl:choose>
<xsl:when test="azn_cred_groups = 'WebAccess' and azn_engine_target_resource = '/WebSEAL/isam9.level2.org-default /jct/hello.html'">
!TRUE!
</xsl:when>
<xsl:otherwise>
!FALSE!
</xsl:otherwise>
</xsl:choose>
We see in this case despite having access via the ACL there is also a check to make sure the user is only a member of the WebAccess group.
##
## Check DNYURL in pdweb.debug.
##
Fine the request in pdweb.debug:
2017-05-09-12:52:23.850-05:00I----- thread(15) trace.pdweb.debug:2 /project/amwebsec900/build/amwebsec900/src/pdweb/webseald/ras/trace/debug_log.cpp:134: ----------------- Browser ===> PD -----------------
Thread_ID:1953863424^M
GET /jct/hello.html HTTP/1.1^M
2017-05-09-12:52:23.851-05:00I----- thread(15) trace.pdweb.debug:2 /project/amwebsec900/build/amwebsec900/src/pdweb/webseald/ras/trace/debug_log.cpp:256: ----------------- DYNURL -----------------
Thread_ID:1953863424
DYNURL:/jct/hello.html REGEXP:/jct/hello.html OBJECT:/Deny
----------------- END DYNURL -----------------
In this case a DYNURL was created of:
/Deny /jct/hello.html
Check the permissions on /WebSEAL/isam9.level2.org-default/Deny:
pdadmin sec_master> object show /WebSEAL/isam9.level2.org-default/Deny
...
Attached ACL: closed-acl
...
pdadmin sec_master> acl show closed-acl
ACL Name: closed-acl
Description:
Entries:
User sec_master TcmdbsvaBRl
We see the ACL does not allow access.
##
## Make sure the correct objectspace is being used.
##
Check in the Reverse Proxy config file:
[server]
#server-name = isam9.level2.org-default
server-name = Other
At restart WebSEAL registers with the Policy Server to pull policy from /WebSEAL/<server-name>. In this case it will use /WebSEAL/Other.
Update <server-name> if not correct and restart.
It might also be the cause that /WebSEAL/Other is the desired objectspace. In this case the policy needs to be moved.
If <server-name> has been verified but a 403 is still returned continue.
##
## Make sure the Policy Server is replicating policy updates automatically.
##
Check the Policy Server conf file:
[ivmgrd]
auto-database-update-notify = no
If set to no then you will have to issue:
pdadmin sec_master> server replicate
Replicate the Policy and retest.
Regardless of setting check the Policy Server message log. The replication might fail due to changes in network or some other reason. You will see a message similar to:
2017-04-27-08:15:49.272-05:00I----- 0x13279384 pdmgrd NOTICE idb notifier IVNotifier.cpp 536 0x7f70ccdfa700
HPDDB0900I Could not send client notification (default-webseald-isam9.level2.org, 0x1354a426).
Can you ping isam9.level2.org from the Policy Server?
Check the WebSEAL message log to see if there is a problem downloading
the latest policy db.
Is this the latest copy as compared to the Policy Server's copy?
/var/PolicyDirector/db/master_authzn.db?
Check by date or just use the Linux sum command if in doubt.
Software locations are:
Policy Server: /var/PolicyDirector/db/master_authzn.db
WebSEAL: /var/pdweb/db/webseald-default.db
Appliance locations in the support file are:
Policy Server: /var/PolicyDirector/db/master_authzn.db
WebSEAL: /var/pdweb/default/db/webseald-default.db
If none of the above resolve the issue then continue on to traces.
##
## Collect traces
##
Enable pdweb.debug, pdweb.snoop, and pdweb.wan.azn traces. Steps are located here:
Appliance: http://www-01.ibm.com/support/docview.wss?uid=swg21663410
Software: https://www-304.ibm.com/support/docview.wss?uid=swg27013204
All three traces are required. Complete information from the above must be collected
Recreate the problem. In the pdweb.wan.azn trace you will see the access decision for the resource.
You can find by looking for search string "INPUTS". For example, debugging access for /jct/hello.html.
2017-04-27-10:00:22.624-06:00I----- thread(9) trace.pdweb.wan.azn:9 /pro
ject/amwebsec900/build/amwebsec900/src/pdwebrte/webcore/amw_azn.c:257:
[192.168.61.1] INPUTS -
protected_resource=/WebSEAL/isam9.level2.org-default/jct/hello.html,
operation=r
2017-04-27-10:00:22.624-06:00I----- thread(9) trace.pdweb.wan.azn:9 /pro
ject/amwebsec900/build/amwebsec900/src/pdwebrte/webcore/amw_azn.c:278:
[192.168.61.1] OUTPUT - permission=1
2017-04-27-10:00:22.654-05:00I----- thread(9) trace.pdweb.wan.azn:9 /project/amwebsec900/build/amwebsec900/src/pdwebrte/webcore/amw_azn.c:147: [192.168.61.1] Attr Name:AZN_PERMINFO_FAIL_REASON , Value: 268809264
The "permissions=1" means denied.
Check the reason code using:
pdadmin sec_master> errtext 268809264
HPDAC1072E The step-up authorization policy on the protected object has denied access.
We see in this case step-up policy has denied access. So, the ACL might be okay but a POP is also in affect.
*** NOTE ***
If the following error is seen:
pdadmin sec_master> errtext 268809260
HPDAC1068E Access to the protected object was denied by EAS override.
pdadmin sec_master>
This means a custom External Authentication Service denied access. This is normally Advanced Access Control Policy. Debugging the Deny for AAC is outside the scope of this document.
If the above error is seen then normal AAC debugging should be used.
##
## Check the object policy all the way to the top. For example,
##
pdadmin sec_master> object show
/WebSEAL/isam9.level2.org-default/jct/hello.html
pdadmin sec_master> object show /WebSEAL/isam9.level2.org-default/jct
pdadmin sec_master> object show /WebSEAL/isam9.level2.org-default
...
It is very import to go all the way to "/".
At each object show check the attached or effective ACL, POP, or AuthzRule.
Does the user have the 'T' perm all the way down to hello.html? If not, then access will stop as soon as there is no 'T' down the path.
##
## Is the Junction case sensitive?
##
Another common problem is the junction is case-insensitive. This means even if the access was /jct/HELLO.html the policy needs to be on /jct/hello.html. This is easy to overlook when debugging.
##
## Does the user have access?
##
Is the user in a group the gets denied? Even if the user is a member of a group that grants access it will get denied. Policy is not a union in cases such as this.
##
## Is the user coming from an IP that is restricted by the POP? We see in this case:
##
pdadmin sec_master> pop show ip-test
Protected object policy: ip-test
Description:
Warning: No
Audit level: none
Quality of protection: none
Time of day access: sun, mon, tue, wed, thu, fri, sat, :anytime:local
IP Endpoint Authentication Method Policy
Auth Level: Forbidden Network: Any Other Network
Auth Level: 1 Network: 192.168.100.0
Netmask: 255.255.255.0
Access is only allowed from the 192.168.100.0 network, but the above access came from 192.168.61.1.
There could also be a time-of-day restriction.
##
## AuthzRule check.
##
What is the AuthzRule checking? Retrieve the rule and examine:
pdadmin sec_master> authzrule show azn_engine_target_resource
Authorization Rule Name: azn_engine_target_resource
Description:
Rule Text: <xsl:choose>
<xsl:when test="azn_cred_groups = 'WebAccess' and azn_engine_target_resource = '/WebSEAL/isam9.level2.org-default /jct/hello.html'">
!TRUE!
</xsl:when>
<xsl:otherwise>
!FALSE!
</xsl:otherwise>
</xsl:choose>
We see in this case despite having access via the ACL there is also a check to make sure the user is only a member of the WebAccess group.
##
## Check DNYURL in pdweb.debug.
##
Fine the request in pdweb.debug:
2017-05-09-12:52:23.850-05:00I----- thread(15) trace.pdweb.debug:2 /project/amwebsec900/build/amwebsec900/src/pdweb/webseald/ras/trace/debug_log.cpp:134: ----------------- Browser ===> PD -----------------
Thread_ID:1953863424^M
GET /jct/hello.html HTTP/1.1^M
2017-05-09-12:52:23.851-05:00I----- thread(15) trace.pdweb.debug:2 /project/amwebsec900/build/amwebsec900/src/pdweb/webseald/ras/trace/debug_log.cpp:256: ----------------- DYNURL -----------------
Thread_ID:1953863424
DYNURL:/jct/hello.html REGEXP:/jct/hello.html OBJECT:/Deny
----------------- END DYNURL -----------------
In this case a DYNURL was created of:
/Deny /jct/hello.html
Check the permissions on /WebSEAL/isam9.level2.org-default/Deny:
pdadmin sec_master> object show /WebSEAL/isam9.level2.org-default/Deny
...
Attached ACL: closed-acl
...
pdadmin sec_master> acl show closed-acl
ACL Name: closed-acl
Description:
Entries:
User sec_master TcmdbsvaBRl
We see the ACL does not allow access.
Related Information
[{"Product":{"code":"SSZU8Q","label":"IBM Security Access Manager"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Reverse Proxy","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB24","label":"Security Software"}}]
Product Synonym
ISAM
Was this topic helpful?
Document Information
Modified date:
14 April 2020
UID
swg22002857