How to optimize URL Rewriting in scans and job for large sites
How can you optimize URL Rewriting for large sites in scans of IBM Security AppScan Standard and jobs of IBM Security AppScan Enterprise?
Web application developers often use a technology called URL Rewriting to hide parameters in the directory structure. Imagine that an entertainment site uses the following rewrite rule.
RewriteRule ^biography/(.+).jsp biography.jsp?artist=$1
This rule tells the web server to convert the URL that you see in the web browser, such as: http:// www.site.com/biography/madonna.jsp
to the following:
The main reason behind URL rewriting is to force Google and other search engines to index all the pages of the site. Another advantage of URL Rewriting is that questions marks and equal signs are removed from the URL making it easy to remember. The whole transformation is entirely hidden from the user.
The problem posed by URL Rewriting for AppScan is that it renders the redundant path limit useless. The parameters are now part of the path and the product has no way of automatically knowing which is the page and which is the parameter.
If there are ten thousands artists on our entertainment site you will now have ten thousand additional URLs in your scan when you should really have just one. Add to that another URL-rewrited parameter that handles the session and changes its value every time you login and you will now have a never ending scan. If this occurs, AppScan will eventually run out of resources.
How do you identify URL rewriting in AppScan Standard?
- If the scan goes past the 500 URL mark, perform the following:
- Pause the scan
- Choose the Application Data view on the left
- Highlight each folder and look at the number of visited URLs displayed in the "Show" drop-down located at top-center of the screen to find the folder with the most URLs. In our example the folder biography would show ten thousand pages.
- Now that you located the problematic folder, you can check to see if all URLs in this folder follow a specific pattern. In our example you would notice that all the pages in the biography folder have celebrity names.
How do you identify URL rewriting in AppScan Enterprise?
- If the job takes too long to execute and the number of pages scanned is very big in the status screen:
- Save current results and stop. It is very important to select Save current results and stop and not Discard results and stop since only the save option will also run the reports on the data gathered up to this point
- Examine the pages report to identify URL rewriting patterns using a similar process as in AppScan Standard.
Handling of URL rewriting for AppScan Standard
First try to optimize the scan by turning on the optimizer (Explore Optimization Module ) which is included in AppScan Standard version 8.0 and later.
- Information on how to run the optimizer can be found in AppScan Help.
- Complementary information can be found in How to use the Optimizer.
If the scan still shows URL Rewriting after using the optimizer, optimize the scan manually by creating custom parameters as follows:
- The first step is to identify the parameter values in the URL. This can be done by comparing the differences between the URLs that are part of the same folder
The difference between:
http:// www.site.com/biography/madonna.jsp and
http:// www.site.com/biography/britney_spears.jsp is the page name. This difference could be comprised in the following regular expression:
- Once identified the URL-rewrited parameters can be added to the list of Custom Parameters under Scan Configuration > Parameters and Cookies > Custom Parameters
If the example the Custom Parameter definition will be:
Reference Name: artist
Defining the parameter this way will actually allow AppScan Standard to send application type tests to this entity. An example of a Cross-Site scripting attack for this site would look like this:
- After defining the parameter, you need to edit its redundancy settings. To do that, click on the Parameters and Cookies tab and then click on the plus sign.
- In the "Type" drop down choose Custom Parameter and then choose the reference name you just defined.
- Under the redundancy tuning settings at the bottom configure the settings as per in the attached screenshot:
- Explore the URL again whenever it is added or removed
- Repeat all adjacent parameter tests whenever this parameter is added or removed
Handling of URL rewriting for AppScan Enterprise
In the Job Configuration, go to Parameters and Cookies. Expand the section Custom Parameter Definitions (Advanced). Click the '+' to add a new custom parameter in this section as follows:
Check the boxes for:
- Ignore the value of any parameters discovered by this parameter definition, when comparing explore requests
- Do not retest neighboring parameters when the value of any parameters discovered by this parameter definition change
|IBM Security AppScan Enterprise||Configuration||Windows||All Versions|
More support for:
IBM Security AppScan Standard
Software version: All Versions
Operating system(s): Windows
Software edition: Standard
Reference #: 1317594
Modified date: 06 September 2018
Translate this page: