How to optimize URL Rewriting in scans and job for large sites

Technote (FAQ)


Question

How can you optimize URL Rewriting for large sites in scans of IBM Security AppScan Standard and jobs of IBM Security AppScan Enterprise and IBM Rational Policy Tester?

Cause

Web application developers often use a technology called URL Rewriting to hide parameters in the directory structure. Imagine that an entertainment site uses the following rewrite rule.
RewriteRule ^biography/(.+).jsp biography.jsp?artist=$1

This rule tells the web server to convert the URL that you see in the web browser, such as: http:// www.site.com/biography/madonna.jsp

to the following: http:// www.site.com/biography.jsp?artist=madonna


The main reason behind URL rewriting is to force Google and other search engines to index all the pages of the site. Another advantage of URL Rewriting is that questions marks and equal signs are removed from the URL making it easy to remember. The whole transformation is entirely hidden from the user.

The problem posed by URL Rewriting for AppScan is that it renders the redundant path limit useless. The parameters are now part of the path and the product has no way of automatically knowing which is the page and which is the parameter.

If there are ten thousands artists on our entertainment site you will now have ten thousand additional URLs in your scan when you should really have just one. Add to that another URL-rewrited parameter that handles the session and changes its value every time you login and you will now have a never ending scan. If this occurs, AppScan will eventually run out of resources.


How do you identify URL rewriting in AppScan Standard?

  • If the scan goes past the 500 URL mark, perform the following:
    1. Pause the scan
    2. Choose the Application Data view on the left
    3. Highlight each folder and look at the number of visited URLs displayed in the "Show" drop-down located at top-center of the screen to find the folder with the most URLs. In our example the folder biography would show ten thousand pages.
    4. Now that you located the problematic folder, you can check to see if all URLs in this folder follow a specific pattern. In our example you would notice that all the pages in the biography folder have celebrity names.


How do you identify URL rewriting in AppScan Enterprise and Policy Tester?

  • If the job takes too long to execute and the number of pages scanned is very big in the status screen:
    1. Save current results and stop. It is very important to select Save current results and stop and not Discard results and stop since only the save option will also run the reports on the data gathered up to this point
    2. Examine the pages report to identify URL rewriting patterns using a similar process as in AppScan Standard.

Answer

Handling of URL rewriting for AppScan Standard


First try to optimize the scan by turning on the optimizer (Explore Optimization Module) which is included in AppScan Standard version 8.0 and later.

  • Information on how to run the optimizer can be found in AppScan Help.
  • Complementary information can be found in How to use the Optimizer.


If the scan still shows URL Rewriting after using the optimizer, optimize the scan manually by creating custom parameters as follows:

  1. The first step is to identify the parameter values in the URL. This can be done by comparing the differences between the URLs that are part of the same folder

    Example:

    The difference between:
    http:// www.site.com/biography/madonna.jsp and
    http:// www.site.com/biography/britney_spears.jsp is the page name. This difference could be comprised in the following regular expression:
    biography/(.+)\.jsp 

  2. Once identified the URL-rewrited parameters can be added to the list of Custom Parameters under Scan Configuration > Parameters and Cookies > Custom Parameters

    If the example the Custom Parameter definition will be:

    Reference Name: artist
    Pattern: biography/(.+)\.jsp
    Location: Path



    Defining the parameter this way will actually allow AppScan Standard to send application type tests to this entity. An example of a Cross-Site scripting attack for this site would look like this:
    http://www.site.com/biography/.jsp
  3. After defining the parameter, you need to edit its redundancy settings. To do that, click on the Parameters and Cookies tab and then click on the plus sign.

  4. In the "Type" drop down choose Custom Parameter and then choose the reference name you just defined.
    Under the redundancy tuning settings at the bottom configure the settings as per in the attached screenshot:

    Check:
    • Explore the URL again whenever it is added or removed
    • Repeat all adjacent parameter tests whenever this parameter is added or removed
    Uncheck:
    • Explore the URL again whenever the value of this parameter changes
    • Repeat all adjacent parameter tests whenever the value of this parameter changes







Handling of URL rewriting for AppScan Enterprise and Policy Tester (5.6x or later)

In the Job Configuration, go to Parameters and Cookies. Expand the section Custom Parameter Definitions (Advanced) . Click the '+' to add a new custom parameter in this section as follows:




Check the boxes for:

  • Ignore the value of any parameters discovered by this parameter definition, when comparing explore requests
  • Do not retest neighboring parameters when the value of any parameters discovered by this parameter definition change

Cross reference information
Segment Product Component Platform Version Edition
Security Security AppScan Enterprise Configuration 5.6, 8.0, 8.5, 8.6, 8.7.0.0, 8.8
Security Rational Policy Tester Configuration 5.6, 8.0, 8.5

Rate this page:

(0 users)Average rating

Document information


More support for:

IBM Security AppScan Standard
Configuration

Software version:

8.0, 8.5, 8.6.0.0, 8.7, 8.8

Operating system(s):

Windows

Software edition:

Standard

Reference #:

1317594

Modified date:

2012-10-05

Translate my page

Machine Translation

Content navigation