By specifying credentials in the administration console, you can enable the Web crawler to access restricted content, such as documents that require a password for access.
If a Web server uses HTTP basic authentication or HTML form-based authentication to restrict access to Web sites, you can specify credentials in the Web crawler's configuration that enable pages on the password-protected Web sites to be crawled. You can also specify options for manually configuring cookie files.