IBM United States
Software Announcement 201-334
November 13, 2001
IBM LoadLeveler for AIX 5L V3 Improves Response Time with IBM Clustering Enhancements
At a Glance
LoadLeveler for AIX 5L, a job management program, allows users to run more jobs in less time by matching the jobs' processing needs with the available resources.
This new version of LoadLeveler for AIX 5L:
For ordering, contact:
Your IBM representative, an IBM Business Partner, or IBM Americas Call Centers at 800-IBM-CALL (Reference: RE001).
LoadLeveler® for AIX® 5L schedules applications to IBM Cluster 1600, RS/6000® SP nodes, or clusters of IBM pSeries or RS/6000 servers. LoadLeveler supports load balancing for interactive Parallel Operating Environment sessions, and focuses on parallel job scheduling and scalability.
This new version of LoadLeveler for AIX 5L:
Planned Availability Dates
LoadLeveler is a job management system that allows users to run more jobs in less time by matching the jobs' processing needs with the available resources. LoadLeveler schedules jobs and provides functions for building, submitting, and processing jobs quickly and efficiently in a dynamic environment. Batch jobs can be submitted, either serial or parallel, to run in the background and require no input from the user. LoadLeveler scheduling algorithms use the availability of consumable resources to determine the machine or machines on which a job will run.
LoadLeveler for AIX 5L Version 3 provides the following functions:
Striped Communication: Parallel jobs can request to use striped communication between nodes. Striping allows the job to use multiple communication paths to the node. Two methods of striping can be used, IP and user space. When a job is submitted using striping, communication is established between nodes that takes advantage of all adapter connections, including the SP Switch2 PCI Attachment Adapter.
AIX Workload Manager Integration: The LoadLeveler administrator can configure LoadLeveler to use the AIX Workload Manager to enforce the resource usage of consumable CPU and consumable memory. In addition, users can obtain Workload Manager statistics on a specific job.
Gang Scheduling: For parallel jobs, LoadLeveler supports Gang scheduling and Backfill scheduling for efficient use of system resources. Gang scheduling provides support for:
Time-sharing involves assigning one or more processors to run multiple applications during a designated time interval. This provides better response to workloads with a mix of short- and long-running jobs. For applications that switch between tasks frequently, coordinated context switching may enhance system performance.
Checkpoint/Restart: Checkpointing allows users to periodically save the state of both serial and parallel jobs automatically. When users enable checkpointing for a job and the job does not complete, it can be restarted from the saved state. When checkpointing is enabled, checkpoints can be initiated from within the application at major milestones, or by the user, by the administrator, or by LoadLeveler external to the application.
64-bit Support: For both interactive and batch jobs, LoadLeveler users and administrators can:
Both sets of 32-bit and 64-bit LoadLeveler APIs and libraries are available for application development. This 64-bit support is essential for running applications with large memory and other resource requirements. The AIX 64-bit kernel is not yet supported by LoadLeveler.
Resource Manager Functions: LoadLeveler incorporates the job management functions, which were previously part of the Parallel System Support Programs (PSSP) Resource Manager. This effectively eliminates the control workstation as a point of failure for parallel jobs and improves the efficiency of managing workloads using the SP switch.
Multiple User Space Tasks Per Adapter: LoadLeveler supports multiple user space tasks on the SP Switch and the SP Switch2.
A parallel job using IP for its communication may consist of up to 2048 tasks.
Multiple Tasks Per Job Per Node: Specified keywords in the job command language allow you to run multiple tasks of a parallel job on the same node.
Multiple Communication Protocols Per Task: LoadLeveler allows a job step to request the network resources required for a task to run both the Message Passing Interface (MPI) and the LAPI communication protocols.
Task to Node Assignment: With added flexibility, a user can specify specific nodes for a job to run based upon keywords provided. Tasks can be assigned in blocks, can be assigned specifically by the user, can be left for LoadLeveler to determine assignment, or a combination.
Process Tracking Function: Allows LoadLeveler to cancel any processes throughout the entire cluster which may have been left behind after a job is terminated consuming resources or degrading performance.
Task Guide Support: A Task Guide is a sequence of panels a user steps through to accomplish a task. The LoadLeveler Task Guide allows modification of LoadLeveler configuration values related to scheduling, such as setting up consumable resources, tuning scheduling and communications keywords, and merging SP machine and adapter information.
Consumable Resources: The use of consumable resources allow users to schedule jobs based on the availability of specific resources. A scheduling option is available to request resources such as CPU for running jobs. Installation defined resources can be specified and considered in scheduling as well. Keywords have been added which can specify which resource to be considered by the LoadLeveler scheduler when dispatching jobs and to specify what resources are available collectively on all machines.
Security: LoadLeveler exploits the Security Services offered in PSSP. With Distributed Computing Environment (DCE), administrators can authenticate user identity so only DCE authenticated users can schedule jobs. In addition, you can delegate credentials so jobs will run with the users DCE authority.
Application Programming Interface Calls: API calls provide the functions which are available through the standalone commands, providing information on parallel jobs originating from a specific node.
Monitoring of Machine Availability:
A daemon is provided to monitor machine availability and notify the
negotiator when a machine is no longer reachable. This daemon provides
quicker and more reliable response with its use of the PSSP Group
Services API (GSAPI).
Accessibility by People with Disabilities
For Software Offerings:
The following feature supports use by people with disabilities:
These programs are not impacted by euro currency.
LoadLeveler is an integral piece of the total System Management solution for the IBM Cluster 1600. Version 3 exploits many of the features in the AIX 5L for POWER Version 5.1 operating system.
LoadLeveler is ideal for users who need to submit jobs as necessary, not according to any pre-set schedule. Other job schedulers that provide a rule-based batch workload management for inter-job dependencies and scheduling jobs at predetermined schedules can be integrated with LoadLeveler function. In particular, AIX 5L Workload Manager is integrated with LoadLeveler Version 3.
LoadLeveler supports load balancing for interactive Parallel Operating Environment sessions and focuses on parallel job scheduling and scalability.
LoadLeveler is fully supported and serviced with continuing development and resource. It is an integrated part of the IBM Cluster 1600 System Management solution.
IBM Software Maintenance Information
IBM Software Maintenance is a 1-year or 3-year contract offering that enables customers to maintain software product version and release currency, with voice support during normal business hours included at no additional charge for the first year, for selected AIX® and UNIX®-based software products at a predetermined cost. One year of Software Maintenance is included in the price of the original license. For products participating in IBM Software Maintenance, customers can receive all software product version and release upgrades and voice support, announced during the contract period. If customers allow either the one-year or three-year software maintenance offering to expire without renewal, there will be an "Maintenance-after-license" option to enable reinstatement in the Software Maintenance offering, on a product by product basis.
The following is general information regarding the Software Maintenance offering being announced:
Definition of Software Upgrade: An upgrade is either a new version (3.x.x to 4.x.x) or a new release (x.3.x to x.4.x) of an IBM program announced as eligible for which a customer enrolled under the IBM Software Maintenance. Software Maintenance entitles customers to future upgrades from only the latest announced version or release of an available eligible product.
Upgrades do not include programs sold separately from the enrolled program, but enhance the performance of that program, or are considered add-ons or utilities. Programs that perform the same functions as the enrolled program but are installed on a non-AIX platform are not considered upgrades.
Note: Software preventative-maintenance releases x.x.1 to x.x.2 (modification level changes, updates, or refreshes) are released under the 1-year or 3-year maintenance program numbers but only when a supply feature has been announced. Supply features are only announced under the 1-year and 3-year options when there is an announced version/release change against the original program product. No-charge preventative maintenance releases should be obtained by Software Maintenance customers in the same manner as non-Software Maintenance customers.
Example: Product A is offered under IBM Software Maintenance and three months later PTFs are incorporated into the program package. This is considered an update that would not be obtained under the 1-year or 3-year maintenance program numbers under the IBM Software Maintenance offering. Using the same example above, three months after the PTFs were incorporated into the program package, IBM announces a new version of product A. At that time, a supply feature is announced for product A under the 1-year and 3-year maintenance program numbers. This allows customers who are entitled by purchase or when the first year of maintenance is included with the license, of either the 1-year or 3-year maintenance offering for product A to order the upgrade under the 1-year or 3-year offering.
IBM may announce a product that can be substituted for an enrolled product. This product is neither a new version, a new release, nor a preventative maintenance release of that enrolled product. In most cases, this product has a different architecture, more function, and a higher license charge than the enrolled product. IBM may provide those customers enrolled in the IBM Software Maintenance offering access to this product at a discounted rate. The terms of this transaction will be included in the new product announcement.
Note: IBM will not offer stand-alone upgrades as a standard product offering. Standalone upgrades to this product/release, and its future versions and releases, will be considered only on a special request basis by IBM.
Contract Duration: The IBM Software Maintenance contracts are on a product by product basis and are offered for one or three years. One year of Software Maintenance is included with the purchase of a new license for the product. The 3-year prepay option covers two additional years of coverage since the first year is included in the product license. The 3-year prepay option can be purchased at the same time a license for an eligible product is purchased. Refunds do not apply if cancellation occurs during the one or three year contract period. Customers are not required to renew under either the one-year or three-year offering when the contract expires but will need to use the Maintenance-after-license option should they let the contract expire and want to acquire a new version or release of an eligible product. If the one-year or three-year contracts aren't renewed, the one or three year contracts will be removed from IBM administrative systems. Contract transfers within an enterprise are allowed.
Payment Options: Customers enroll in the plan at a reduced "Initial" price at the same time a new license for an eligible program product is purchased. Customers can also enroll in the plan after purchase of a new license for an eligible program product but must purchase the "Maintenance-after-license" option.
Software Maintenance Prices: The IBM Software Maintenance program prices are based on the published price of the software program at the time of the contract preparation.
Minimum Contract Value: None
Notification of Upgrade Availability and Ordering: Changes are currently underway to the Web server where customers will register for e-mail notifications when Software Maintenance products are upgraded. Availability of this service is expected first quarter 2002. The Software Maintenance component of the Web server will be found at:
A customer ordering a new license for LoadLeveler® for AIX 5L, LP product number 5765-E69, must include a 1-year (5771-LLV) or a 3-year (5773-LLV) maintenance order to be entitled to future version or release updates of the licensed program. Not only will customers be entitled to future version or release updates, this new offering includes a year of voice support at no additional cost. Customers entering a 1-year or 3-year Software Maintenance offering for LoadLeveler, should specify type/model 5771-LLV or 5773-LLV. After selecting the 1-year or 3-year type/model, customers should add the applicable billing features with a quantity equal to the number of licenses in the contract.
Product information will be available on day of announcement through Offering Information (OITOOL) at:
No hardcopy publications are shipped with this program.
The following publications are offered in displayable softcopy form:
These displayable manuals are part of the basic machine-readable material and are viewable in PDF and HTML formats.
Terms and conditions for use of the machine-readable files are shipped with the files.
The Publication Notification System (PNS) is available by order number/product number. Customers currently subscribing to PNS will automatically receive notifications by e-mail. Customers who wish to subscribe, can visit the PNS Web site at:
The IBM Publications Center Portal:
The Publications Center is a worldwide central repository for IBM product publications and marketing material with a catalog of 70,000 items. Extensive search facilities are provided, as well as payment options via credit card. Furthermore, a large number of publications are available online in various file formats, which can currently be downloaded free of charge.
Specified Operating Environment
Compatibility: The Gang scheduling option will not function until all nodes in the system have been upgraded to the latest version of LoadLeveler. Configuration and administration files that specify new Gang scheduling keywords will not be compatible with previous versions of LoadLeveler.
LoadLeveler 3.1 can coexist with LoadLeveler 2.2 in a mixed cluster; some machines in the cluster have LoadLeveler 3.1 installed while the rest of the cluster is running LoadLeveler 2.2. However, there are restrictions to running in a mixed cluster and new Version 3.1 functions will be available in a mixed cluster. In addition, when running a mixed 2.2 and 3.1 cluster, the LoadLeveler 2.2 machines must have 18.104.22.168 (U478841) or higher installed. Refer to the Migration Considerations sections of the Install Memo for IBM LoadLeveler and IBM LoadLeveler for AIX 5L, Using and Administering, Version 3 Release 1 publications.
LoadLeveler 2.2 operates with 32-bit data only, while LoadLeveler 3.1 operates with both 32-bit and 64-bit. Since 2.2 cannot interpret 64-bit data, LoadLeveler converts all 64-bit integers originating from 3.1 commands, daemons, or APIs into 32-bit integers before sending the data to a 2.2 target.
Support for the Checkpoint/Restart function will be available via PTF. Refer to APARs IY24116 and IY24117 for information on availability and support of this function.
The SP Switch adapter and the RS/6000 SP System Attachment adapter are not supported with Gang scheduling.
Support on the pSeries 690 is initially limited to non-PSSP stand-alone servers with a maximum of 16 operating system images and 128 processors.
LoadLeveler 3.1 provides support for 64-bit applications that run on POWER nodes with AIX 5L 5.1. This 64-bit support is essential for running applications with large memory and other resource requirements. However, the AIX 64-bit kernel is not yet supported by LoadLeveler.
The LoadLeveler function allowing only DCE authenticated users to use
LoadLeveler to schedule jobs is not available without PSSP security
services. One of the configurable options for delegation of DCE
credentials, so that jobs run with the user's DCE authority, does not
require PSSP and is not limited.
Security, Auditability, and Control
LoadLeveler uses the security and auditability features of the AIX 5L Operating System
The customer is responsible for evaluation, selection, and implementation of security features, administrative procedures, and appropriate controls in application systems and communication facilities.
LoadLeveler for AIX 5L, Version 3 has two charge units. Charge units for SP based products are based on nodes; charge units for workstations and servers are based on the number of physical installations of the product (for example, installation on a system or LPAR partition). These installations are available in variable order quantities based on the number of workstations or servers with which LoadLeveler will be installed. On the SP, LoadLeveler is shipped with the hardware. When adding a LoadLeveler license to a cluster, order using the tier equal to the total quantity in the cluster. Upgrades may be acquired via the IBM Software Maintenance program up to the current level of use authorized for the qualifying programs.
Program Name: LoadLeveler for AIX 5L, V3.1
New users of LoadLeveler Version 3.1 should specify Type 5765, Model E69, asset feature number 9001, and the feature number equal to the total number of desired licenses to be installed times the quantity of installs. For example, to order 12 licenses of LoadLeveler, specify the feature number for Order Type Description 11 to 50 Installs, with a quantity of 12.
Note: Charge units for SP based products are based on nodes; charge units for workstations and servers are based on the number of physical installations of the product (for example, installation on a system or LPAR partition).
Machine Type/Model: 5765-E69 OTC Order Type Feature Feature Description Number Number Medium 1 to 5 Installs 0001 5809 CD-ROM 6 to 10 Installs 0002 11 to 50 Installs 0003 51 or greater Installs 0004 Per RS/6000 SP Node 0005
OTC = One-Time Charge
Software Subscription Option for AIX
Customers who purchased upgrade protection for LoadLeveler V2 (5765-D61), under Software Subscription Option for AIX (5692-SSO), are entitled to receive Version 3.1 at no additional charge. Eligible customers should add CD-ROM media supply feature number 0596 under their existing 5692-SSO record. LoadLeveler for AIX 5L is offered on CD-ROM media only.
5692-SSO Eligible Media Billing Feature Supply Annual Subscription Feature Description Charge(1) CD-ROM From 5765-D61 V2.2 to 0596 5765-E69 V3.1 1 to 5 Installations 6499 6503 6504 6 to 10 Installations 6500 6505 6506 11 to 50 Installations 6501 6507 6508 51 or More Installations 6502 6509 6510
Note: Software Subscription Option for AIX (5692-SSO) is no longer an offering. The replacement offering is called IBM Software Maintenance.
General Information on IBM Software Maintenance
The IBM Software Maintenance offering was previously announced. Refer to Software Announcement 201-201 , dated July 10, 2001.
The following are some key points from that announcement.
Note: The following term differs from Software Announcement 201-201 : Stand-alone upgrades to this product/release, and its future versions and releases, will be considered only on a special request basis from IBM.
Ordering Information for IBM Software Maintenance
Current Licensees of LoadLeveler V2.2
Customers with the current level of LoadLeveler for AIX (5765-D61), who are interested in IBM Software Maintenance, and do not have an active AIX Software Subscription or AIX/UNIX Upgrade Protection contract, should select the appropriate feature from the table below.
Customers interested in ordering an upgrade to LoadLeveler for AIX 5L (5765-E69) who do not have an active AIX Software Subscription or AIX/UNIX Upgrade Protection contract, should contact their IBM representative or IBM reseller and request an order be placed for either the 1-year (5771-LLV) or 3-year (5771-LLV) Software Maintenance offering and specify the supply feature and appropriate Maintenance-after-license feature from the table below.
New Licensees of LoadLeveler
The software license for new customers of LoadLeveler for AIX 5L (5765-E69) will include IBM Software Maintenance for one year. Customers should then select the 1-year or 3-year program number, the appropriate feature number, and desired quantity to extend coverage for their software licenses.
Note: Please note the following for new licenses:
OTC Charge 1-Year Software Maintenance 5771-LLV
Maint. After- Current Registration Renewal License Licensee (NC) Billing Billing Supply Feature Feature Feature Feature Program Name Number Number Number Number LoadLeveler (5765-E69) 5809 1 to 5 Installs 0062 0067 0072 6 to 10 Installs 0063 0068 0073 11 to 50 Installs 0064 0069 0074 51 or More Installs 0065 0070 0075 Per RS/6000 SP Node 0066 0071 0076
OTC 3-Year Software Maintenance 5773-LLV
Maint. After- Current Renewal License Licensee Registration Billing Billing Supply Feature Feature Feature Feature Program Name Number Number Number Number LoadLeveler (5765-E69) 5809 1 to 5 Installs 0061(2) 0066 0071 6 to 10 Installs 0062(2) 0067 0072 11 to 50 Installs 0063(2) 0068 0073 51 or More Installs 0064(2) 0069 0074 Per RS/6000 SP Node 0065(2) 0070 0075
Withdrawal of Software Subscription for AIX Feature Numbers
The following feature numbers are withdrawn as of April 8, 2002.
Program Feature Description Number Number Software Subscription for 5692-SSO 6499 6503 6504 LoadLeveler 2.2 6500 6505 6506 6501 6507 6508 6502 6509 6510
Customization Options for 5765-E69: Select the appropriate feature numbers to customize your order with delivery options desired. These features can be specified on the initial or MES orders.
Feature Description Number Initial Shipments Serial Number Only (suppresses initial 3444 shipment of documentation) Expedite Shipments Local IBM Office Expedite 3445 (for IBM use only) Customer Expedite Process Charge 3446 ($30 charge for each product)
Expedite shipments will be processed to receive 72-hour delivery from the time IBM Software Delivery and Fulfillment (SDF) receives the order. SDF will then ship the order via overnight air transportation.
Terms and Conditions
The prices provided in this announcement are suggested retail prices for the U.S. only and are provided for your information only. Dealer prices may vary, and prices may also vary by country. Prices are subject to change without notice. For additional information and current prices, contact your local IBM representative.
Program Name: LoadLeveler for AIX 5L, Version 3
OTC Machine Feature Description Type/Model Number OTC 1 to 5 Installs 5765-E69 0001 $2,306 6 to 10 Installs 0002 2,231 11 to 50 Installs 0003 1,862 51 or greater Installs 0004 1,107 Per RS/6000 SP Node 0005 0
OTC 1-Year Software Maintenance 5771-LLV
Maint. Registra- After- tion Renewal License NC Billing Billing Feature Feature Feature Program Name Number OTC Number OTC Number OTC LoadLeveler (5765-E69) 1 to 5 Installs 0062 NC 0067 $513 0072 $1,025 6 to 10 Installs 0063 NC 0068 496 0073 992 11 to 50 Installs 0064 NC 0069 414 0074 828 51 or More Installs 0065 NC 0070 246 0075 492 Per RS/6000 SP Node 0066 NC 0071 246 0076 492
OTC 3-Year Software Maintenance 5773-LLV
Maint. Registra- After- tion Renewal License NC Billing Billing Feature Feature Feature Program Name Number OTC Number OTC Number OTC LoadLeveler (5765-E69) 1 to 5 Installs 0061(3) $1,025 0066 $1,538 0071 $2,563 6 to 10 Installs 0062(3) 992 0067 1,487 0072 2,479 11 to 50 Installs 0063(3) 828 0068 1,241 0073 2,069 51 or More Installs 0064(3) 492 0069 738 0074 1,230 Per RS/6000 SP Node 0065(3) 492 0070 738 0075 1,230
Customer Financing: IBM Global Financing offers attractive financing to credit-qualified commercial and government customers and Business Partners in more than 40 countries around the world. IBM Global Financing is provided by the IBM Credit Corporation in the United States. Offerings, rates, terms, and availability may vary by country. Contact your local IBM Global Financing organization. Country organizations are listed on the Web at:
Use Priority/Reference Code: RE001 Phone: 800-IBM-CALL Fax: 800-2IBM-FAX Internet: firstname.lastname@example.org Mail: IBM Atlanta Sales Center Dept. RE001 P.O. Box 2690 Atlanta, GA 30301-2690
You can also contact your local IBM Business Partner or IBM representative. To identify them, call 800-IBM-4YOU.
Note: Shipments will begin after the planned availability date.