Select the Standard tab for upgrade instructions using the iFix installation script.

Select the Manual tab for the instructions on how to manually upgrade each component.

Important Notice: Ensure your environment has met all the requirements in the Prerequisites section above before applying this iFix.

1.) Stop SmartCloud Orchestrator on Central Server 1

Change to directory: /iaas/scorchestrator/
Stop SmartCloud Orchestrator: ./SCOrchestrator.py --stop
===>>> Stopping Smart Cloud Orchestrator
...
===>>> Stopping Smart Cloud Orchestrator complete

Wait until all components have stopped

2.) Backup your SCO installation

Backup the Central Servers and Region Server virtual machines:
for VMware hosted virtual machines, review Taking Snaphots for information about taking snapshots of the virtual machines.

3.) Copy 2.3.0.1-CSI-ISCO-IF0005.tar on Central Server 1

4.) Extract 2.3.0.1-CSI-ISCO-IF0005.tar

Extract 2.3.0.1-CSI-ISCO-IF0005.tar in a directory of your choice <your_dir> by running the following command:

Note: Do not place the package under /root path

tar -xf 2.3.0.1-CSI-ISCO-IF0005.tar

The following files are extracted:

SmartCloud_Orchestrator_2.3.0.1_IF0005.fxtag
   ifix5.py
   0001-enable-multiple-keystone-all-worker-processes.patch              
   PSIRT1876.patch                          
   iaasgateway-2013.1-1.1.4.ibm.201409112157.noarch.rpm
   rtc-184008-keystone-11c387264-ifix-el6.tar.gz
   8.5.0.0-WS-BPMPCPD-IFJR48696.zip                          
   iaasgateway_http.cfg                                  
   keystone_zz00267.patch
   smartcloud-2013.1-1.1.3.ibm.201407310025.noarch.rpm
   smartcloud-2013.1-1.1.4.ibm.201407310043.noarch.rpm
   helper.py                                                
   IFIX005_IWD_SCO2301_20140902-0433-971.tar
   install_script                  
     -9mjhzl_iaasgateway.py
     -9mjhzl_keystone.py  
     -bpm_jr48696.py  
     -helper.py  
     -iwd_fix.py
     -psirt_1838.py  
     -psirt_1876.py  
     -zz00267.py
   2.3.0.1-CSI-ISCO-IF0004
     -3.1.0.4-IBM-SCE-FP004-201407010432.zip
     -3.1.0.4-IBM-SCE-IF001-201407170051.zip
     -helper.py
     -ibm-java-i386-sdk-6.0-16.0.i386.rpm
     -ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
     -IBM_SmartCloud_Provisioning_Core_2.3.0.1_IF0004-2.3.0.1.fxtag
     -ICCT_Install_2.3.0.1-17.zip
     -ifix4.py
     -il_install_package_23025.zip
     -il_proxy_install_package_23025.zip  
     -openstack.tar
     -SP7_rpm_list
     -sce310_jre_linux_installer.bin
     -SmartCloud_Orchestrator_2.3.0.1_IF0004.fxtag
     -install_script
       -create_new_repo.py
       -helper.py
       -iwd_fix.py
       -java632sr16fp1.py
       -java664sr16fp1.py
       -sce310fp4.py
       -sce310fp4fix.py
       -sce310java664sr16fp1.py
       -update_pack.py
       -vilp23025.py
       -vils23025.py
       -was80javasdk32-ifpi19109.py
       -was85javasdk64-ifpi19108.py
     -2.3.0.1-CSI-ISCO-IF0003
       -helper.py
       -ifix3.py
       -SmartCloud_Orchestrator_2.3.0.1_IF0003.fxtag
       -SE59389
         -se59389_fix.py
         -se59389_fix.tar.gz
       -scp-pdcollect
         -Components_nonroot.xml
         -Components.xml
         -Environment.xml
         -PDCOLLECT_Central_Server_Template.xml
       -ZZ00201
         -iwd0002_0006checksum.txt
         -iwd0002_0006.py
         -opt/ibm/.../maestro.util.jar
       -ZZ00234
         -automation.groovy
       -ZZ00240
         -00_SCOrchestrator_Toolkit_2301_20140403_ifix1.twx
       -ZZ00242
         -config_network.sh
       -ZZ00244_ZZ00246
         -plugin.com.ibm.orchestrator.rest-1.0.1.1.jar

5.) Upgrading the Image Construction and Composition Tool (ICCT)

- Review the topic Upgrading the Image Construction and Composition Tool in the IBM Knowledge Center for details.

- In step "Download and extract the compressed file to the computer where you want to upgrade or install the Image Construction and Composition Tool fix pack" use ICCT_Install_2.3.0.1-17.zip

6.) Prepare the 32-bit Java installation used by the VIL on Central Server 2

- In "Security Bulletin: Multiple vulnerabilities in current IBM SDK for Java for WebSphere Application Server April 2014 CPU" click on link named "PI19109" in section "For 8.0.0.0 through 8.0.0.8:"

- In "PI19109: SHIP SDK 626 SR8 FOR WSAS V8.0.0.X" you need to download the correct package (Linux 32-bit x86 AMD/Intel Java SDK) by clicking the FC link which lets you download the following package:

8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 (94943549 Bytes)

- When downloaded, copy 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 to the Central Server 1, in the ifix directory <your_dir>/2.3.0.1-CSI-ISCO-IF0004/.

7.) Prepare the 64-bit Java installation used by the Business Process Manager on Central Server 4

- In "Security Bulletin: Multiple vulnerabilities in current IBM SDK for Java for WebSphere Application Server April 2014 CPU" click on link named "PI19108" in section "For V8.5.0.0 through 8.5.5.2 Full Profile:"

- In "PI19108: SHIP SDK 626 SR8 FOR WSAS V8.5.0.X AND V8.5.5.X" you need to download the correct package (Linux 64-bit x86 AMD/Intel Java SDK) by clicking the FC link which lets you download the following package:

8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 (103391758 Bytes)

- When downloaded, copy 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 to the Central Server 1, in the ifix directory <your_dir>/2.3.0.1-CSI-ISCO-IF0004/.

8.) Run the iFix installation script on Central Server 1

- From the directory where you unpacked the iFix package, run the iFix installation script by using the following command:

./ifix5.py --cs2=<cs2> --cs3=<cs3> --cs4=<cs4> --rs=<regionservers> --cn=<compute_node> --wasa=wasadmin --wasp=passw0rd

Example: ./ifix5.py --cs2=10.10.0.12 --cs3=10.10.0.13 --cs4=10.10.0.14 --rs=10.10.0.15,10.10.0.16 --cn=10.10.0.17 --wasa=wasadmin --wasp=passw0rd

You can use ./ifix5.py -h to show the usage.
Options:
    -h, --help          show this help message and exit
    --cs2=CS2           hostname/ip address of central server 2
    --cs3=CS3           hostname/ip address of central server 3
    --cs4=CS4           hostname/ip address of central server 4
    --rs=RS             list of hostnames/ip addresses of region servers format
                        server1,server2,server3,...
    --cn=CN             (Optional)list of hostnames/ip addresses of compute node format
                        compute1,compute2,compute3,...
    --wasa=WASADMIN     virtual image library WAS administrator ID to be used
                        during the installation procedure
    --wasp=WASPASSWORD  virtual image library WAS administrator password

You can refer to log file ifix5.log in the script path for more detail information of the installation process.
Other log information:
VIL install/upgrade log: /tmp/fresh_install_vil.log, upgrade_vil.log on Central Server 2
VIL Proxy install/upgrade log: /tmp/fresh_install_proxy.log, upgrade_proxy.log on Region Server
IWD fix install log: /tmp/iwdifix5.log Central Server 3

Note: If you have already applied ifix4 manually in your environment, manually create a file '/home/db2inst1/.db2ifix0004' on Central Server 1 before running script ifix5.py.

9.) Enable IWD notifications --- this step is optional, intended for large environments only (thousands of instances).
To enable the IWD notifications, contact the IBM Customer Support for SmartCloud Orchestrator and request the LA fix 2.3.0.1-CSI-ISCO-LA0012 for PMR 02984,999,000. Once done, you can proceed with the following steps (9.x).

(9.1) Update [DEFAULT] section in /etc/nova/nova.conf on each region server

Put the three lines under section #### QPID ####

notification_driver=nova.openstack.common.notifier.rabbit_notifier
instance_usage_audit = True
notify_on_state_change = vm_and_task_state
instance_usage_audit_period = hour

(9.2) Update /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.openstack.notifications-4.0.0.1/config/zero.config on Central Server 3:

/config/openstack/notifications = {
"general": {
"queue_factory_class":
"org.apache.qpid.jndi.PropertiesFileInitialContextFactory",
"notifications_topic": "nova/notifications.info",
"vm_expiration_sec": 6000,
"vm_load_on_init": true,
"resolving_threads": 5,
"resolving_threads_processing_delay_sec": 5
},
"regions": {
"Region_1_name" : { <------Note: here is the region server name
 "brokerlist" : "tcp://BROKER_1_ADDRESS:5672" <------Note: here is the region server ip
},
                        "Region_2_name" " { <------Note: For each of the region server, need a part like that.
                "brokerlist" : "tcp://BROKER_2_ADDRESS:5672"
}
}
}

Note: The "region server name" can be found in the /root/openrc file of the Region Server, looking at NOVA_REGION_NAME variable

(9.3) Ensure CS-3 can use "telnet $region_server_ip 5672" to connect to region server

Here need to ensure each region server have enable the port 5672 access for cs-3.
There are two ways to enable port 5672 on region server.

Method 1:
On region server:
vim /etc/sysconfig/iptables

add below two lines, right before the iprules of port 5672

-A INPUT -s $cs-3_ip/32 -p udp -m udp --dport 5671:5672 -j ACCEPT
-A INPUT -s $cs-3_ip/32 -p tcp -m tcp --dport 5671:5672 -j ACCEPT

Then run 'iptables-save' and 'service iptables restart'
Try command "telnet $region_server_ip 5672" on cs-3. If it can connect to the region server, then you can restart iwd service.

Method 2:
On region server:
iptables-save >/tmp/iptables.save

Then modify the file /tmp/iptables.save and add two lines like below:
case 1:
-A INPUT -j nova-api-INPUT <---right after this line add below two lines.
-A INPUT -s $cs-3_ip/32 -p udp -m udp --dport 5671:5672 -j ACCEPT
-A INPUT -s $cs-3_ip/32 -p tcp -m tcp --dport 5671:5672 -j ACCEPT
case 2:
If there is no line like "nova-aip-INPUT" in iptables.save, please add the two lines right before the iprules of port 5672.
-A INPUT -s $cs-3_ip/32 -p udp -m udp --dport 5671:5672 -j ACCEPT
-A INPUT -s $cs-3_ip/32 -p tcp -m tcp --dport 5671:5672 -j ACCEPT

Carry out the command to restore iptables:
iptables-restore < /tmp/iptables.save

Then restart iptables.

(9.4) Restart IWD service
On CS-3, use 'service iwd restart' to restart iwd service and wait about 15min for its initialization.

(9.5) Try command "telnet $region_server_ip 5672" on cs-3. If telnet is not installed on cs-3, please run 'yum install telnet' to install it.
You should get a message on the shell like this:

Trying $region_server_ip...
Connected to $region_server_ip.
Escape character is '^]'.

10.) Start SmartCloud Orchestrator on Central Server 1

- Stop SmartCloud Orchestrator on Central Server 1 (to enable a clean start in the following step)
Change to directory: /iaas/scorchestrator/
Stop SmartCloud Orchestrator: ./SCOrchestrator.py --stop
===>>> Stopping Smart Cloud Orchestrator
...
===>>> Stopping Smart Cloud Orchestrator complete

Wait until all components have stopped

- Start SmartCloud Orchestrator on Central Server 1
Change to directory: /iaas/scorchestrator/
Start SmartCloud Orchestrator: ./SCOrchestrator.py --start

Wait until all components have started

- Start iaasgateway cluster service on Central Server 2

Login Central Server 2 and perform below commands to start iaasgateway cluster service

service openstack-iaasgateway stop
Stopping openstack-iaasgateway: [ OK ]

service openstack-iaasgateway01 start
Starting openstack-iaasgateway01: [ OK ]

service openstack-iaasgateway02 start
Starting openstack-iaasgateway02: [ OK ]

service openstack-iaasgateway03 start
Starting openstack-iaasgateway03: [ OK ]

service httpd start
Starting httpd: [ OK ]

11.) Additional manual steps required to install fix for APAR ZZ00240.

Note: This fix is also included in iFix3 (2.3.0.1-CSI-ISCO-IF0003) and higher IFIX versions. If you have already applied 2.3.0.1-CSI-ISCO-IF0003 to your SCO environment, please ignore this part.

Step (11.1). Copy the folder 2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/ZZ00240 to a directory of your choice where the BPM brower is being run from.

Step (11.2). Import toolkit SCOrchestrator_Toolkit provided by this fix

Logon to IBM Process Designer with admin/$your_smartcloud_password

In Process Center, select tab Toolkits and click on toolkit SCOrchestrator_Toolkit

Check that SCOrchestrator_Toolkit version 2301_20140213 is the current version
(top in the stack below "Current")

Click on Toolkits and then on Import Toolkit

In the Import Toolkit dialog, click on Browse... and select 00_SCOrchestrator_Toolkit_2301_20140403_ifix1.twx
provided with this fix and click OK

Click the Import button in dialog Import Toolkit - Will be imported - SCOrchestrator_Toolkit - 2301-20140403_ifix1
and wait for the import to be finished

Click on toolkit SCOrchestrator_Toolkit

Check that SCOrchestrator_Toolkit version 2301_20140403_ifix1 is the current version
(top in the stack below "Current")


Step (11.3). Upgrade dependency of toolkit SCOrchestrator_Scripting_Utilities_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Scripting_Utilities_Toolkit

Check that SCOrchestrator_Scripting_Utilities_Toolkit version 2301_20140226 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1


Step (11.4). Create new snapshot of toolkit SCOrchestrator_Scripting_Utilities_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Scripting_Utilities_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Scripting_Utilities_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11.5). Upgrade dependency of toolkit SCOrchestrator_Support_IaaS_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_IaaS_Toolkit

Check that SCOrchestrator_Support_IaaS_Toolkit version 2301_20140227 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1


Step (11.6). Create new snapshot of toolkit SCOrchestrator_Support_IaaS_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_IaaS_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Support_IaaS_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11.7). Upgrade dependency of toolkit SCOrchestrator_Support_vSys_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_vSys_Toolkit

Check that SCOrchestrator_Support_vSys_Toolkit version 2301_20140310 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (11.8). Create new snapshot of toolkit SCOrchestrator_Support_vSys_Toolkit

In Process Center, click on Toolkits and click on SCOrchestrator_Support_vSys_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that SCOrchestrator_Support_vSys_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11.9). Upgrade dependencies of toolkit TivSAM_Integration_Toolkit

In Process Center, click on Toolkits and click on TivSAM_Integration_Toolkit

Check that TivSAM_Integration_Toolkit version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Manage and select check box "Allow users to update toolkit" to make the toolkit editable

Click on Snapshots and click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Scripting_Utilities_Toolkit (2301_20140226) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (11.10). Create new snapshot of toolkit TivSAM_Integration_Toolkit

In Process Center, click on Toolkits and click on TivSAM_Integration_Toolkit

Click on Create New Snapshot

In the Create New Snapshot dialog, type in the following snapshot name:
2301_20140403_LA0001
Optionally provide an additional description

Click on Create

Check that TivSAM_Integration_Toolkit version 2301_20140403_LA0001 is the
current version (top in the stack below "Current")


Step (11.11). Upgrade dependencies of process app Sample_Support_Iaas_ProcessApp

In Process Center, click on Process Apps and click on Sample_Support_Iaas_ProcessApp

Check that Sample_Support_Iaas_ProcessApp version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (11.12). Upgrade dependencies of process app Sample_Support_vSys_ProcessApp

In Process Center, click on Process Apps and click on Sample_Support_vSys_ProcessApp

Check that Sample_Support_vSys_ProcessApp version 2301_20140310 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_IaaS_Toolkit (2301_20140227) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Support_vSys_Toolkit (2301_20140310) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001


Step (11.13). Upgrade dependencies of process app TSAMITK_SampleApp

In Process Center, click on Process Apps and click on TSAMITK_SampleApp

Check that TSAMITK_SampleApp version 2301_20140305 is the current version
(top in the stack below "Current")

Click on Open in Designer

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Toolkit (2301_20140213) and select "Upgrade dependency to 2301_20140403_ifix1"

Check that the dependency changed to 2301_20140403_ifix1

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Scripting_Utilities_Toolkit (2301_20140226) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

In the tree view on the left hand, right-click the yellow triangle under TOOLKITS -
SCOrchestrator_Integration_Toolkit (2301_20140305) and select "Upgrade dependency to 2301_20140403_LA0001"

Check that the dependency changed to 2301_20140403_LA0001

Please repeat the above described procedure for every toolkit or process application in your process centre.

Step (11.14). Upgrade REST endpoint in database

Logon to Central-Server-1

su - db2inst1
db2 connect to BPMDB
db2 set schema bpmuser
db2 "select guid,substr(default_value,1,29) as default_value,last_modified from lsw_env_var where name='restEndpoint'"

You should see one entry where default value is equals to https://localhost

Update this entry:
db2 "update LSW_ENV_VAR set default_value='https://<Central-Server-3-IP>' where name = 'restEndpoint' and default_value='https://localhost'&quot;

Double-check that the update was successful:

db2 "select guid,substr(default_value,1,29) as default_value,last_modified from lsw_env_var where name='restEndpoint'"

db2 commit
db2 connect reset

Stop/Start the BPM admin server on Central-Server-4
/opt/ibm/BPM/v8.5/profiles/Node1Profile/bin/stopServer.sh SingleClusterMember1
/opt/ibm/BPM/v8.5/profiles/Node1Profile/bin/startServer.sh SingleClusterMember1


12.) Apply fix for APAR ZZ00256.

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0003. If you have already applied 2.3.0.1-CSI-ISCO-IF0003 or higher IFIX version to your SCO environment, please ignore this part.

If your SCO server OS locale is non-English, please perform the following steps to return the correct status of openstack services to SAAM.

On each of the SCO servers (Central-servers, region-servers and KVM compute nodes), find the file 'openstack-servicectrl.sh' in '/home/saam/' or '/home/ <yourmechid>/root/'.

(1) make a backup of openstack-servicectrl.sh
(2) edit 'openstack-servicectrl.sh',

Modify the line
CTRL_CMD="/sbin/service $SERVICE_ID"

to
CTRL_CMD="/etc/init.d/${SERVICE_ID}"
LANG=C

(3) check the output of openstack-servicectrl.sh and ensure it can get the correct status of openstack services.

For example, on central-server 2 run:

$ LANG=C /etc/init.d/openstack-keystone status
keystone (pid 2817) is running...

$ ./openstack-servicectrl.sh openstack-keystone status
root: openstack-servicectrl.sh (28621): openstack-keystone status monitor detected openstack-keystone online.

13.) Apply fix for APAR SE58688 - KVM nodes going offline

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0003. If you have already applied 2.3.0.1-CSI-ISCO-IF0003 or higher IFIX version to your SCO environment, please ignore this part.

KVM compute nodes are going offline and can't deply any new patterns and VMs.
This is a bug of openstack https://bugs.launchpad.net/oslo-incubator/+bug/1211338.

On each kvm region server and compute node, backup the file /usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py
Then change it as bellow.
find the words:
{"type": "Direct"}
modify it to:
{"type": "direct"}


14.) Apply fix for ZZ00242 on Central Server 1.

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0003. If you have already applied 2.3.0.1-CSI-ISCO-IF0003 or higher IFIX version to your SCO environment, please ignore this part.

(1) backup file config_network.sh in $your_sco_install_media/installer/scripts/

(2) Update config_network.sh with the one in folder 2.3.0.1-CSI-ISCO-IF0003/ZZ00242


15.) Apply fix for the DB2 security vulnerabilities CVE-2013-6747 and CVE-2014-0963.

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0003. If you have already applied 2.3.0.1-CSI-ISCO-IF0003 or higher IFIX version to your SCO environment, please ignore this part.

(1) downloading the DB2 fix pack

In order to address security vulnerabilities CVE-2013-6747 and CVE-2014-0963, the steps described in security bulletin here http://www.ibm.com/support/docview.wss?uid=swg21675724 should be implemented.

Clicking the above link brings you to:
"Security Bulletin: Security vulnerabilities have been identified in IBM® DB2® shipped with SmartCloud Orchestrator (CVE-2013-6747, CVE-2014-0963)"

From here click:
"IBM® DB2® is impacted by multiple TLS/SSL security vulnerabilities (CVE-2013-6747, CVE-2014-0963)"
Which brings you to:
http://www-01.ibm.com/support/docview.wss?uid=swg21671732

From here click:
"http://www.ibm.com/support/docview.wss?uid=swg24037557"
Note: This is for version V10.1 FP3a - if you have already upgraded DB2, select the version that matches your environment.
Which brings you to:
"DB2 Version 10.1 Fix Pack 3a for Linux, UNIX, and Windows"

Under the section "Download fix pack images per operating system" expand "Linux 64-bit, x86-64 on AMD64 and Intel EM64T"
Click "DB2 Server Fix Pack"
This brings you to Fix Central and "interim fix: DB2-linuxx64-server-10.1.0.3-FP003a"
Check the box and click Continue
Login to FC if prompted
Under "Download files using https" you will be able to download:
v10.1fp3a_linuxx64_server.tar.gz (733 MB)

(2) Installing the DB2 fix pack on Central Server 1 and each Region Servers (if shared DB is not used)

Ensure the SCO processes have been stopped using SCOrchestrator.py stop as outlined in section III step 1 above
Note: The output of the command /iaas/scorchestrator/SCOrchestrator.py will list the servers that are running DB2. Update each of these servers, starting with central-server-1 and then moving on to each region server.

Perform following steps to install the DB2 fix.
<1> Copy the fix pack to central server 1 to a folder of your choice <db2fixpack>
<2> As user db2inst1 stop DB2 and DAS:
db2stop
/opt/ibm/db2/v10.1/das/bin/db2admin stop
<3> As user root extract and install the DB2 fix pack
cd <db2fixpack>
tar zxvf v10.1fp3a_linuxx64_server.tar.gz
cd server
./installFixPack -n -b /opt/ibm/db2/v10.1 -f db2lib
Please wait for the install to complete.

<4> Re-run the install command to verify the fix pack has been applied
./installFixPack -n -b /opt/ibm/db2/v10.1/ -f db2lib

Need to repeat the steps above on each Region Server which uses standalone DB2.

16.) (Optional) Apply fix for APAR SE59801

Note: This fix can only be applied to the vmware region server which attempts to connect to a vCenter with resource pools defined with similar names but with a difference in upper/lower case.

Please ignore this fix if your region server don't have the above issue.

On the vmware region server:
(1) check smartcloud version: rpm -qa |grep smartcloud
If your smartcoud is 'smartcloud-2013.1-1.1.3.ibm.xxxxxxx.noarch.rpm', please use fix
rpm -Uvh smartcloud-2013.1-1.1.3.ibm.201407310025.noarch.rpm

If your smartcoud is 'smartcloud-2013.1-1.1.4.ibm.xxxxxxx.noarch.rpm', please use fix
rpm -Uvh smartcloud-2013.1-1.1.4.ibm.201407310043.noarch.rpm

(2) Modify the configure file
step 1.Please make sure there is no VMs under resource pools before enable the setting in step 2. Otherwise, OpenStack will fail to manage
those vms after applying this fix.

step 2.Please add the following line into /etc/nova/smartcloud.conf after install the ifix:
filter_out_cloud_resource_pool=True

(3) restart openstack-smartcloud service and ensure it is up and running
service openstack-smartcloud restart
service openstack-smartcloud status

*********************************************************************************************
WARNING: If you ran the script ifix6.py as outlined on the Standard tab and it completed successfully, then most of the steps in this section can be skipped. You will need to refer to the instructions on the Standard tab for additional manual steps which still need to be executed after running the script ifix5.py.

Note: Be sure to execute all the steps described in this section to ensure that any of the fixes in the pile (i.e. iFix3, iFix4 and iFix5) are included.
*********************************************************************************************

1.) Inventory Tagging

To make it easier for support to find out which SmartCloud Orchestrator fixes are installed on your system, copy some files to
/opt/ibm/SmartCloud_Orchestrator/properties/version/ on Central-Server-3.
(if not present on Central-Server-3 create the directory)

scp *.fxtag \
Central-Server-3:/opt/ibm/SmartCloud_Orchestrator/properties/version/

2.) Manual steps to apply IWD fixes on Central Server 3.

Note I: This fix is also included in 2.3.0.1-CSI-ISCO-IF0003. If you have already applied 2.3.0.1-CSI-ISCO-IF0003 or higher version to your SCO environment, please ignore this part.

Note II: This fix depends on IFIX005_IWD_SCO2301_20140902-0433-971/iwd_sco2301_20140902-0433-971.tar and can not be installed separately. This fix will not work without IFIX005_IWD_SCO2301_20140902-0433-971/iwd_sco2301_20140902-0433-971.tar installed.

(2.1) Copy the following files from Central Server 1 to $your_folder on Central Server 3.

2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/SmartCloud_Orchestrator_2.3.0.1_IF0003.fxtag
2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/ZZ00234/automation.groovy
2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/ZZ00244_ZZ00246/plugin.com.ibm.orchestrator.rest-1.0.1.1.jar

(2.2) Stop IWD server on Central Server 3:

service iwd stop

(2.3) Backup the following files on Central Server 3:

mkdir -p /tmp/$your_backup
cd /tmp/$your_backup
cp --preserve /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/maestro.util-4.0.0.1/lib/maestro.util.jar maestro.util.jar_ori
cp --preserve /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/orchestrator.ui-4.0.0.1/app/resources/automation.groovy automation.groovy_ori
cp --preserve /drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.task/1.0.1.0 plugin.com.ibm.orchestrator.rest-1.0.1.1.jar plugin.com.ibm.orchestrator.rest-1.0.1.1.jar_ori

(2.4) Update the files on Central Server 3

cd $your_folder
cp -f maestro.util.jar /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/maestro.util-4.0.0.1/lib/
cp -f automation.groovy /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/orchestrator.ui-4.0.0.1/app/resources/
cp -f plugin.com.ibm.orchestrator.rest-1.0.1.1.jar /drouter/ramdisk2/mnt/raid-volume/raid0/usr/servers/kernelservices/plugins/bundles/com.ibm.orchestrator.task/1.0.1.0/

(2.5) Start IWD server on Central Server 3:

service iwd start

3.) Manual steps to update pdcollect files on Central Server 1.

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0003. If you have already applied 2.3.0.1-CSI-ISCO-IF0003 or higher version to your SCO environment, please ignore this part.

(1) Backup below files in '/iaas/pdcollect' on Central Server 1 (If you are using no-root pdcollect, please find the files in /home/<yourmechid>)
Components_nonroot.xml
Components.xml
Environment.xml
PDCOLLECT_Central_Server_Template.xml

(2) Update the files in /iaas/pdcollect with the ones in folder '2.3.0.1-CSI-ISCO-IF0004/2.3.0.1-CSI-ISCO-IF0003/scp-pdcollect/'


4.) Upgrade Virtual Image Library (VIL) and VIL distributed proxy components on Central Server 2 and each Region server.

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have already applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- Make sure VIL service and VIL proxy service are all stopped

(1) on Central Server 2, check vil service status by running the following command:

service vil status

If vil service is not stopped, run the following command to stop it.

service vil stop

(2) on each Region server, check vilProxy service status by running the following command:

service vilProxy status

If vilProxy service is not stopped, run the following command to stop it.

service vilProxy stop

- Upgrade VIL proxy on each Region Server where it is installed:

(1) copy il_proxy_install_package_23025.zip to the Region Server, in a directory of your choice.

(2) unpack il_proxy_install_package_23025.zip into a temporary directory <your_path>:

unzip il_proxy_install_package_23025.zip -d <your_path>

(3) navigate to <your_path> and upgrade Virtual Image Library proxy by running the following script:

cd <your_path>
./install_vil.sh -proxy -vilServer <hostnameFqdn>

Where
vilServer: Specifies the fully qualified host name or the IP address of the Virtual
Image Library server (central-server-2).
If you specify the fully qualified host name of the Virtual Image Library
server, the proxy must be able to resolve that host name. If it cannot,
add the IP address and host name to /etc/hosts before installing the
proxy.

- Upgrade VIL server and proxy on Central Server 2:

(1) copy il_install_package_23025.zip to Central Server 2, in a directory of your choice.

(2) unpack il_install_package_23025.zip into a temporary directory <your_path>:

unzip il_install_package_23025.zip -d <your_path>

(3) navigate to <your_path> and upgrade Virtual Image Library by running the following script:

cd <your_path>
./install_vil.sh -u wasadmin -p $smartcloud_password

5) Update the Java 64-bit installation on Central Server 1

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have already applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Update to Java 6 SR16 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.0

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"

6) Update the Java 64-bit installation on Central Server 2 (used by Public Cloud Gateway)

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have already applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Copy ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm to Central Server 2,
in a directory of your choice.

- Update to Java 6 SR16 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.0

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"

7.) Install WAS interim fix 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 for VIL on Central Server 2

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have already applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- In "Security Bulletin: Multiple vulnerabilities in current
IBM SDK for Java for WebSphere Application Server April 2014 CPU"
(http://www-01.ibm.com/support/docview.wss?uid=swg21673013) click on link named "PI19109"
in section "For 8.0.0.0 through 8.0.0.8:"

- In "PI19109: SHIP SDK 626 SR8 FOR WSAS V8.0.0.X"
(http://www-01.ibm.com/support/docview.wss?uid=swg24037708) you need to download
the correct package (Linux 32-bit x86 AMD/Intel Java SDK) by clicking the FC link
which lets you download the following package:

8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 (94943549 Bytes)

- When downloaded, copy 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 to the Central Server 2,
in a directory of your choice (<your_dir>).

- Stop VIL server if not already stopped, by running the following command:

service vil stop

- Install the interim fix IFPI19109, by running the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl install 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109 -installationDirectory /opt/IBM/WebSphere/AppServer/
-repositories <your_dir>/8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109.zip

- If the installation completes successfully, the following message is displayed:
"Installed 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI19109_8.0.0.20140602_1627 to the /opt/IBM/WebSphere/AppServer directory."

- If the installation fails with the following message:
"ERROR: An interim fix for the Java SDK is installed already. Uninstall interim fix 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI08995
before installing a different Java SDK interim fix."

the interim fix IFPI08995 must first be uninstalled, by running the following command:
/opt/IBM/InstallationManager/eclipse/tools/imcl uninstall 8.0.0.0-WS-WASJavaSDK-LinuxX32-IFPI08995 -installationDirectory /opt/IBM/WebSphere/AppServer/

and then the interim fix IFPI19109 can be installed as described above.

- Verify that the interim fix has been installed, checking that it is properly listed among the output of the following command:
/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

8) Update the Java 64-bit installation on Central Server 3 (used by SCUI)
Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have alreay applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- Check the Java current version:
/opt/ibm/java-x86_64-60/bin/java -fullversion

- Copy ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm to Central Server 3,
in a directory of your choice.

- Update to Java 6 SR16 if not already installed:
yum localupdate ibm-java-x86_64-sdk-6.0-16.0.x86_64.rpm
...
Updated:
ibm-java-x86_64-sdk.x86_64 0:6.0-16.0

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-x86_64-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"

9.) Update the Java 32-bit installation on Central Server 3 (used by IBM Workload Deployer)

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have already applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- Check the Java current version:
/opt/ibm/java-i386-60/bin/java -fullversion

- Copy ibm-java-i386-sdk-6.0-16.0.i386.rpm to Central Server 3,
in a directory of your choice.

- Update to Java 6 SR16 if not already installed:
yum localupdate ibm-java-i386-sdk-6.0-16.0.i386.rpm
...
Updated:
ibm-java-i386-sdk.i386 0:6.0-16.0

Complete!

- Check that the updated version is Java 6 SR16 FP1:
/opt/ibm/java-i386-60/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxi3260sr16-20140418_01 (SR16)"

- Copy new security (unrestricted) policy files
cp /opt/ibm/java-i386-60/demo/jce/policy-files/unrestricted/*.jar /opt/ibm/java-i386-60/jre/lib/security/

10.) Install WAS interim fix 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 for BPM on Central Server 4

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have already applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- In "Security Bulletin: Multiple vulnerabilities in current
IBM SDK for Java for WebSphere Application Server April 2014 CPU"
(http://www-01.ibm.com/support/docview.wss?uid=swg21673013) click on link named "PI19108"
in section "For V8.5.0.0 through 8.5.5.2 Full Profile:"

- In "PI19108: SHIP SDK 626 SR8 FOR WSAS V8.5.0.X AND V8.5.5.X"
(http://www-01.ibm.com/support/docview.wss?uid=swg24037710) you need to download
the correct package (Linux 64-bit x86 AMD/Intel Java SDK) by clicking the FC link
which lets you download the following package:

8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 (103391758 Bytes)

- When downloaded, copy 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 to the Central Server 4,
in a directory of your choice (<your_dir>).

- Stop BPM if not already stopped, by running the following command (wait for services to stop and confirm with
service bpm status afterwards):

service bpm stop

- Install the interim fix IFPI19108, by running the following command:

/opt/IBM/InstallationManager/eclipse/tools/imcl install 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108 -installationDirectory /opt/ibm/BPM/v8.5
-repositories <your_dir>/8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI19108.zip

- If the installation fails with the following message:
"ERROR: An interim fix for the Java SDK is installed already. Uninstall interim fix 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI08994
before installing a different Java SDK interim fix."

the interim fix IFPI08995 must first be uninstalled, by running the following command:
/opt/IBM/InstallationManager/eclipse/tools/imcl uninstall 8.5.0.0-WS-WASJavaSDK-LinuxX64-IFPI08994 -installationDirectory /opt/ibm/BPM/v8.5

and then the interim fix IFPI19109 can be installed as described above.
- Verify that the interim fix has been installed, checking that it is properly listed among the output of the following command:
/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

11.) Update the 64-bit Java installation used by SmartCloud Entry on each VMware Region Server

Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have already applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- Make sure SmartCloud Entry is stopped on each VMware Region Server
by running the following command:

service sce status
IBM SmartCloud Entry is stopped

- Copy sce310_jre_linux_installer.bin to the VMware Region Server, in a directory of your choice.

- Update to Java 6 SR16 from the sce310_jre_linux_installer.bin downloaded directory:

./sce310_jre_linux_installer.bin

Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...

Launching installer...

===============================================================================
Choose Locale...
----------------

1- Deutsch
->2- English
...

CHOOSE LOCALE BY NUMBER:
===============================================================================
(created with InstallAnywhere)
-------------------------------------------------------------------------------

Preparing CONSOLE Mode Installation...

===============================================================================
Introduction
------------

InstallAnywhere will guide you through the JRE (Java Runtime Environment)
update for IBM SmartCloud Entry.

The JRE version of this update is 1.6.0-SR16.

It is strongly recommended that you quit all programs before continuing with
this installation.

Respond to each prompt to proceed to the next step in the installation. If you
want to change something on a previous step, type 'back'.

You may cancel this installation at any time by typing 'quit'.

PRESS <ENTER> TO CONTINUE:

===============================================================================
License Agreement
-----------------

Installation and use of IBM SmartCloud Entry JRE Update requires acceptance of
the following License Agreement:

International Program License Agreement

...

DO YOU ACCEPT THE TERMS OF THIS LICENSE AGREEMENT? (Y/N): Y

===============================================================================
IBM SmartCloud Entry Install Locations
--------------------------------------

The following IBM SmartCloud Entry installation locations require the JRE
version to be updated to version 1.6.0-SR16 FP1.

Select the install location to proceed with the JRE update.

->1- /opt/ibm/SCE31

ENTER THE NUMBER FOR YOUR CHOICE, OR PRESS <ENTER> TO ACCEPT THE DEFAULT::

===============================================================================
Pre-Installation Summary
------------------------

Please review the following information before continuing:

Installation folder:
/opt/ibm/SCE31

Product selected for update:
IBM SmartCloud Entry 3.1

Current JRE version:
1.6.0-SR13

Update JRE version:
1.6.0-SR16

PRESS <ENTER> TO CONTINUE:

===============================================================================
Ready To Install
----------------

InstallAnywhere is now ready to update the JRE version of IBM SmartCloud Entry
3.1 at the following installation location:

/opt/ibm/SCE31

PRESS <ENTER> TO INSTALL:

===============================================================================
Installing...
-------------

[==================|==================|==================|==================]
[------------------|------------------|------------------|------------------]

===============================================================================
Installation Complete
---------------------

Congratulations. The IBM SmartCloud Entry JRE update installed successfully at
the following location:

/opt/ibm/SCE31

PRESS <ENTER> TO EXIT THE INSTALLER:

- Check that the updated Java version used by SmartCloud Entry is Java 6 SR16 FP1:
/opt/ibm/SCE31/jre/bin/java -fullversion
java full version "JRE 1.6.0 IBM Linux build pxa6460sr16-20140418_01 (SR16)"

12) Upgrade OPENSTACK components
Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have alreay applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

- Update yum repos with new OPENSTACK packages

(1)On Central Server 1, extract openstack.tar to your ifix path

tar -xf openstack.tar

Two folders are extracted, openstack_norach and openstack_x86.

Copy rpm packages in the above two folders to SCO yum repo.

copy -rf openstack_norach/* /data/repos/scp/ibm-rpms/openstack_noarch/
copy -rf openstack_x86/* /data/repos/scp/ibm-rpms/openstack_x86/

Run below commands to update yum repo:

yum clean all
createrepo /data/repos/scp/

(2)On each SCO Region server, perform same actions like (1) to update its yum repo.

- Backup configuration files

(1)On Central Server 2, backup below files/folders
/root/keystonerc
/etc/keystone
/etc/iaasgateway
/etc/my.cnf
/etc/qpid

(2)On each Region Server, backup below files/folders
/root/.SCE31
/root/openrc
/root/keystonerc
/etc/cinder
/etc/nova
/etc/glance
/etc/qpid
/etc/my.cnf

(3)On each Compute node, backup below files/folders
/etc/nova
/etc/my.cnf
/root/openrc

- Upgrade OPENSTACK rpm packages

On Central Server 2, each region server and each compute node, perform below action:

(1)Copy SP7_rpm_list to a directory of your choice on the server, <your_path>

(2)Run below command to update OPENSTACK components:
cd <your_path>
yum clean all
for i in $(cat SP7_rpm_list); do yum update $i;done

- Restore configuration files

(1)On Central Server 2:
/root/keystonerc
/etc/keystone
/etc/iaasgateway
/etc/my.cnf
/etc/qpid

(2)On each Region Server:
/root/.SCE31
/root/openrc
/root/keystonerc
/etc/cinder
/etc/nova
/etc/glance
/etc/qpid
/etc/my.cnf

(3)On each Compute node:
/etc/nova
/etc/my.cnf
/root/openrc

- Upgrade SCE on each Vmware Region server

(1)Copy 3.1.0.4-IBM-SCE-FP004-201407010432.zip to a directory of your choice on the server, <your_path>

(2)Extart 3.1.0.4-IBM-SCE-FP004-201407010432.zip in the directory where you copied it.

(3)Backup /opt/ibm/SCE31/program/skc.ini

(4)Upgrade SCE:

Login into SmartCloud Entry console running
telnet localhost 7777

You will see a prompt like
osgi>

In this console type the following commands (the commands are in italics, then you can also see an example of the output):
osgi> showrepos

Metadata repositories:

Artifacts repositories:

repositories:

Artifacts repositories:

file:/C:/Users/IBM_ADMIN/.eclipse/207580638/p2/org.eclipse.

equinox.p2.core/cache/

If the repository that is storing the extracted files is not available, use the addrepo command to add that repository.
osgi> addrepo file:<the absolute path when you unpacked the zip file>
SKC update repository added

Install the updates by using the installupdates command.
osgi> installupdates
SKC updates to install:
com.ibm.cfs.product 3.1.0.3-201403100300 ==> com.ibm.cfs.product 3.1.0.4-201407010432
SKC update done

When the update is complete, activate the changes by using the close command to end the OSGi session, then restarting SmartCloud Entry.
osgi> close

(5)Restore the original copy of /opt/ibm/SCE31/program/skc.ini

(6)Restart SmartCloud Entry running
service sce restart

- Apply patches to OpenStack Components

Note: Please use ./SCOrchestrator.py to stop all the services before applying the changes.

(1) use below command to find python directory:

$ python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"

For example:
$ python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
/usr/lib/python2.6/site-packages

From the command, '/usr/lib/python2.6/site-packages' is #{your_python_dir} which will be used in following steps.

(2) apply keystone patch on CS-2
login to CS-2 and carry out below commands:
$ scp root@$cs-1_ip:/data/repos/scp/patch/keystone.patch <your_directory>
$ cd #{your_python_dir}
$ patch -p1 -N -f < <your_directory>/keystone.patch

(3) apply OpenStack patches on each Region Server
login to Region Server and carry out below commands:

-- apply nova patch
$ cd #{your_python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch
$ cd /usr
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch

Note: For this part, please ignore the error message reporting "can't find file to patch" for below files:
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.json.tpl
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.xml.tpl
nova/tests/integrated/test_api_samples.py
nova/tests/test_configdrive2.py
nova/tests/test_nova_manage.py
doc/api_samples/all_extensions/extensions-get-resp.json
doc/api_samples/all_extensions/extensions-get-resp.xml

-- apply glance patch
$ cd #{python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/glance.patch

-- apply cinder patch
$ cd #{python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/cinder.patch

(4) apply nova patch on each KVM compute node
login to KVM compute node and carry out below commands:
$ scp root@$rs_ip:/data/repos/scp/patch/nova.patch <your_directory>
$ cd #{your_python_dir}
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch
$ cd /usr
$ patch -p1 -N -f < /data/repos/scp/patch/nova.patch

Note: For this part, please ignore the error message reporting "can't find file to patch" for below files:
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.json.tpl
nova/tests/integrated/api_samples/all_extensions/extensions-get-resp.xml.tpl
nova/tests/integrated/test_api_samples.py
nova/tests/test_configdrive2.py
nova/tests/test_nova_manage.py
doc/api_samples/all_extensions/extensions-get-resp.json
doc/api_samples/all_extensions/extensions-get-resp.xml

13) Manual steps to install fix for vCenter 5.5 support on each VMware Region Server.
Note: This fix is also included in 2.3.0.1-CSI-ISCO-IF0004. If you have alreay applied 2.3.0.1-CSI-ISCO-IF0004 to your SCO environment, please ignore this part.

(1) Copy package '3.1.0.4-IBM-SCE-IF001-201407170051.zip' to the region server $your_folder.

(2) On VMware Region Server, extract the fix package:
cd $your_folder
unzip 3.1.0.4-IBM-SCE-IF001-201407170051.zip

(3) Ensure SCE service is stopped.
service sce status
If SCE service is running, stop it with below command.
service sce stop

(4) Backup configure file:
cp /opt/ibm/SCE31/program/skc.ini $your_folder/

(5) Install SCE ifix:

Login into SmartCloud Entry console running
telnet localhost 7777

You will see a prompt like
osgi>

In this console type the following commands (the commands are in italics, then you can also see an example of the output):
osgi> showrepos

Metadata repositories:

Artifacts repositories:

repositories:

Artifacts repositories:

file:/C:/Users/IBM_ADMIN/.eclipse/207580638/p2/org.eclipse.

equinox.p2.core/cache/

If the repository that is storing the extracted files is not available, use the addrepo command to add that repository.
osgi> addrepo file:<the absolute path when you unpacked the zip file>
SKC update repository added

Install the updates by using the installupdates command.
osgi> installupdates
SKC updates to install:
com.ibm.cfs.product 3.1.0.4-201407010432 ==> com.ibm.cfs.product 3.1.0.4-201407162230
SKC update done

When the update is complete, activate the changes by using the close command to end the OSGi session, then restarting SmartCloud Entry.
osgi> close

(6)Restore the original copy of /opt/ibm/SCE31/program/skc.ini

(7)Add "-Dhttps.protocols=TLSv1" in /opt/ibm/SCE31/program/skc.ini

(8)Restart SmartCloud Entry running
service sce restart

(9)Start SCE service:
service sce start

14) Manual steps to apply the IWD cumulative fix on Central Server 3.

(1) Copy IFIX005_IWD_SCO2301_20140902-0433-971.tar on Central Server 3.
(2) Extract IFIX005_IWD_SCO2301_20140902-0433-971.tar in a directory of your choice by running the following command:
tar -xf IFIX005_IWD_SCO2301_20140902-0433-971.tar

(3) Change directory to the one where IFIX005_IWD_SCO2301_20140902-0433-971 was extracted.

The following files are extracted:
install_fix.sh
README.txt
version
checksums_20140307-0054-235
checksums_20140515-1212-353
checksums_20140827-0603-225
checksums_20140411-0853-495
checksums_20140602-0115-622
checksums_20140902-0433-971
checksums_20140514-0635-815
checksums_20140711-0842-753
iwd_sco2301_20140902-0433-971.tar
iwd_db2_sco2301_20140902-0433-971.tar
removefiles.txt

(4) Modify DB2 database
Connect to DB2 server on Central Server 1, change user to db2inst1 and backup two databases, rainmake and storhous.
Execute all SQL commands included in SQL files in iwd_db2_sco2301_20140902-0433-971.tar archive.
SQL files:
opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.cloud-4.0.0.1/sql/db2/create-4.0.0.1-ifix.sql
opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.users-4.0.0.1/sql/db2/create-4.0.0.1-ifix.sql

Example:
Copy iwd_db2_sco2301_20140902-0433-971.tar file to Central Server 1. Extract it to a directory of your choice.
su - db2inst1
db2 connect to rainmake
db2 -tvsf <path_to_file>/opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.cloud-4.0.0.1/sql/db2/create-4.0.0.1-ifix.sql
db2 -tvsf <path_to_file>/opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.users-4.0.0.1/sql/db2/create-4.0.0.1-ifix.sql

or line by line:
db2 ALTER TABLE groups ADD COLUMN tmpname VARCHAR(64)
...
(5) Use script to intall the cumulative fix
Run command: ./install_fix.sh (Note: this need to be carry out in current path of install_fix.sh)
The script makes a backup copy of all files under /opt/ibm/rainmaker-backup/20140902-0433-971, verifies all files that are being modified by the IFIX installer and installs the fix.

(6) Manual steps to intall IWD cumulative fix
*WARNING: This way is recommended if your SCO 2.3.0.1 environment is not a standard installation. If you ran the script ./install_fix.sh in step (5) and it completed successfully then the steps in this part may be skipped.

-Stop the workload deployer.
In order to stop the workload deployer, log as root into the Central Server 3 host (workload deployer machine) and execute following command:
service iwd stop

-Make a backup
It is strongly recommended to create a backup copy of all files that will be replaced with the iFix before applying the iFix.
Note: The backuped files should not be stored in IWD folders tree (locations /opt/ibm/* /drouter/*)

Backup the files list in 'checksums_20140902-0433-971' and 'removefiles.txt'.
Backup two databases: rainmake and storhous.

-Install the fix
Verify that IWD component is at required version to apply the iFix. Go to folder where iFix is extracted and run following command for each of the provided checksum files:

md5sum -c checksumfile

If all files are identical (OK status) the fix procedure may be continued.

Ensure you have backuped the files required in Step 3.2.

Replace all IWD files which are listed in 'checksums_20140902-0433-971' with the ones in package 'iwd_sco2301_20140902-0433-971'.
For example,
tar -xvf iwd_sco2301_20140902-0433-971.tar
cp ./opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.openstack.shim-4.0.0.1/lib/rainmaker.openstack.shim.jar /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.openstack.shim-4.0.0.1/lib/rainmaker.openstack.shim.jar
...

Remove all files listed in "removefiles.txt" file.

Validate fix installation
md5sum -c checksums_20140902-0433-971

If all files are identical (OK status) the fix is installed successfully.

In case of any failure, please try the fix again or restore the files from backup.

(7) Enable IWD notifications --- this step is optional, intended for large environments only (thousands of instances). To enable the IWD
notifications, contact the IBM Customer Support for SmartCloud Orchestrator and request the LA fix 2.3.0.1-CSI-ISCO-LA0012 for PMR 02984,999,000. Once done, you can proceed with the following steps (<1>,<2>,<3>,<4>,<5>).

<1> Update [DEFAULT] section in /etc/nova/nova.conf on each region server
Put the three lines under section #### QPID ####

notification_driver=nova.openstack.common.notifier.rabbit_notifier
instance_usage_audit = True
notify_on_state_change = vm_and_task_state
instance_usage_audit_period = hour

<2> Update /opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.openstack.notifications-4.0.0.1/config/zero.config on Central Server 3:

/config/openstack/notifications = {
"general": {
"queue_factory_class":
"org.apache.qpid.jndi.PropertiesFileInitialContextFactory",
"notifications_topic": "nova/notifications.info",
"vm_expiration_sec": 6000,
"vm_load_on_init": true,
"resolving_threads": 5,
"resolving_threads_processing_delay_sec": 5
},
"regions": {
"Region_1_name" : { <------Note: here is the region server name
"brokerlist" : "tcp://BROKER_1_ADDRESS:5672" <------Note: here is the region server ip
},
"Region_2_name" " { <------Note: For each of the region server, need a part like that.
"brokerlist" : "tcp://BROKER_2_ADDRESS:5672"
}
}
}


<3> Ensure CS-3 can use "telnet $region_server_ip 5672" to connect to region server

Here need to ensure each region server have enable the port 5672 access to the cs-3.
There are two ways to enable region server port 5672 access to the cs-3.

method 1:
On region server:
vim /etc/sysconfig/iptables

add below two lines, right before the iprules of port 5672

-A INPUT -s $cs-3_ip/32 -p udp -m udp --dport 5671:5672 -j ACCEPT
-A INPUT -s $cs-3_ip/32 -p tcp -m tcp --dport 5671:5672 -j ACCEPT

Then run 'iptables-save' and 'service iptables restart'
Try command "telnet $region_server_ip 5672" on cs-3. If it can connect to the region server, then you can restart iwd service.

method 2:
On region server:
iptables-save >/tmp/iptables.save

Then modify the file /tmp/iptables.save and add two lines like below:

-A INPUT -j nova-api-INPUT <---right after this line add below two lines.
-A INPUT -s $cs-3_ip/32 -p udp -m udp --dport 5671:5672 -j ACCEPT
-A INPUT -s $cs-3_ip/32 -p tcp -m tcp --dport 5671:5672 -j ACCEPT

Carry out the command to restore iptables:
iptables-restore < /tmp/iptables.save

Then restart iptables.

<4> Restart IWD service
On CS-3, use 'service iwd restart' to restart iwd service and wait about 15min for its initialization.

<5> Try command "telnet $region_server_ip 5672" on cs-3. If telnet is not installed on cs-3, please run 'yum install telnet' to install it.
You should get a message on the shell like this:

Trying $region_server_ip...
Connected to $region_server_ip.
Escape character is '^]'.

15) Apply fix for PSIRT 1838

(1)copy the file rtc-184008-keystone-11c387264-ifix-el6.tar.gz to a directory of your choice <your_dir> on Central Server 2.
(2)extract rtc-184008-keystone-11c387264-ifix-el6.tar.gz by following command:
cd <your_dir>; tar xvf rtc-184008-keystone-11c387264-ifix-el6.tar.gz
(3)backup the keystone configure file /etc/keystone
(4)install keystone rpms with command:
cd noarch; rpm -Uvh openstack-keystone-2013.1.5.1-201408010306.ibm.13.noarch.rpm python-keystone-2013.1.5.1-201408010306.ibm.13.noarch.rpm
(5)check installed keystone version:
rpm -qa |grep keystone

16) Apply fix for PSIRT 1876
(1) copy PSIRT1876.patch to a directory of your choice <your_dir> on Central Server 2.
(2) get python path with command: python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
(3) backup file $python_path/keystone/token/controllers.py
(4) patch keystone with following command:
cd $python_path; patch -p1 -N -f < <your_dir>/PSIRT1876.patch

17) Apply fix for APAR ZZ00267
(1) copy keystone_zz00267.patch to a directory of your choice <your_dir> on Central Server 2.
(2) get python path with command: python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
(3) backup file: mv $python_path/keystone/middleware/ldapauth.py /tmp
(4) patch keystone with following command:
cd $python_path; patch -p1 -N -f < <your_dir>/keystone_zz00267.patch

18) Apply fix for keystone multi-worker support
(1) copy 0001-enable-multiple-keystone-all-worker-processes.patch to a directory of your choice <your_dir> on Central Server 2.
(2) get python path with command: python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"
(3) Backup below files before applying the fix.

/usr/bin/keystone-all
$python_path/keystone/common/config.py
$python_path/keystone/common/wsgi.py

(4) Stop the keystone service using command 'service openstack-keystone stop'.
(5) Apply the fix
cd $python_path
patch -p1 < <your_dir>/0001-enable-multiple-keystone-all-worker-processes.patch

You can get the output like:
... ...
|Change-Id: If74f13bc2898e880649ee809967f5b5859b793c6
|---
| bin/keystone-all | 37 ++-
| etc/keystone.conf.sample | 8 +
| keystone/common/config.py | 2 +
| keystone/common/wsgi.py | 31 ++-
| keystone/openstack/common/loopingcall.py | 148 ++++++++++
| keystone/openstack/common/service.py | 446 ++++++++++++++++++++++++++++++
| keystone/openstack/common/threadgroup.py | 122 ++++++++
| 7 files changed, 785 insertions(+), 9 deletions(-)
| create mode 100644 keystone/openstack/common/loopingcall.py
| create mode 100644 keystone/openstack/common/service.py
| create mode 100644 keystone/openstack/common/threadgroup.py
|
|diff --git a/bin/keystone-all b/bin/keystone-all
|index 2fdc8c7..a28f31c 100755
|--- a/bin/keystone-all
|+++ b/bin/keystone-all
--------------------------
File to patch: /usr/bin/keystone-all <======== Note: here you need to input the file path here.
patching file /usr/bin/keystone-all
can't find file to patch at input line 159
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/etc/keystone.conf.sample b/etc/keystone.conf.sample
|index 9e66eb6..b70edbc 100644
|--- a/etc/keystone.conf.sample
|+++ b/etc/keystone.conf.sample
--------------------------
File to patch: <======== Note: here you need to click 'Enter'
Skip this patch? [y] y <======== Note: here you need to input 'y'
Skipping patch.
1 out of 1 hunk ignored
patching file keystone/common/config.py
patching file keystone/common/wsgi.py
patching file keystone/openstack/common/loopingcall.py
patching file keystone/openstack/common/service.py
patching file keystone/openstack/common/threadgroup.py

(6) Modify the file /etc/keystone/keystone.conf and specify the number of workers you want.
Typically these would be set to the number of CPUs.

For example:
[DEFAULT]
....
public_workers = 2
admin_workers = 2
.....

(7) start keystone service using 'service openstack-keystone start'
And ensure it is up and running with 'service openstack-keystone status'
Also, you can check the number of keystone process with 'ps -ef | grep keystone'

19) Apply fix for Iaasgateway cluster support
(1) copy iaasgateway-2013.1-1.1.4.ibm.201409112157.noarch.rpm to a directory of your choice <your_dir> on Central Server 2.
(2) backup below file before applying the fix.
/etc/iaasgateway/iaasgateway.conf
(3) applying the fix.
<1> prepare http server as loadbalancer
- Check if there is already a http server on Central Server 2
service httpd status
If there is already a http server, stop it with below command:
service httpd stop
If there is no http server installed, use below command to install one:
yum install httpd

- Update httpd.conf with loadbalancer configuration
Modify the file /etc/httpd/conf/httpd.conf with below two changes:
(1) update listen port to gateway port
# Listen 80
Listen 9973
(2) append loadbalancer configuration to the end the file
<VirtualHost *:9973>
ProxyRequests off
<Proxy balancer://mycluster>
# three node gateway cluster
BalancerMember http://127.0.0.1:12001
BalancerMember http://127.0.0.1:12002
BalancerMember http://127.0.0.1:12003

Order Deny,Allow
Deny from none
Allow from all

ProxySet lbmethod=byrequests
</Proxy>

# path of requests to balance "/" -> everything
ProxyPass / balancer://mycluster/
</VirtualHost>

<2> patch the iaasgateway for clustering
cd <your_dir>
rpm -Uvh iaasgateway-2013.1-1.1.3.ibm.201408250136.noarch.rpm
Please ignore the warning message "/etc/iaasgateway/iaasgateway.conf created as /etc/iaasgateway/iaasgateway.conf.rpmnew"

<3> prepare configure file for cluster members
Perform below commands:

cd /etc/iaasgateway/
cp iaasgateway.conf iaasgateway00.conf
vi iaasgateway00.conf

#It should look like below before applying this fix:
[service]
iaasgateway_listen = <central-server-2-ip>
iaasgateway_listen_port = 9973

#Update it to:
iaasgateway_listen = 127.0.0.1
iaasgateway_listen_port = 1200X
iaasgateway_user_entry = <central-server-2-ip>
iaasgateway_user_entry_port = 9973

# copy configure files and update port
cp iaasgateway00.conf iaasgateway01.conf
sed -i 's/1200X/12001/' iaasgateway01.conf
cp iaasgateway00.conf iaasgateway02.conf
sed -i 's/1200X/12002/' iaasgateway02.conf
cp iaasgateway00.conf iaasgateway03.conf
sed -i 's/1200X/12003/' iaasgateway03.conf


<4> prepare init scripts and update configure file

cd /etc/init.d/
cp openstack-iaasgateway openstack-iaasgateway01
cp openstack-iaasgateway openstack-iaasgateway02
cp openstack-iaasgateway openstack-iaasgateway03
sed -i 's/prog=openstack-iaasgateway/prog=openstack-iaasgateway01/' openstack-iaasgateway01
sed -i 's/iaasgateway.conf/iaasgateway01.conf/' openstack-iaasgateway01
sed -i 's/prog=openstack-iaasgateway/prog=openstack-iaasgateway02/' openstack-iaasgateway02
sed -i 's/iaasgateway.conf/iaasgateway02.conf/' openstack-iaasgateway02
sed -i 's/prog=openstack-iaasgateway/prog=openstack-iaasgateway03/' openstack-iaasgateway03
sed -i 's/iaasgateway.conf/iaasgateway03.conf/' openstack-iaasgateway03

<5> startup of cluster

Perform below commands to start iaasgateway cluster service

service openstack-iaasgateway stop
Stopping openstack-iaasgateway: [ OK ]

service openstack-iaasgateway01 start
Starting openstack-iaasgateway01: [ OK ]

service openstack-iaasgateway02 start
Starting openstack-iaasgateway02: [ OK ]

service openstack-iaasgateway03 start
Starting openstack-iaasgateway03: [ OK ]

service httpd start
Starting httpd: [ OK ]

<6> check the iaasgateway service status
- try to open below link in browser and it should be the same as it was before applying this fix.
http://<central-server-2-ip>:9973/providers
- check listening ports with below command:
# netstat -nap | grep 1200 | grep LISTEN
tcp 0 0 127.0.0.1:12001 0.0.0.0:* LISTEN 7269/python
tcp 0 0 127.0.0.1:12002 0.0.0.0:* LISTEN 7286/python
tcp 0 0 127.0.0.1:12003 0.0.0.0:* LISTEN 7303/python

- check loadbalancer listening:
# netstat -nap | grep 9973 | grep LISTEN
tcp 0 0 :::9973 :::* LISTEN 7321/httpd

- try to login SCO UI