Fresh installation of IBM Cloud Orchestrator

Step 1: Review the Installing topic in the IBM Knowledge Center.

IMPORTANT NOTE: For high-availability installations only: After you download the SA_MP_v4.1_Lnx.tar file, you must rename the file to SA_MP_4.1_Linux.tar or the installation procedure will fail.

Upgrade of IBM Cloud Orchestrator from version 2.5

The following upgrade scenarios are supported:

• IBM Cloud Orchestrator V2.5 -> IBM Cloud Orchestrator V2.5 Fix Pack 1 (2.5.0.1)

Step 1: Review the Upgrading from IBM Cloud Orchestrator V2.5 topic in the IBM Knowledge Center.

Upgrade of IBM Cloud Orchestrator from version 2.4.0.2 or later

The following upgrade scenarios are supported:

• IBM Cloud Orchestrator V2.4 Fix Pack 2 (2.4.0.2) -> IBM Cloud Orchestrator V2.5 Fix Pack 1 (2.5.0.1)
• IBM Cloud Orchestrator V2.4 Fix Pack 3 or later (2.4.0.3) -> IBM Cloud Orchestrator V2.5 Fix Pack 1

Step 1: Review the Migrating from IBM Cloud Orchestrator V2.4.0.2 or later topic in the IBM Knowledge Center.

Step 2: If you are migrating from an IBM Cloud Orchestrator 2.4.0.2 (or later) environment, review the Before Migration, During Migration and After Migration sections and apply the steps appropriate for your setup.

Note: All references to version 2.4.0.2 also apply to version 2.4.0.3 or later.

Before migrations in preparation phase

Record allocated availability zones for domains and projects

On the IBM Cloud Orchestrator 2.4.0.2 system, take note of the availability zones that are added to each domain and project.

---

Record Domain Quotas

On the IBM Cloud Orchestrator 2.4.0.2 system, take note of the Domain quotas for the default domain.

---

KVM regions and non-default domains

When migrating a KVM region, some virtual machines in non-default domains may not shut down properly before being migrated.
As a result, they are in an 'active' state with their volumes still attached during the migration and there might be issues when restarting those instances after the migration.

Solution:
Before the migration, update the /opt/ico_install/{release_directory}/installer/stop_24_services.sh file on the IBM Cloud Orchestrator 2.5 server to add the --all-tenants flag after the nova and cinder list commands by running the following commands:

• cd /opt/ico_install/{release directory}/installer
• mv stop_24_services.sh stop_24_services.sh_old && sed "/shut down/,$ s%list%list --all-tenants%g" stop_24_services.sh_old > stop_24_services.sh && chmod +x stop_24_services.sh

After the migration, if you have issues with instances which fail to restart, you must manually stop and then start the instances again.

---

Before running the first region migration

After migrating a VMware region, you can not see or detach migrated volumes.

Solution:
Before running the first region migration, adjust the migration script by running the following commands:

cd /opt/ico_install/{release directory}/installer/upgrade
mv process_nova.py process_nova.py_old && head -n -14 process_nova.py_old > process_nova.py && chmod +x process_nova.py

---

Instruction changes in the Knowledge Center

When following the procedure described in the Preparing for the migration topic, you must complete step 9 before step 7.

---

---

During Migration

Additional steps to configure LDAP

If you are using LDAP user authentication, when migrating the first region, you must perform some additional steps to configure LDAP users in the IBM Cloud Orchestrator 2.5.0.1 environment.

Depending on the type of region that you are migrating, perform the following steps after step 3 of the procedure described in the Migrating a VMware region or Migrating a KVM region topic in the IBM Knowledge Center.

Solution:

1. Copy the ldap_configure.sh script from the IBM Cloud Orchestrator 2.5.0.1 server to the destination node of the first region that you are migrating:
   
   scp /opt/ico_install/V2501/installer/ldap_configure.sh root@{destination_first_region}:/tmp



2. Log in to the destination node as user root, go to the directory where you copied the script, and run the ldap_configure.sh script as follows:
   
   ./ldap_configure.sh

Additional Knowledge Center updates related to this issue:

• Update to the Migrating a KVM region topic:
  
  
  
  The commands in step 1b must be replaced with the following:
  
  source ~/openrc
  glance image-list --all-tenants
  heat stack-list
  cinder list --all-tenants
  neutron net-list
  neutron subnet-list --all-tenants
  nova list --all-tenants
  nova image-list
  nova flavor-list
  
  The commands in step 4 must be replaced with the following:
  
  source ~/v3rc
  openstack endpoint list
  openstack user list
  openstack project list
  openstack domain list
  openstack image list
  heat stack-list
  cinder list --all-tenants
  openstack network list
  neutron subnet-list --all-tenants
  nova list --all-tenants
  openstack server list
  openstack flavor list
  
  The note in step 3.e of the procedure must be replaced with the following:
  
  "Note: If any of the compute nodes are not visible in the OpenStack Dashboard after restarting the services for the region, or if the instances do not start successfully, you must manually restart the openstack-nova-compute service by running the following command as user root on the relevant nodes: systemctl restart openstack-nova-compute"
  
  Step 6c must be replaced with the following:
  
  "Debug any errors that are displayed on the console output and check the upgrade, export and import logs in the /var/log/ico_install directory on the IBM Cloud Orchestrator V2.5.0.1 Server for any further issues which may have occurred during the data migration."


• Update to the Migrating a VMware region topic:
  
  
  
  The commands in step 1b must be replaced with the following:
  
  source ~/openrc
  glance image-list --all-tenants
  heat stack-list
  cinder list --all-tenants
  neutron net-list
  neutron subnet-list --all-tenants
  nova list --all-tenants
  nova image-list
  nova flavor-list
  
  The commands in step 4 must be replaced with the following:
  
  source ~/v3rc
  openstack endpoint list
  openstack user list
  openstack project list
  openstack domain list
  openstack image list
  heat stack-list
  cinder list --all-tenants
  openstack network list
  neutron subnet-list --all-tenants
  nova list --all-tenants
  openstack server list
  openstack flavor list
  
  Step 6c must be replaced with the following:
  
  "Debug any errors that are displayed on the console output and check the upgrade, export and import logs in the /var/log/ico_install directory on the IBM Cloud Orchestrator V2.5.0.1 Server for any further issues which may have occurred during the data migration."

---

Running the upgrade-prereq-checker script on regions with nova networks

During the migration, if the upgrade-prereq-checker.sh script is run with the check-regions flag on a system which has a Nova network and no Neutron network is available, the script fails with a stack trace error message. The traceback error occurs after the "checking if the SOURCE_CENTRAL_DB_PASSWORD is correct" message, as per the following example.

Note: If this error occurs then it indicates that you cannot migrate that region because the migration of regions that use Nova networks is not supported.

Example command:



# ./upgrade-prereq-checker.py ico_upgrade.rsp --check-regions
INFO: Checking that prerequisite packages are installed on local machine
...
INFO:  - Checking that regions have not already been migrated
INFO:    - Status: Success
INFO:  - Checking that parameter SOURCE_CENTRAL_DB_PASSWORD is correct
INFO:    - Status: Success
Traceback (most recent call last):
  File "./upgrade-prereq-checker.py", line 93, in <module>
    if not migrate_validator.validate(install_packages, check_regions, param_dict, install_base=installdir):
  File "/opt/ico_install/V2501/ico-install/migrate_validator/__init__.py", line 165, in validate
    ks.make_service_call('neutron', '/v2.0/networks.json')
  File "/opt/ico_install/V2501/ico-install/migrate_validator/os_client.py", line 256, in make_service_call
    endpoint = self.get_service_endpoint(service_name, endpoint_type=endpoint_type)
  File "/opt/ico_install/V2501/ico-install/migrate_validator/os_client.py", line 183, in get_service_endpoint
    ep_stanza = filter_one(fn,manifest['endpoints'])
TypeError: 'NoneType' object has no attribute '__getitem__'
#

---

---

After Migration

Availability zones

When migrating from an IBM Cloud Orchestrator 2.4.0.2 environment, the allocated availability zones for the domains and projects are not migrated to the IBM Cloud Orchestrator 2.5.0.1 system.

Solution:

To address this issue, availability zones for the domains and projects should be allocated to match the availability zones that were recorded in the step above Record allocated availability zones for domains and projects.

---

Heat stack deployment

After migrating a region from an IBM Cloud Orchestrator 2.4.0.2 environment, Heat stack deployments might time out or fail.

Solution:

1. Restart the Heat services by running the following commands on the IBM Cloud Manager controller:
   
   systemctl restart openstack-heat-engine
systemctl restart openstack-heat-api



2. Delete the failed deployment and deploy the Heat stack again.

---

KVM cinder volumes

After migrating from IBM Cloud Orchestrator 2.4.0.2, the following issues related to KVM cinder volumes might occur:

• After the migration, KVM cinder volumes might not automatically be reattached on the IBM Cloud Orchestrator 2.5.0.1 region.
• If you roll back the migration of a KVM region, KVM cinder volumes might not automatically be reattached on the IBM Cloud Orchestrator 2.4.0.2 region.

Solution:
To address this issue, after the migration or the rollback, you must manually reattach any volumes as required. The file /opt/ico_install/V2501/installer/cinder-attachments on the IBM Cloud Orchestrator 2.5.0.1 Server contains a list of the IBM Cloud Orchestrator 2.4.0.2 associations of cinder volume uuids to the corresponding instance uuids. You can use this file as a reference to manually re-attach any volumes as required.

---

Check logs

When following the procedure described in the Migrating IBM Cloud Orchestrator data topic, perform the following step after step 1:

Debug any errors that are displayed on the console output and check the upgrade, export, and import log files in the /var/log/ico_install directory on the IBM Cloud Orchestrator V2.5.0.1 Server for any further issues which may have occurred during the data migration.

Troubleshoot any issues which occurred during the IBM Cloud Orchestrator migration.

If you see any critical errors in the output during the migration which indicate that the migration was not successful, run the following steps:

• To reduce system downtime during the debug process, you may roll the system back to use the IBM Cloud Orchestrator V2.4.0.2 environment by restarting the Public Cloud Gateway by running the following command on the IBM Cloud Orchestrator V2.4.0.2 Central Server 2:
  
  systemctl start pcg
  
  For more information about restarting IBM Cloud Orchestrator V2.4.0.2, review Managing the services topic.
  
  For more information about restarting IBM Cloud Orchestrator V2.4.0.3, review Managing the services topic.

---

Domain Quotas

When migrating from an IBM Cloud Orchestrator 2.4.0.2 environment, domain quotas for the default domain are not migrated to the IBM Cloud Orchestrator 2.5.0.1 system.

Solution:
To address this issue, domain quotas should be adjusted to match the ones that were recorded in the step above Record Domain Quotas.

---

Request history is not migrated automatically

Request history is not migrated automatically because of IBM Cloud Orchestrator 2.4.0.2 running in parallel with IBM Cloud Orchestrator 2.5.

The IBM Cloud Orchestrator 2.4.0.2 database can be used as an archive for the IBM Cloud Orchestrator 2.4.0.2 request history as it is still stored there. If you want to export the details of the request history to a file for purposes such as request auditing, run the following the procedure.

Note: This procedure generates the /home/db2inst1/request_history.csv file which can be opened in a spreadsheet application in comma separated fashion.

1. Log in to the IBM Cloud Orchestrator 2.4 central database server (by default, Central Server 1).
2. Run the following commands:
• su - db2inst1
• db2 connect to BPMDB
• db2 "export to request_history.del of del select UUID, (date('1970-01-01')+(CREATED/86400000) days ) as CREATED, (date('1970-01-01')+(UPDATED/86400000) days ) as UPDATED, PROJECT, \"USER\", TRIM(CAST(REPLACE(SUBSTR(PARMSTRING, (LOCATE('extended_task_name',PARMSTRING)+21) ,(LOCATE('}',PARMSTRING))), '\"}', '')as varchar(2000)) ) as PROCESS, MESSAGEKEY, REMOTECORRELATIONKEY from ORCHESTRATOR.TASK, ORCHESTRATOR.TASKEXCEPTION where ISACTION=0 and REMOTECORRELATIONKEY>0 and DESCRIPTION_ID=ID and PARMSTRING LIKE '%extended_task_name%' UNION ALL select UUID, (date('1970-01-01')+(CREATED/86400000) days ) as CREATED, (date('1970-01-01')+(UPDATED/86400000) days ) as UPDATED, PROJECT, \"USER\", TRIM(CAST(REPLACE(SUBSTR(PARMSTRING, (LOCATE('process\":',PARMSTRING)+10) ,(LOCATE('\",\"',PARMSTRING)-10)), '\",\"', '')as varchar(2000)) ) as PROCESS, MESSAGEKEY, REMOTECORRELATIONKEY from ORCHESTRATOR.TASK, ORCHESTRATOR.TASKEXCEPTION where ISACTION=0 and REMOTECORRELATIONKEY>0 and DESCRIPTION_ID=ID and PARMSTRING NOT LIKE '%extended_task_name%'"
• echo "REQUEST_ID, DATE_CREATED, DATE_UPDATED, PROJECT, USER, PROCESS, MESSAGE, REMOTE_PROCCESS_ID " > request_history.csv
• cat request_history.del >> request_history.csv

---

Editing domains or projects

After migrating from an IBM Cloud Orchestrator 2.4.0.2 environment, when editing a domain or a project, the following error might be displayed in the OpenStack Dashboard:

Error: Unable to retrieve availability zone data.

In the httpd log file, the following error is displayed:

==> openstack-dashboard-error.log <== {Tue Dec 08 18:37:29.258353 2015} {:error} {pid 9343} Recoverable error: ('Connection aborted.', error(113, 'No route to host'))

Solution:
Restart the pcg and httpd services.

• On the IBM Cloud Manager Controller: systemctl restart httpd
• On the IBM Cloud Orchestrator server: systemctl restart pcg