Upgrading and Migrating ClearQuest 7.1.x Full-Text Search to ClearQuest 8.0 when the Full-Text Search Administrator tool was not used for the original Full-Text Search deployment

Answer

Before you start the upgrade and migration process, you should take note of the following:

1. It is strongly recommended that you first try this in a test environment that is identical or similar to your production environment to make sure it works in your production environment.
2. There is some down time where Full-Text Search services will be unavailable to users, which is highlighted below. Make sure you take this into consideration (see Step#4).
3. Verify that you have enough free disk space. The process for determining how much space is needed is explained below.
4. This procedure must be started prior to the ClearQuest 8.0 upgrade process and can be completed after installing ClearQuest 8.0.
5. Your ClearQuest database is not affected by this upgrade and migration.

Note: In lieu of this technote, you may also opt to re-deploy ClearQuest Full-Text Search from scratch. This entails using the ClearQuest Full-Text Search Administrator tool to re-deploy ClearQuest Full-Text Search and re-index your database, as well as performing any necessary tuning, after having successfully upgraded to ClearQuest 8.0.

Assumptions made in this guide (use of common terms):

In those steps, the following assumptions are made for which you need to provide your own values:

1. The ClearQuest database set name (or connection name) is TextSearch,
2. The ClearQuest user database name is SAMPL,
3. The ClearQuest admin user-name is admin
4. The ClearQuest admin password is pw
5. The existing ClearQuest 7.1.x Full-Text Search home directory path is C:\CQ.Solr\SAMPL\ on Windows, or /opt/CQ.Solr/SAMPL/ on Unix.
6. The new ClearQuest 8.0 Full-Text Search home will be at C:\CQ.Search\TextSearch_SAMPL\ on Windows, or /opt/CQ.Search/TextSearch_SAMPL/ on Unix.
7. The ClearQuest 7.1.x Full-Text Search WAS profile name is cqsearchprofile and its port number is 14080.
8. The ClearQuest 8.0 Full-Text Search WAS profile name will be cqsearchprofile and its port number is 14080.
9. The by-step instructions will be performed on the machine where you currently have ClearQuest 7.1.x Full-Text Search deployed, prior to upgrading to ClearQuest 8.0

Your values will most likely be different based on your existing ClearQuest 7.1.x Full-Text Search deployment. You will need to provide your own values as you follow these step-by-step instructions. Also, if you have deployed ClearQuest 7.1.x Full-Text Search for additional ClearQuest databases, you will need to use that deployment's values, and repeat these instructions for each deployment, one at a time.

To upgrade and migrate your ClearQuest 7.1.x Full-Text Search deployment to ClearQuest 8.0 Full-Text Search, take the following steps:

---

Step #1: Calculating how much free disk space you need

The amount of free disk space you need is based on two factors, a) the required disk space to install ClearQuest 8.0, and b) the index size of your existing Full-Text Search deployment.

To figure out how much free disk space you need to install ClearQuest 8.0, please consult your ClearQuest 8.0 release notes and documentation. To figure out how much free disk space you need to deploy ClearQuest 8.0 Full-Text Search, take the following steps:

1. On Windows, using Windows explorer, navigate to your ClearQuest 7.1.x Full-Text Search home directory, which is usually at the path "C:\CQ.Solr\". In this directory, you will see a directory called "SAMPL". Right click on the "SAMPL" directory and from the drop down menu, select "Properties". The number next to "Size on disk" is how much additional free disk space you will need to deploy ClearQuest 8.0 Full-Text Search.
   
2. On Unix, from a command shell, navigate to your ClearQuest 7.1.x Full-Text Search home directory, which is usually at "/opt/CQ.Solr/". In this directory, you will see a directory called "SAMPL". From this location, issue the command "du -sh SAMPL". The number reported back is how much additional free disk space you will need to deploy ClearQuest 8.0 Full-Text Search.

Note: Once your ClearQuest 8.0 Full-Text Search deployment is completed, you can delete the old ClearQuest 7.1.x Full-Text Search deployment, and thus reclaim the disk space it was using. You may NOT delete it prior to the upgrade and migration otherwise you will lose your existing index, which would require you to fully re-index your ClearQuest database as well as reconfigure your ClearQuest Full-Text Search deployment.

Step #2: Identify your ClearQuest 7.1.x Full-Text Search deployment settings

To identify your existing ClearQuest 7.1.x Full-Text Search deployment that you will be upgrading and migrating to ClearQuest 8.0, take the following steps:

1. On the machine where you have ClearQuest 7.1.x Full-Text Search deployed, point your browser to http://localhost:14080/solr/admin/index.jsp. If the URL works, and you see Apache's Solr Admin page, than the port number for this deployment is 14080. Write it down as you will need it later on. If the URL doesn't work, you will need to try another port, which is usually 15080, 16080, etc. otherwise you will need find out what port you used when you originally deployed ClearQuest 7.1.x Full-Text Search.
   
2. On the Apache Solr administration page, look for the text "cwd=<path>" where "<path>" is usually "C:\Program Files\IBM\RationalSDLC\common\CM\profiles\cqsearchprofile" on Windows, or "/opt/ibm/RationalSDLC/common/CM/profiles/cqsearchprofile" on Unix. The last part of this path, "cqsearchprofile" is your ClearQuest 7.1.x Full-Text Search profile name. Write it down as you will need it later on.
   
3. Using your OS's file explorer, navigate to "C:\Program Files\IBM\RationalSDLC\common\CM\profiles\<cq-search-profile-name>\config\cells\<cell>\nodes\<node>\servers\server1\" on Windows, or "/opt/ibm/RationalSDLC/common/CM/profiles/<cq-search-profile-name>/config/cells/<cell>/nodes/<node>/servers/server1/" on Unix. In this directory, locate a file called "server.xml", and open it using a text editor such as notepad, or vi. In the text editor, look for the text "maximumHeapSize". If you find it, write down the value assigned to it; if you don't all that means is you are using the default, which means the value is not set, thus you don't need to record anything down.

As noted above, if on this machine, you have multiple ClearQuest 7.1.x Full-Text Search deployments for additional ClearQuest user databases, you will need to repeat #1 to #3 and gather the deployment information, which are: profile-name, port-number and maximumHeapSize setting (if any).

profile-name ex) cqsearchprofile
port-number ex) 14080
maximumHeapSize ex) 1024

Step #3: Create a backup of ClearQuest 7.1.x Full-Text Search restoration files

Before progressing with ClearQuest 8.0 upgrade, consider backing up the following two files, which you will need if you have to restore your ClearQuest 7.1.x Full-Text Search deployment:

1. Using your OS's file explorer, navigate to "C:\Program Files\IBM\RationalSDLC\common\CM\profiles\<cq-search-profile-name>\config\cells\<cell>\nodes\<node>\servers\server1\" on Windows, or "/opt/ibm/RationalSDLC/common/CM/profiles/<cq-search-profile-name>/config/cells/<cell>/nodes/<node>/servers/server1/" on Unix.
   
2. Copy the file "server.xml" to a safe backup location for this profile.
   
3. Using your OS's file explorer, navigate to "C:\Program Files\IBM\RationalSDLC\common\CM\profiles\cqsearchprofile\config\cells\dfltCell\applications\cqtsdbcrawler.ear\deployments\cqtsdbcrawler\cqtsdbcrawler.war\WEB-INF\" on Windows, or "/opt/ibm/RationalSDLC/common/CM/profiles/<cq-search-profile-name>/config/cells/<cell>/applications/cqtsdbcrawler.ear/deployments/cqtsdbcrawler/cqtsdbcrawler.war/WEB-INF/" on Unix.
   
4. Copy the file "web.xml" to a safe backup location for this profile.

These two files are all that you need to restore if you have to revert back to ClearQuest 7.1.x Full-Text Search.

Step #4: Install ClearQuest 8.0

Follow ClearQuest 8.0 guide to upgrade from ClearQuest 7.1.x to ClearQuest 8.0. Once you complete the upgrade, continue with this guide.

NOTE: At this point, your ClearQuest Full-Text Search service will be down, and will remain down until you complete these instructions. It is recommended that you plan this down time and set end user expectations accordingly. This down period will only affect ClearQuest Full-Text Search services; all other ClearQuest functionality will still be available.

Step #5: Create a WebSphere WAS profile for ClearQuest 8.0 Full-Text Search re-deployment

Once the upgrade to ClearQuest 8.0 is completed, for each of the ClearQuest 7.1.x Full-Text Search deployments you identified under Step #2, you will need to repeat Step #5 (this step) through Step #14.

1. From a Windows command prompt, or a Unix shell, navigate to:
   "C:\Program Files\IBM\RationalSDLC\ClearQuest\" on Windows, or
   "/opt/ibm/RationalSDLC/clearquest/" on Unix, or
   the <cqhome> directory of your ClearQuest 8.0 install path.
   
2. At the command prompt, issue this command:
   On Windows:
   cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome C:\CQ.Search --init_cq_fts
   On Unix:
   cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome /opt/CQ.Search --init_cq_fts

This step will create a new directory below "C:\CQ.Search\" on Windows or "/opt/CQ.Search/" on Unix. In this directory, there will be a new sub-directory called "TextSearch_SAMPL". Your results will be determined by your ClearQuest database set name and user database name in the format of "<dbset>_<userdb>".

Step #6: Customize the ClearQuest 8.0 Full-Text Search configuration file

1. Edit the file:
   "C:\CQ.Search\TextSearch_SAMPL\cqtsadmin-TextSearch-SAMPL.xml" on Windows, or
   "/opt/CQ.Search/TextSearch_SAMPL/cqtsadmin-TextSearch-SAMPL.xml" on Unix.
   
2. In this file, locate the XML block <ftsWASProfileName> and in this XML block change the XML node from: <newValue required="no"></newValue> to: <newValue required="no">cqsearchprofile</newValue>. If your ClearQuest Full-Text Search profile name is different than "cqsearchprofile" use it instead. This is the value of your ClearQuest 7.1.x Full-Text Search profile name that you determined in #1.
   
3. Also in this file, locate the XML block <ftsWASProfilePort> and in this XML block change the XML node: <newValue required="no"></newValue> to: <newValue required="no">14080</newValue>. If your ClearQuest Full-Text Search port number is different than "14080" use it instead. This is the value of your ClearQuest 7.1.x Full-Text Search port number that you determined in #1.
   
4. Also in this file, locate the XML block <ftsWASProfileMaxHeapSize> and in this XML block change the XML node: <newValue required="no"></newValue> to: <newValue required="no">####</newValue> , where "####" is the value for "maximumHeapSize" (if you have it.) This is the value of your ClearQuest 7.1.x Full-Text Search JVM maximum heap size that you determined in Step#2.

Step #7: Copy over files from the old ClearQuest 7.1.x Full-Text Search home the new ClearQuest 8.0 Full-Text Search home directory

1. Using your OS's file explorer, navigate to your old ClearQuest 7.1.x Full-Text Search home directory, which is usually at "C:\CQ.Solr\SAMPL\Solr\" on Windows, or "/opt/CQ.Solr/SAMPL/Solr/" on Unix and locate the following files:
   - batch_recovery.properties,
   - update_recovery.properties,
   - CQ-<*>.xml, Entity-<*>.txt
   - key-<*>.txt.
   
   Here, "<*>" will be unique to your original ClearQuest 7.1.x Full-Text Search setup. The name of those files will be based on what you named them when you originally deployed ClearQuest 7.1.x Full-Text Search, but is usually "<dbset>-<userdb>", such as "CQ-TextSearch-SAMPL.xml".
   
2. Copy those 4 files to your ClearQuest 8.0 Full-Text Search home directory, which is now at "C:\CQ.Search\TextSearch_SAMPL\".
   
3. Examine the name of those 4 files that you copied over,
   - batch_recovery.properties,
   - update_recovery.properties,
   - CQ-<*>.xml, Entity-<*>.txt
   - key-<*>.txt.
   
   and rename them as follows:
   > CQ-<*>.xml to CQ-<dbset>-<userdb>.xml
   > Entity-<*>.txt to Entity-<dbset>-<userdb>.txt
   > Solr-<*>.txt to Solr-<dbset>-<userdb>.txt
   > key-<*>.txt to key.txt

Notice how, key-<*>.txt, unlike CQ-<*>.xml, and Entity-<*>.xml, was simply renamed to key.txt without your "<dbset>-<userdb>".

Step #8: Copy over Solr's flat-file index from ClearQuest 7.1.x Full-Text Search home to ClearQuest 8.0 Full-Text Search home

1. Using your OS's file explorer, navigate to your ClearQuest 8.0 Full-Text Search home directory, at:
   "C:\CQ.Search\TextSearch_SAMPL\Solr\solr\" on Windows, or "/opt/CQ.Search/TextSearch_SAMPL/Solr/solr/" on Unix.
   
2. At this location, create a directory called "data" and within it, create a directory called "index", such that you will end up with the path: "C:\CQ.Search\TextSearch_SAMPL\Solr\solr\data\index\" on Windows, or "/opt/CQ.Search/TextSearch_SAMPL/Solr/solr/data/index/" on Unix.
   
3. Using your OS's file explorer, navigate to your old ClearQuest 7.1 Full-Text Search home directory, at old path: "C:\CQ.Solr\SAMPL\Solr\solr\data\index\" on Windows, or "/opt/CQ.Solr/SAMPL/Solr/solr/data/index/" on Unix and copy all files in the "index" directory over to ClearQuest 8.0 Full-Text Search home directory's "index" at new path: "C:\CQ.Search\TextSearch_SAMPL\Solr\solr\data\index\" on Windows, or "/opt/CQ.Search/TextSearch_SAMPL/Solr/solr/data/index/" on Unix.

Step #9: Copy over Solr's schema.xml from ClearQuest 7.1.x Full-Text Search home to ClearQuest 8.0 Full-Text Search home

1. Using your OS's file explorer, navigate to your ClearQuest 7.1.x Full-Text Search home directory, at:
   "C:\CQ.Solr\SAMPL\Solr\solr\conf\" on Windows, or "/opt/CQ.Solr/SAMPL/Solr/solr/conf/" on Unix and locate the file "schema.xml".
   
2. Copy the file "schema.xml" to your ClearQuest 8.0 Full-Text Search home directory at:
   "C:\CQ.Search\TextSearch_SAMPL\Solr\solr\conf\" on Windows, or "/opt/CQ.Search/TextSearch_SAMPL/Solr/solr/conf/" on Unix.

Step #10: Customize CQ-<dbset>-<userdb>.xml file for ClearQuest 8.0 Full-Text Search

1. Using your OS's file explorer, navigate to your ClearQuest 8.0 Full-Text Search home directory at:
   "C:\CQ.Search\TextSearch_SAMPL\" on Windows, or "/opt/CQ.Search/TextSearch_SAMPL/" on Unix, and locate the file "CQ-TextSearch-SAMPL.xml".
   
2. Edit the file "CQ-TextSearch-SAMPL.xml" and look for the XML block <CQDatabase>. If your ClearQuest is replicated (i.e.: MultiSite'ed), there will be multiple such blocks, you will need to locate the one for your master site. To find it, match the XML element "sitename" in this XML block; it will contain the site name of this block, such as: <CQDatabase siteame="MyMasterSiteName">.
   
3. In the XML block <CQDatabase>, locate the sub-XML block called <repository> within this block locate the XML element "keyfile".
   
4. Change its value i.e.: keyfile="value" to "C:/CQ.Search/TextSearch_SAMPL/key.txt" on Windows, or
   "/opt/CQ.Search/TextSearch_SAMPL/key.txt" on Unix. I.e.:<repository dbset="TextSearch" dbname="SAMPL" username="admin" password="t4mlHHX6iF/NqZ32RmXyEA==" keyfile="C:/CQ.Search/TextSearch_SAMPL/key.txt"></repository>

Step #11: Customize cqtsadmin-<dbset>-<userdb>.xml file for ClearQuest 8.0 Full-Text Search

1. Using your OS's file explorer, navigate to your ClearQuest 8.0 Full-Text Search home directory at: "C:\CQ.Search\TextSearch_SAMPL\" on Windows, or "/opt/CQ.Search/TextSearch_SAMPL/" on Unix, and locate the file "cqtsadmin-TextSearch-SAMPL.xml".
   
2. Edit the file "cqtsadmin-TextSearch-SAMPL.xml" Locate the XML block <ftsWASProfileName>. Find the sub-XML block called <newValue required="no"></newValue>
   
3. Change it to <newValue required="no">#profile-name#</newValue>, where "#profile-name#" is the profile-name you determined in Step #2 from you ClearQuest 7.1.x Full-Text Search deployment.
   
4. While still in the editor, locate the XML block <ftsWASProfilePort>. Find the sub-XML block called <newValue required="no"></newValue>
   
5. Change it to <newValue required="no">#port-number#</newValue>, where "#port-number#" is the port-number you determined in Step#2 from you ClearQuest 7.1.x Full-Text Search deployment.
   
6. If in Step #2, you also have "maximumHeapSize", then locate the XML block <ftsWASProfileMaxHeapSize>. Find the sub-XML block <newValue required="no"></newValue>
   
7. Change it to <newValue required="no">#max-heap-size#</newValue> where "#max-heap-size#" is the value for "maximumHeapSize" you determined in Step#2 from you ClearQuest 7.1.x Full-Text Search deployment.

Step #12: Create a new WAS profile for ClearQuest 8.0 Full-Text Search deployment

1. From a Windows command prompt, or a Unix shell, navigate to: "C:\Program Files\IBM\RationalSDLC\ClearQuest\" on Windows, or "/opt/ibm/RationalSDLC/clearquest/" on Unix, or to wherever you installed ClearQuest 8.0.
   
2. At the command prompt, run the command:
   On Windows: cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome C:\CQ.Search --create_fts_was_profile automatic
   On Unix: cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome /opt/CQ.Search --create_fts_was_profile automatic

Step #13: Enabling ClearQuest 8.0 Full-Text Search

1. From a Windows command prompt, or a Unix shell, navigate to "C:\Program Files\IBM\RationalSDLC\ClearQuest\" on Windows, or "/opt/ibm/RationalSDLC/clearquest/" on Unix, or to wherever you installed ClearQuest 8.0.
   
2. At the command prompt, run the command:
   On Windows: cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome C:\CQ.Search --set_solr_home --start_fts_was_profile --set_was_max_mem --start_update_idx
   On Unix: cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome /opt/CQ.Search --set_solr_home --start_fts_was_profile --set_was_max_mem --start_update_idx

Step #14: Verify ClearQuest 8.0 Full-Text Search is working and correctly configured

In this verification step, it is assumed that you will use your own values based on your settings. For example, the port we use is 14080, this may be different for you; the same is true for the paths and profile-name. Each Full-Text Search enabled ClearQuest user database you have deployed will have its own values, as recorded in Step #2.

1. On the machine where you have ClearQuest 8.0 Full-Text Search deployed, point your browser to http://localhost:14080/solr/admin/index.jsp. If the URL works, look for the text "cwd=<path>" where "<path>" will be:
   "C:\Program Files\IBM\RationalSDLC\ClearQuest\cqweb\cqsearchprofile" on Windows, or "/opt/ibm/RationalSDLC/clearquest/cqweb/cqsearchprofile" on Unix. If the URL doesn't work or you don't see the correct path, you will need to verify your steps.
   
2. Using your OS's file explorer, navigate to:
   "C:\Program Files\IBM\RationalSDLC\ClearQuest\cqweb\cqsearchprofile\logs\server1\" on Windows, or "/opt/ibm/RationalSDLC/clearquest/cqweb/cqsearchprofile/logs/server1/" on Unix, and locate the file "SystemOut.log".
   
3. Open this file in a text editor and navigate to the end of it. Do a simple text search upwards in this file for "Login database". If after this line you see an error being reported, you will need to review your steps.
   
4. From a client machine, other than the server machine where ClearQuest 8.0 Full-Text Search is running, logon to ClearQuest Web and verify that you are able to user ClearQuest Full-Text Search. If not, you will need to review your steps.

If the above three validations do not report any issues, you are done. You have successfully upgraded and migrated your ClearQuest 7.1.x Full-Text Search to ClearQuest 8.0 Full-Text Search.

After few days or weeks, based on your organization's policy, you can now delete the old ClearQuest 7.1.x Full-Text Search deployment to save space as it is no-longer needed.

---

Collecting Diagnostic Data to aid troubleshooting

Note: If any of the above three validations fail, take the following steps to gather diagnostic data which will assist IBM Rational Client Support to help you with the issue.

1. From a Windows command prompt, or a Unix shell, navigate to:
   "C:\Program Files\IBM\RationalSDLC\ClearQuest\" on Windows, or
   "/opt/ibm/RationalSDLC/clearquest/" on Unix, or
   to wherever you installed ClearQuest 8.0.
   
2. At the command prompt, run the command (with your values):
   On Windows: cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome C:\CQ.Search --gather_diagnostic_data
   On Unix: cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome /opt/CQ.Search --gather_diagnostic_data
   
3. Collect the diagnostic data directory mentioned at the completion of this command, and send it in a ZIP file to IBM Rational Client Support for further assistance.

---

Reverting back to ClearQuest 7.1.x Full-Text Search

While you should not need to take the following steps, if you need to roll-back to ClearQuest 7.1.x Full-Text Search, take these steps:

1. From a Windows command prompt, or a Unix shell, navigate to:
   "C:\Program Files\IBM\RationalSDLC\ClearQuest\" on Windows, or
   "/opt/ibm/RationalSDLC/clearquest/" on Unix, or
   to wherever you installed ClearQuest 8.0.
   
2. At the command prompt, run the command:
   On Windows: cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome C:\CQ.Search --delete_fts_was_profile
   On Unix: cqperl cqtsadmin.pl --username admin --password pw --dbset TextSearch --userdb SAMPL --ftshome /opt/CQ.Search --delete_fts_was_profile
   
3. See your ClearQuest 8.0 User's Guide to un-install ClearQuest 8.0 Full-Text Search and re-install ClearQuest 7.1.x Full-Text Search.
   
4. Using your OS's file explorer, navigate to:
   "C:\Program Files\IBM\RationalSDLC\common\CM\profiles\<cq-search-profile-name>\config\cells\<cell>\nodes\<node>\servers\server1\" on Windows, or
   "/opt/ibm/RationalSDLC/common/CM/profiles/<cq-search-profile-name>/config/cells/<cell>/nodes/<node>/servers/server1/"
   
5. Restore the file "server.xml" that you backed up in Step #3.
   
6. Using your OS's file explorer, navigate to:
   "C:\Program Files\IBM\RationalSDLC\common\CM\profiles\cqsearchprofile\config\cells\dfltCell\applications\cqtsdbcrawler.ear\deployments\cqtsdbcrawler\cqtsdbcrawler.war\WEB-INF\" on Windows, or
   "/opt/ibm/RationalSDLC/common/CM/profiles/<cq-search-profile-name>/config/cells/<cell>/applications/cqtsdbcrawler.ear/deployments/cqtsdbcrawler/cqtsdbcrawler.war/WEB-INF/" on Unix.
   
7. Restore the file "web.xml" that you backed up in Step #3.
   
8. Using ClearQuest 7.1.x Full-Text Search User's Guide, start the WAS profile for this deployment.