IBM Support

MustGather: PowerVC Data Collection

Question & Answer


Question

What data is needed to debug PowerVC problems?

Answer

IBM PowerVC – Virtualization Center is designed to improve and simplify the management of virtual machines (VMs) and LPARs on Power Systems servers. PowerVC is built on OpenStack services, which allows the necessary interaction to critical underlying components--most notably, HMC and PowerVM (VIOS).

For any problem, a clear problem description and environment details are required:
A clear problem description allows support to more quickly envision your environment, the issue you have, and how it affects your business. For every PowerVC issue we like to see the following data:

A clear description of your PowerVC software and hardware environment.
a) Includes the current fix and service pack level of:
PowerVC
HMC/Novalink
VIOS


b) Error messages and screen capture of the errors your see, including TIMESTAMPS for errors seen or actions you take during the recreation of a problem.

c) Time difference between the PowerVC Linux server, HMC/Novalink, and VIOS servers. Use the "date" command, and let us know whether the systems are synchronized by NTP.
 

For a PowerVC installation problem or error, collect:
1. Gather the contents of the /opt/ibm/powervc/log/ directory.
2. Run rpm -qa command and send output.
3.For PowerVC 2.0.2 or later, run powervc-opsmgr apply-ifix -c clustername -l command and send output.
4. Run ls -al /etc command so we can see which (if any) OpenStack directories were created.

You can clear your logs:

If powervc-diag creates a data collection that is file that is too large, you can clear your logs.
Stop PowerVC services: powervc-services stop
Clear the content of these directories (ONLY):
/var/log/nova
/var/log/cinder
/var/log/keystone

Start PowerVC services: powervc-services start
Then, re-create the failure and make sure to note the timestamp of the error before you collect a powervc-diag again.

Collect the following PowerVC data for all postinstallation problems:

Method 1  - Run powervc-diag manually

Use the powervc-diag command.

Run /
usr/bin/powervc-diag to get powervc logs and configuration files.
Note: Run as root user.


If the files are large, you might need to add more time with "-w".  For example:  powervc-diag -w 9999

For PowerVC 1.4.4, powervc-diag will automatically attempt to collect Novalink data, if there are any Novalink hosts managed by PowerVC.
For PowerVC 2.0.x, powervc-diag will automatically attempt to collect HMC and Novalink data, if there are any HMC or Novalink hosts managed by PowerVC. 

Where powervc-diag generates two files in the same directory, please upload them both.

Example:

powervc-diag.tgz

powervc-diag-ansible.tgz


To collect only PowerVC data, as HMC or Novalink data might be large, add "--management-node-only".  
For example, to collect PowerVC data only, from the PowerVC system and add some time run:
powervc-diag --management-node-only -w 9999

More details and parameters for the powervc-diag command can be found here:
PowerVC 1.4.4:  https://www.ibm.com/docs/en/powervc/1.4.4?topic=problems-powervc-diag-command
PowerVC 2.0.0: https://www.ibm.com/docs/en/powervc/2.0.0?topic=problems-powervc-diag-command
PowerVC 2.0.1:  https://www.ibm.com/docs/en/powervc/2.0.1?topic=problems-powervc-diag-command

Size notes regarding size: https://www.ibm.com/docs/en/powervc/2.0.1?topic=powervc-hardware-software-requirements

Upload to the PMR by this link:
http://www.ecurep.ibm.com/app/upload

Method 2 - Use the QMGCOLLECT Script
A script is provided to help with data collection. The script's intention is to connect via SSH to the PowerVC management host and collect data. After that, it will download the data to the local system via SCP. When that is done, 
to upload the data to IBM via secure FTP (sftp), check the script and instructions as listed here:
http://www.ibm.com/support/docview.wss?uid=nas8N1020618
 
Optional Method:  If all else fails, use the following command to gather data on the PowerVC management host:
tar -cvzf manualpwrvclogs.tar.gz /var/log/ /etc/nova /etc/cinder /etc/neutron /etc/keystone /etc/glance /home/

(This assumes that the system is used only for PowerVC and the /home directory would contain no user files that you would not want to upload.)

Upload to the PMR by this link:

http://www.ecurep.ibm.com/app/upload
 

More data that might be requested:

Depending on the problem, IBM support might ask for: verbose/debug logging, VIOS snap data and HMC or Novalink logs. Sending this data proactively could avoid delays in PMR resolution.

Gather PowerVM NovaLink logs
If your systems are managed via NovaLink (instead of HMC managed), you need to run an extra command at PowerVC 1.3.3 to get the remote logs.

The command is:
powervc-diag --novalinkinfo <NovaLink_host_ip>

For PowerVC 1.4.0 and later, the powervc-diag without any parameters collect NovaLink information.

Manually collect Novalink logs

/var/log/*

To enable verbose/debug logging in PowerVC
Refer to the following document:

Simple method to enable debug logging for PowerVC

Note: If collecting HMC data along with other data (SNAP, IBM i, etc.), please get date/timestamps from all devices for a frame of reference.
HMC logs:

1. Log into HMC as hscpe
2. Run pedbg -c -q 4


Note: If collecting SNAP data with other data (HMC, IBM i, etc.), please get date/timestamps from all devices for a frame of reference.
VIOS snap data:

 
1. Logged in as padmin, gather snap from each node.
If using SSP, a snap from the primary DBN node and failing node is needed.
2. Run $ snap (creates /home/padmin/snap.pax.Z)

[{"Product":{"code":"SSXK2N","label":"PowerVC Standard Edition"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"--","Platform":[{"code":"PF016","label":"Linux"}],"Version":"Version Independent","Edition":"","Line of Business":{"code":"LOB57","label":"Power"}}]

Document Information

Modified date:
07 November 2022

UID

nas8N1020600