Question & Answer
Question
What data is needed to debug PowerVC problems?
Answer
IBM PowerVC – Virtualization Center is designed to improve and simplify the management of virtual machines (VMs) and LPARs on Power Systems servers. PowerVC is built on OpenStack services, which allows the necessary interaction to critical underlying components--most notably, HMC and PowerVM (VIOS).
For any problem, a clear problem description and environment details are required: |
A clear description of your PowerVC software and hardware environment.
a) Includes the current fix and service pack level of:
PowerVC
HMC/Novalink
VIOS
b) Error messages and screen capture of the errors your see, including TIMESTAMPS for errors seen or actions you take during the recreation of a problem.
c) Time difference between the PowerVC Linux server, HMC/Novalink, and VIOS servers. Use the "date" command, and let us know whether the systems are synchronized by NTP.
For a PowerVC installation problem or error, collect: |
You can clear your logs: |
If powervc-diag creates a data collection that is file that is too large, you can clear your logs.
Stop PowerVC services: powervc-services stop
Clear the content of these directories (ONLY):
/var/log/nova
/var/log/cinder
/var/log/keystone
Start PowerVC services: powervc-services start
Then, re-create the failure and make sure to note the timestamp of the error before you collect a powervc-diag again.
Collect the following PowerVC data for all postinstallation problems: |
Method 1 - Run powervc-diag manually
Use the powervc-diag command.
Run /usr/bin/powervc-diag to get powervc logs and configuration files.
Note: Run as root user.
If the files are large, you might need to add more time with "-w". For example: powervc-diag -w 9999
For PowerVC 1.4.4, powervc-diag will automatically attempt to collect Novalink data, if there are any Novalink hosts managed by PowerVC.
For PowerVC 2.0.x, powervc-diag will automatically attempt to collect HMC and Novalink data, if there are any HMC or Novalink hosts managed by PowerVC.
Where powervc-diag generates two files in the same directory, please upload them both.
Example:
powervc-diag.tgz
powervc-diag-ansible.tgz
To collect only PowerVC data, as HMC or Novalink data might be large, add "--management-node-only".
For example, to collect PowerVC data only, from the PowerVC system and add some time run:
powervc-diag --management-node-only -w 9999
More details and parameters for the powervc-diag command can be found here:
PowerVC 1.4.4: https://www.ibm.com/docs/en/powervc/1.4.4?topic=problems-powervc-diag-command
PowerVC 2.0.0: https://www.ibm.com/docs/en/powervc/2.0.0?topic=problems-powervc-diag-command
PowerVC 2.0.1: https://www.ibm.com/docs/en/powervc/2.0.1?topic=problems-powervc-diag-command
Size notes regarding size: https://www.ibm.com/docs/en/powervc/2.0.1?topic=powervc-hardware-software-requirements
Upload to the PMR by this link:
http://www.ecurep.ibm.com/app/upload
Method 2 - Use the QMGCOLLECT Script
A script is provided to help with data collection. The script's intention is to connect via SSH to the PowerVC management host and collect data. After that, it will download the data to the local system via SCP. When that is done, to upload the data to IBM via secure FTP (sftp), check the script and instructions as listed here:
http://www.ibm.com/support/docview.wss?uid=nas8N1020618
Optional Method: If all else fails, use the following command to gather data on the PowerVC management host:
tar -cvzf manualpwrvclogs.tar.gz /var/log/ /etc/nova /etc/cinder /etc/neutron /etc/keystone /etc/glance /home/
(This assumes that the system is used only for PowerVC and the /home directory would contain no user files that you would not want to upload.)
Upload to the PMR by this link:
http://www.ecurep.ibm.com/app/upload
More data that might be requested: |
Depending on the problem, IBM support might ask for: verbose/debug logging, VIOS snap data and HMC or Novalink logs. Sending this data proactively could avoid delays in PMR resolution.
Gather PowerVM NovaLink logs
If your systems are managed via NovaLink (instead of HMC managed), you need to run an extra command at PowerVC 1.3.3 to get the remote logs.
The command is:
powervc-diag --novalinkinfo <NovaLink_host_ip>
For PowerVC 1.4.0 and later, the powervc-diag without any parameters collect NovaLink information.
Manually collect Novalink logs
/var/log/*
To enable verbose/debug logging in PowerVC
Refer to the following document:
Simple method to enable debug logging for PowerVC
Note: If collecting HMC data along with other data (SNAP, IBM i, etc.), please get date/timestamps from all devices for a frame of reference.
HMC logs:
1. | Log into HMC as hscpe |
2. | Run pedbg -c -q 4 |
Note: If collecting SNAP data with other data (HMC, IBM i, etc.), please get date/timestamps from all devices for a frame of reference.
VIOS snap data:
1. | Logged in as padmin, gather snap from each node. If using SSP, a snap from the primary DBN node and failing node is needed. |
2. | Run $ snap (creates /home/padmin/snap.pax.Z) |
Was this topic helpful?
Document Information
Modified date:
07 November 2022
UID
nas8N1020600