MustGather: Collect Troubleshooting Data for the Resolver Component of z/OS Communications Server
The following information is a collection of some commonly seen problems related to the z/OS Communications Server Resolver, as well as the diagnostics output required to diagnose various types of Resolver problems. The Communications Server IP Configuration Guide has information in the sections The Resolver and Configuration Files for TCP/IP Applications that are good general references on the Resolver. The Communications Server IP Configuration Reference has detailed implementation information on the Resolver.
This document replaces informational APARs II13452, II13398, and II13399.
Resolving the problem
|Table of Contents|
- Wrong PROC being used to start the resolver
- EZA0609E, EZA3000I, EZZ6700E, or EZA52281
- Multiple NSINTERADDR statements defined in TCPDATA file, yet only the first name server is queried
- Various GETHOSTBYNAME, GETHOSTBYADDR, NSLOOKUP, and other problems
- Incorrect TCPIP.DATA statements being used when _CEE_ENVFILE JCL is used to specify the RESOLVER_CONFIG environment variable
- What is the Resolver GLOBALTCPIPDATA setup statement
- What is the Resolver DEFAULTTCPIPDATA setup statement
- How can I get more than a single domain name appended for a hostname resolution request?
- How can I influence what the resolver uses for resolution requests?
- How to specify the Trace Resolver output location for the TSO environment
- How to specify the Trace Resolver output location for the z/OS UNIX shell environment
- How to specify the Trace Resolver output location for the MVS batch job environment
- How to specify the Trace Resolver output location for the z/OS batch environment
|Common Problems and Questions|
|Symptom: Wrong PROC being used to start the resolver|
- HOMETEST: EZA0609E Found no IP addresses corresponding to TCP Host Name
- NSLOOKUP: EZA3000I *** Can't find initialize address for serve
- NAMED: EZZ6700E named: could not establish affinity with TCPIPjobname
- SMTP: EZA5228I Name Resolution Method : Host Tables when method should be to use the Nameserver from the NSINTERADDR parameter in TCPIP.DATA
|Symptom: Multiple NSINTERADDR statements are defined in TCPDATA file, yet only the first name server is queried. Why aren't any of the other name servers in the list queried?|
- Getlocalhost fails from java call (running 2 stacks).
- Changes to NSINTERADDR statements not being picked up even after recycling TCPIP.
- NSLOOKUP results in EZB3000I *** Can't find initialize address for server : No response from server.
- NSLOOKUP cannot find default server
- PING hostname results in EZA0457E - UNKNOWN HOST.
- Unable to OPING/OTRACERT using host name - resolver config
- TSO users are not able to FTP to hostnames-tcpdata missing
- FTP batch jobs having problems resolving hostname
- Invoking osnmp -v -h ip01 -c publicpark command results in
EZZ3301I ERROR RETURNED FROM SNMP STARTUP
- SNMP Mgr trace shows
P1: IDSTMVS.I0392675.SOURCE.WS@UTIL(449): Cannot get local IP address
- Domain Name Server (DNS) does not work in batch mode, but does work from TSO
- FTP CLIENT COMMAND: FTP RemoteHostAddr results in
EZA1450I IBM FTP ...
EZA1551I Unknown host: remhost
but FTP IPaddress works fine.
- /etc/hosts not consulted for some applications.
- IP names ( gethostbyname() ) are not being consulted.
This will cause the resolver to reread the TCPIP.DATA statements and the local host files. However, which file it will read is based on if you are in a UNIX or an MVS environment. Additional notes are as follows:
- Refer to the Environment Variable, _BPXK_SETIBMOPT_TRANSPORT, whenever configuring a system with multiple transport drivers or TCP/IP stacks. This environment variable establishes affinity with a particular stack.
- One can invoke PING with the address from the NsInterAddr statementt to confirm that one can reach the defined NameServer.
|Symptom: Incorrect TCPIP.DATA statements being used when _CEE_ENVFILE JCL is used to specify the RESOLVER_CONFIG environment variable.|
The record format of the MVS data set specified by RESOLVER_CONFIG is RECFM=F or RECFM=FB with an LRECL in the range 80 - 256.
For example, if for OMPROUTE the following section of the start procedure's JCL is used:
//OMPROUTE PROC PARMS='' //OMPROUTE EXEC PGM=OMPROUTE,REGION=4096K,TIME=NOLIMIT, // PARM=('POSIX(ON)', // 'ENVAR("_CEE_ENVFILE=DD:STDENV")/&PARMS') //* //STDENV DD DISP=SHR,DSN=USER3.OMPROUTE.ENVIRON .........and the contents of USER3.OMPROUTE.ENVIRON is
RESOLVER_CONFIG=//'USER3.TCPPARMS(TCPDATA)' OMPROUTE_FILE=//'USER3.OMPROUTE.CONFIG'The data set USER3.OMPROUTE.ENVIRON should be RECFM=V and the MVS data set USER3.TCPPARMS should be RECFM=FB or RECFM=F with LRECL in the range 80 - 256. Note also that the data set containing the TCPIP.DATA statements (member TCPDATA in this example) should not have sequence numbers in it. The sequence numbers will be considered parameters on statements that allow multiple parameters. This will lead to syntax error messages in the Trace Resolver output when processing the NSINTERADDR, SEARCH, SORTLIST, LOADDBCSTABLES and LOOKUP statements. TCPIP.DATA statements having syntax errors are ignored.
|Symptom: What is the Resolver GLOBALTCPIPDATA setup statement?|
The intent of the resolver's GLOBALTCPIPDATA setup statement support is to allow an installation's network administrator to ensure that only certain resolver related TCPIP.DATA statements are used. The z/OS Communications Server IP Configuration Guide (SC31-8775 ) has a section The resolver that details the support provided. The section also describes considerations for multiple stack CINET environments where the GLOBALTCPIPDATA statement might not be usable.
When the GLOBALTCPIPDATA is specified, its TCPIP.DATA statements will become the first read. *Any* parameters found in this file will be global settings for the MVS image and all TCPIP stacks. Also if a GLOBALTCPIPDATA has been specified, then all resolver statements will be obtained only from this file. Any of the resolver statements specified in files lower in the search order will be ignored.
Resolver statements are those required by the resolver to process queries. Resolver TCPIP.DATA statements are:
|Symptom: What is the Resolver DEFAULTTCPIPDATA setup statement?|
|Symptom: How can I get more than a single domain name appended for a hostname resolution request?|
|Symptom: How can I influence what the resolver uses for resolution requests?|
|Collecting a Resolver CTRACE|
Resolver CTRACE is primarily used by the IBM support center when analyzing problems with the Communications Server provided TCPIP Resolver.
Additional information about these debugging facilities is contained in Diagnosing resolver problems in the z/OS Communications Server IP Diagnosis Guide. Resolver CTRACE can be written to either an External writer or collected in the Resolver's address space (Internal).
|External Writer CTRACE|
To have Resolver CTRACE written to an external writer data set, a writer proc first needs to be created. This proc either needs to be in SYS1.PROCLIB or in a library concatenated via the MASTER JCL. The amount of cylinder space to be allocated will be provided by the support center.
Sample Writer proc :
//CTWTR1 PROC //IEFPROC EXEC PGM=ITTTRCWR //TRCOUT01 DD DSNAME=IBMUSER.CTRACE1,VOL=SER=xxxxxx, // UNIT=xxxxx,SPACE=(CYL,(xxx),,CONTIG), // DISP=(NEW,CATLG) //SYSPRINT DD SYSOUT=*1. The following step starts the external CTRACE writer. The same CTRACE writer can be used for multiple components; for instance, TCPIP packet trace (SYSTCPDA) and TCPIP stack CTRACE (SYSTCPIP) can share the Resolver CTRACE writer.
Care should be taken when sharing a CTRACE writer because high volumes of trace data might cause trace records to be lost. Another consideration is that the first CTRACE record type recorded will determine which CTRACE formatter will be used. This might require copying the desired Component types to a separate data set for formatting.
See How to Collect PKTTRACE and CTRACE on z/OS f or the TCPIP stack packet trace and TCPIP stack CTRACE procedures.
After the writer has been successfully started, you can proceed with starting the trace.
2. This step starts CTRACE and gives it the Resolver component to use for tracing. The required reply attaches the external writer that was previously started so it can be used to write the CTRACE records.
TRACE CT,ON,COMP=SYSTCPRE,SUB=(resolverprocname) R xx,WTR=CTWTR1,OPTIONS=(ALL),END
Note: If you have not customized the Resolver by creating your own start procedure, the name RESOLVER should be substituted for "resolverprocname". The options of ALL should be used to collect the trace data. The only other option is MINIMUM which is the default but only gives exception trace records. There is the ability to filter CTRACE collection with the JOBNAME= or ASID= parameter.
3. Verify that Trace was started successfully:
The Display command output should show the values of the parameters specified. For example, OPTIONS ALL and WRITER CTWTR1 should be displayed.
4. Recreatethe problem ...
5. Stop CTRACE comp(systcpre) and disconnect writer
TRACE CT,ON,COMP=SYSTCPRE,SUB=(resolverprocname) R xx,WTR=DISCONNECT,END TRACE CT,OFF,COMP=SYSTCPRE,SUB=(resolverprocname) TRACE CT,WTRSTOP=CTWTR1,FLUSH
|Resolver Address Space CTRACE|
Because the Resolver address space CTRACE data will be captured via an MVS console Dump command, an external CTRACE writer will not be needed. Be aware that collecting CTRACE data in the Resolver address space might result in lost trace data because trace buffers wrapping is very possible. The dump command should be issued as soon as the problem happens. Another option to reduce trace wrapping would be to collect the dump via a trap or slip command given by the support center.
1. Starting Resolver CTRACE comp(SYSTCPRE):
TRACE CT,ON,COMP=SYSTCPRE,SUB=(resolverprocname) R XX,OPTIONS=(ALL),END
Note: If you have not customized the Resolver by creating your own start procedure, the name RESOLVER should be substituted for "resolverprocname". The options of ALL should be used to collect the trace data. The only other option is MINIMUM which is the default but only gives exception trace records.
There is the ability to filter CTRACE collection with the JOBNAME= or ASID= parameter.
2. Verify that Trace was started successfully.
The Display command output should show the values of the parameters specified. For example, OPTIONS ALL should be displayed.
3. Establish the console dump environment to dump the Resolver address space. Note the usage of the "CONT", continue, option which returns a reply number.
Issue the following MVS command from the system console:
DUMP COMM=('text') R xx,JOBNAME=(resolverprocname),CONT R xx,SDATA=(ALLNUC,CSA,LPA,LSQA,RGN,SWA,SQA,TRT,PSA,GRSQ),CONT
If the support center has requested other address spaces to be dumped, they can be added to the JOBNAME= parameter.
4. Recreate the problem ...
5. Collect the Resolver address space dump by replying to the outstanding reply number from the DUMP COMM command.
6. Stop Resolver CTRACE:
|Collecting TRACE RESOLVER Output|
Trace Resolver output can be used by the IBM support center as well as application programmers and networking system programmers to diagnose problems in resolving IP hostnames to IP addresses or IP addresses to IP hostnames. Trace Resolver is also helpful in determining the values of TCPIP.DATA statements and where their values were obtained.
Additional information about these debugging facilities is contained in Diagnosing Resolver Problems in the z/OS Communications Server IP Diagnosis Guide .
Trace Resolver output can be activated in one of the following ways:
- Specifying the z/OS UNIX (C/C++ only) RESOLVER_TRACE environment variable or a SYSTCPT DD allocation. Specification of the RESOLVER_TRACE environment variable or allocation of the SYSTCPT DDname will dynamically activate Trace Resolver output regardless of the TCPIP.DATA or the _res structure's resDebug specification. This dynamic activation of Trace Resolver can be useful when you are not sure where the TCPIP.DATA statements will be found.
- Specifying the TCPIP.DATA statement
- Setting the debug option (resDebug) in an application's _res structure
The search order used by the resolver to determine if Trace Resolver output is desired follows:
- Environment variable RESOLVER_TRACE (z/OS UNIX C/C++ only)
- SYSTCPT DD allocation
- TRACE RESOLVER or OPTIONS DEBUG statement stdout or SYSPRINT must be allocated or no trace data is generated. If using either of these TCPIP.DATA statements, they should be the *first* TCPIP.DATA statement so that all trace information can be collected.
- resDebug bit that is set on in the options field of the _res structure stdout or SYSPRINT must be allocated or no trace data is generated.
- A TSO user's terminal screen
- z/OS UNIX standard out
- JES sysout
- An MVS sequential data set (member of a PDS is not supported). The data set must already exist or be allocated as NEW with the following DCB characteristics. An LRECL in the range 80 - 256 with an RECFM of Fixed Block. For an LRECL of 128 or larger, the last six print positions will be the storage address of the MVS TCB that issued the resolver call. This can be helpful with multitask applications.
- An HFS file. The file can either be an existing file or it will be dynamically allocated by the resolver when needed. The maximum line length used in the file is 255 characters. The last six print positions will be the storage address of the MVS TCB that issued the resolver call. This can be helpful with multitask applications.
If the Trace Resolver output is using an MVS data set or HFS file, the output will be for the resolver services invoked by the last command or UNIX process. If possible, it is recommended to use SYSOUT=* or z/OS UNIX standard out so that multiple resolver service invocations (for example, a multitask environment) can be traced.
|How to specify the Trace Resolver output location for the TSO environment|
- To use the user's terminal
alloc dd(systcpt) da(*)
alloc dd(systcpt) da(appl.restrace)The userid will be used as the first qualifier for the data set. To specify a fully qualified data set name, use single quotation marks:
alloc dd(systcpt) da('user3.resolver.trace')
|How to specify the Trace Resolver output location for the z/OS UNIX shell environment|
- To use standard out:
If desired, stdout can be redirected when the z/OS UNIX command is issued. If your application was compiled with the z/OS C/C++ Language Environment Native ASCII support, do not use STDOUT. If you use STDOUT with ASCII programs, the trace data will not be readable. Instead, send the trace data to an MVS data set or HFS file as described below.
- To use a new HFS file or existing MVS data set:
export RESOLVER_TRACE=/tmp/myjob.resolv.trace export RESOLVER_TRACE="//appl.restrace"The userid will be used as the first qualifier for the data set.
- To disable the Trace Resolver output:
set -A RESOLVER_TRACE
|How to specify the Trace Resolver output location for the MVS batch job environment|
- To use the recommended JES SYSOUT
//SYSTCPT DD SYSOUT=* //SYSPRINT DD SYSOUT=*
|How to specify the Trace Resolver output location for the z/OS batch environment|
- If the application resides in an HFS file, you should use BPXBATSL to run the program because DD allocations will be passed to the application. If the application does forks, the DD allocations will not be passed to the new process so the Trace Resolver output can no longer be collected for these type applications.
- To use the recommended JES SYSOUT
//SYSTCPT DD SYSOUT=*
- Because BPXBATSL cannot allocated stdout to SYSOUT=*, either of the following STDOUT DD JCL statements could be used. Note for the HFS file that OTRUNC is not specified on the PATHOPTS so that Trace Resolver output will be appended to the HFS file. You will need to ensure that the file does not fill the specified directory (/tmp/).
//STDOUT DD DISP=SHR,DSN=USER3.APPL.RESTRACE //STDOUT DD PATH='/tmp/appl.stdout', // PATHOPTS=(OWRONLY,OCREAT), // PATHMODE=SIRWXU
//STDENV DD DISP=SHR,DSN=USER3.APPL.ENVIRONThe STDENV data set can be of a variable (nonspanned) or fixed record format. It can contain multiple environment variables. Sample contents are shown below. Note that environment variables must start in column 1 and the data set must not have any sequence numbers in it because sequence numbers will be treated as part of the environment variable.
For the RESOLVER_TRACE environment variable, any blanks from a fixed format STDENV data set will be removed. This might not be true for all variables so a variable record format data set is recommended.
RESOLVER_TRACE=//'USER3.APPL.RESTRACE' _BPXK_SETIBMOPT_TRANSPORT=TCPCSFor applications that fork, it is recommended than an MVS data set is used as usage of an HFS file might result in a C03 ABEND when the forked process ends.
More support for:
z/OS Communications Server
Software version: 1.6, 1.7, 1.8, 1.9, 1.10, 1.11, 1.12, 1.13, 2.1, 2.2
Operating system(s): z/OS
Reference #: 1316784
Modified date: 18 April 2013