IBM Support

MustGather: Collect Troubleshooting Data for the Resolver Component of z/OS Communications Server

Troubleshooting


Problem

The following information is a collection of some commonly seen problems related to the z/OS Communications Server Resolver, as well as the diagnostics output required to diagnose various types of Resolver problems. The [Communications Server IP Configuration Guide] has information in the sections [The Resolver] and [Configuration Files for TCP/IP Applications] that are good general references on the Resolver. The [Communications Server IP Configuration Reference] has detailed implementation information on the Resolver. This document replaces informational APARs II13452, II13398, and II13399.

Diagnosing The Problem

Resolving The Problem

Table of Contents

Common Problems and Questions Collect a Resolver CTRACE Collecting TRACE RESOLVER Output

Common Problems and Questions


Symptom: Wrong PROC being used to start the resolver
Resolution: If you code RESOLVER_PROC(DEFAULT) in the BPXPMRxx member, the resolver will use hardcoded defaults for its configuration options. If you want to specify the setup configuration options for the resolver, you need to create a resolver JCL procedure in your procedure library and also code RESOLVER_PROC(your_resolver_proc) in your BPXPRMxx member. If you do not code RESOLVER_PROC in the BPXPMRxx member, the resolver will use RESOLVER_PROC(DEFAULT).
Symptom:
  • HOMETEST: EZA0609E Found no IP addresses corresponding to TCP Host Name
  • NSLOOKUP: EZA3000I ***  Can't find initialize address for serve
  • NAMED: EZZ6700E named: could not establish affinity with TCPIPjobname
  • SMTP: EZA5228I Name Resolution Method : Host Tables when method should be to use the Nameserver from the NSINTERADDR parameter in TCPIP.DATA
Resolution: Ensure that there are no extraneous characters in your TCPIP.DATA such as sequence numbers. They can cause problems with the parsing of the data set.
Symptom: Multiple NSINTERADDR statements are defined in TCPDATA file, yet only the first name server is queried. Why aren't any of the other name servers in the list queried?
Resolution: After the resolver has successfully contacted a name server, it stops without contacting the remaining name servers for that query. A successful contact includes NEGATIVE responses, such as NXDOMAIN (nonexistent domain). Name servers beyond the first in the list are used only if the name server currently being contacted is down or unreachable through the network. ResolverUdpRetries indicates the maximum number of times an attempt is made to reach a given name server if a response is not received within the current timeout interval. ResolverUdpRetries is applicable if ResolveVia UDP is coded or used by default.
Symptom:
  • Getlocalhost fails from java call (running 2 stacks).
  • Changes to NSINTERADDR statements not being picked up even after recycling TCPIP.
  • NSLOOKUP results in EZB3000I *** Can't find initialize address for server : No response from server.
  • NSLOOKUP cannot find default server
  • PING hostname results in EZA0457E - UNKNOWN HOST.
  • Unable to OPING/OTRACERT using host name - resolver config
  • TSO users are not able to FTP to hostnames-tcpdata missing
  • FTP batch jobs having problems resolving hostname
  • Invoking osnmp -v -h ip01 -c publicpark command results in
    EZZ3301I ERROR RETURNED FROM SNMP STARTUP
  • SNMP Mgr trace shows
    P1: IDSTMVS.I0392675.SOURCE.WS@UTIL(449): Cannot get local IP address
  • Domain Name Server (DNS) does not work in batch mode, but does work from TSO
  • FTP CLIENT COMMAND: FTP RemoteHostAddr results in
    EZA1450I IBM FTP ...
    EZA1551I Unknown host: remhost
    but FTP IPaddress works fine.
  • /etc/hosts not consulted for some applications.
  • IP names ( gethostbyname() ) are not being consulted.
Resolution: The above problems are usually a result of having the incorrect resolver configuration files being used by an application. The solution to these symptoms is to understand the search orders used in native MVS and z/OS UNIX environments, and configure your system properly. The best approach is to review Resolver Configuration Files and ensure that your system is using the intended configuration files. For long running applications, for example, servers, it is possible with z/OS V1R2 and later releases to have them use updated resolver files without stopping and restarting them by invoking the system operator command: 
Modify Resolver,Refresh

This will cause the resolver to reread the TCPIP.DATA statements and the local host files. However, which file it will read is based on if you are in a UNIX or an MVS environment. Additional notes are as follows:
  • Refer to the Environment Variable, _BPXK_SETIBMOPT_TRANSPORT, whenever configuring a system with multiple transport drivers or TCP/IP stacks. This environment variable establishes affinity with a particular stack.
  • One can invoke PING with the address from the NsInterAddr statementt to confirm that one can reach the defined NameServer.
Symptom: Incorrect TCPIP.DATA statements being used when _CEE_ENVFILE JCL is used to specify the RESOLVER_CONFIG environment variable.
Resolution: When using a _CEE_ENVFILE DD card in JCL, ensure that the record format of the MVS data set specified by the _CEE_ENVFILE DD is RECFM=V or RECFM=VB. This is recommended by z/OS Language Environment so that trailing blanks from a fixed format data set will not be used as part of the environment variable's value.
The record format of the MVS data set specified by RESOLVER_CONFIG is RECFM=F or RECFM=FB with an LRECL in the range 80 - 256.
For example, if for OMPROUTE the following section of the start procedure's JCL is used: 
    //OMPROUTE PROC PARMS=''
//OMPROUTE EXEC PGM=OMPROUTE,REGION=4096K,TIME=NOLIMIT, 
//    PARM=('POSIX(ON)', 
//          'ENVAR("_CEE_ENVFILE=DD:STDENV")/&PARMS') 
//* 
//STDENV   DD DISP=SHR,DSN=USER3.OMPROUTE.ENVIRON 
            .........
and the contents of USER3.OMPROUTE.ENVIRON is 
    RESOLVER_CONFIG=//'USER3.TCPPARMS(TCPDATA)'
OMPROUTE_FILE=//'USER3.OMPROUTE.CONFIG'
The data set USER3.OMPROUTE.ENVIRON should be RECFM=V and the MVS data set USER3.TCPPARMS should be RECFM=FB or RECFM=F with LRECL in the range 80 - 256. Note also that the data set containing the TCPIP.DATA statements (member TCPDATA in this example) should not have sequence numbers in it. The sequence numbers will be considered parameters on statements that allow multiple parameters. This will lead to syntax error messages in the Trace Resolver output when processing the NSINTERADDR, SEARCH, SORTLIST, LOADDBCSTABLES and LOOKUP statements. TCPIP.DATA statements having syntax errors are ignored.
Symptom: What is the Resolver GLOBALTCPIPDATA setup statement?
Resolution: The Resolver can be configured so that installation-wide TCPIP.DATA statements will be used. These statements will be used regardless of what a batch job or interactive user has specified for its TCPIP.DATA.

The intent of the resolver's GLOBALTCPIPDATA setup statement support is to allow an installation's network administrator to ensure that only certain resolver related TCPIP.DATA statements are used. The z/OS Communications Server IP Configuration Guide (SC31-8775 ) has a section The resolver that details the support provided. The section also describes considerations for multiple stack CINET environments where the GLOBALTCPIPDATA statement might not be usable.

When the GLOBALTCPIPDATA is specified, its TCPIP.DATA statements will become the first read. *Any* parameters found in this file will be global settings for the MVS image and all TCPIP stacks. Also if a GLOBALTCPIPDATA has been specified, then all resolver statements will be obtained only from this file. Any of the resolver statements specified in files lower in the search order will be ignored.

Resolver statements are those required by the resolver to process queries. Resolver TCPIP.DATA statements are:
  • DomainOrigin/Domain
  • NSInterAddr/NameServer
  • NSPortAddr
  • ResolveVia
  • ResolverTimeOut
  • ResolverUDPRetries
  • Search
  • SortList
Other statements not specified in GLOBALTCPIPDATA can still be located in one of the TCPIP.DATA files in the search order. The z/OS Communications Server IP Configuration Guide's Resolver configuration files section describes the search order used for the native MVS and z/OS UNIX environment.
Symptom: What is the Resolver DEFAULTTCPIPDATA setup statement?
Resolution: The Resolver can be configured so that an installation can replace the final location that TCPIP.DATA statements are searched for. Before z/OS V1R2, only TCPIP.TCPIP.DATA could be used as the final location; now with the DEFAULTTCPIPDATA statement, any MVS data set or HFS file can be specified.
Symptom: How can I get more than a single domain name appended for a hostname resolution request?
Resolution: There is a SEARCH TCPIP.DATA statement. On the SEARCH you can specify an ordered list of up to 6 domain names. The domain names will be appended in order when trying to resolve a hostname to an IP address. The resolution will stop with the first successful resolution. The first SEARCH domain name listed is used as the local DomainOrigin value. The SEARCH and DomainOrigin statements are mutually exclusive. If both are in the TCPIP.DATA statements, the last valid statement found is used.
Symptom: How can I influence what the resolver uses for resolution requests?
Resolution: There is a LOOKUP TCPIP.DATA statement. The LOOKUP statement allows you to specify the order in which the DNS and local host files are to be used for resolution requests. The statement's parameters are DNS or LOCAL, where DNS means using the name servers specified by the TCPIP.DATA NSINTERADDR statements and LOCAL means using the local host files. Either or both parameters can be specified. The default is LOOKUP DNS LOCAL.
Collecting a Resolver CTRACE

Resolver CTRACE is primarily used by the IBM support center when analyzing problems with the Communications Server provided TCPIP Resolver.

Additional information about these debugging facilities is contained in Diagnosing resolver problems in the z/OS Communications Server IP Diagnosis Guide. Resolver CTRACE can be written to either an External writer or collected in the Resolver's address space (Internal).

    External Writer CTRACE

    To have Resolver CTRACE written to an external writer data set, a writer proc first needs to be created. This proc either needs to be in SYS1.PROCLIB or in a library concatenated via the MASTER JCL. The amount of cylinder space to be allocated will be provided by the support center.
    Sample Writer proc : 
      //CTWTR1 PROC
//IEFPROC  EXEC PGM=ITTTRCWR
//TRCOUT01 DD DSNAME=IBMUSER.CTRACE1,VOL=SER=xxxxxx,
//         UNIT=xxxxx,SPACE=(CYL,(xxx),,CONTIG),
//         DISP=(NEW,CATLG)
//SYSPRINT DD SYSOUT=*
    1. The following step starts the external CTRACE writer. The same CTRACE writer can be used for multiple components; for instance, TCPIP packet trace (SYSTCPDA) and TCPIP stack CTRACE (SYSTCPIP) can share the Resolver CTRACE writer.

    Care should be taken when sharing a CTRACE writer because high volumes of trace data might cause trace records to be lost. Another consideration is that the first CTRACE record type recorded will determine which CTRACE formatter will be used. This might require copying the desired Component types to a separate data set for formatting.

    See How to Collect PKTTRACE and CTRACE on z/OS for the TCPIP stack packet trace and TCPIP stack CTRACE procedures. 
         TRACE CT,WTRSTART=CTWTR1

    After the writer has been successfully started, you can proceed with starting the trace.

    2. This step starts CTRACE and gives it the Resolver component to use for tracing. The required reply attaches the external writer that was previously started so it can be used to write the CTRACE records. 
         TRACE CT,ON,COMP=SYSTCPRE,SUB=(resolverprocname)
     R xx,WTR=CTWTR1,OPTIONS=(ALL),END

    Note: If you have not customized the Resolver by creating your own start procedure, the name RESOLVER should be substituted for "resolverprocname". The options of ALL should be used to collect the trace data. The only other option is MINIMUM which is the default but only gives exception trace records. There is the ability to filter CTRACE collection with the JOBNAME= or ASID= parameter.

    3. Verify that Trace was started successfully: 
         D TRACE,COMP=SYSTCPRE,SUB=(resolverprocname)

    The Display command output should show the values of the parameters specified. For example, OPTIONS ALL and WRITER CTWTR1 should be displayed.

    4. Recreatethe problem ...

    5. Stop CTRACE comp(systcpre) and disconnect writer 
         TRACE CT,ON,COMP=SYSTCPRE,SUB=(resolverprocname)
     R xx,WTR=DISCONNECT,END
     TRACE CT,OFF,COMP=SYSTCPRE,SUB=(resolverprocname)
     TRACE CT,WTRSTOP=CTWTR1,FLUSH



    Resolver Address Space CTRACE

    Because the Resolver address space CTRACE data will be captured via an MVS console Dump command, an external CTRACE writer will not be needed. Be aware that collecting CTRACE data in the Resolver address space might result in lost trace data because trace buffers wrapping is very possible. The dump command should be issued as soon as the problem happens. Another option to reduce trace wrapping would be to collect the dump via a trap or slip command given by the support center.

    1. Starting Resolver CTRACE comp(SYSTCPRE): 
         TRACE CT,ON,COMP=SYSTCPRE,SUB=(resolverprocname)
     R XX,OPTIONS=(ALL),END

    Note: If you have not customized the Resolver by creating your own start procedure, the name RESOLVER should be substituted for "resolverprocname". The options of ALL should be used to collect the trace data. The only other option is MINIMUM which is the default but only gives exception trace records.

    There is the ability to filter CTRACE collection with the JOBNAME= or ASID= parameter.

    2. Verify that Trace was started successfully. 
         D TRACE,COMP=SYSTCPRE,SUB=(resolverprocname)

    The Display command output should show the values of the parameters specified. For example, OPTIONS ALL should be displayed.

    3. Establish the console dump environment to dump the Resolver address space. Note the usage of the "CONT", continue, option which returns a reply number.
    Issue the following MVS command from the system console: 
         DUMP COMM=('text')
     R xx,JOBNAME=(resolverprocname),CONT
     R xx,SDATA=(ALLNUC,CSA,LPA,LSQA,RGN,SWA,SQA,TRT,PSA,GRSQ),CONT

    If the support center has requested other address spaces to be dumped, they can be added to the JOBNAME= parameter.

    4. Recreate the problem ...

    5. Collect the Resolver address space dump by replying to the outstanding reply number from the DUMP COMM command. 
         R xx,END

    6. Stop Resolver CTRACE: 
         TRACE CT,OFF,COMP=SYSTCPRE,SUB=(resolverprocname)
Collecting TRACE RESOLVER Output

Trace Resolver output can be used by the IBM support center as well as application programmers and networking system programmers to diagnose problems in resolving IP hostnames to IP addresses or IP addresses to IP hostnames. Trace Resolver is also helpful in determining the values of TCPIP.DATA statements and where their values were obtained.

Additional information about these debugging facilities is contained in Diagnosing Resolver Problems in the z/OS Communications Server IP Diagnosis Guide.

Trace Resolver output can be activated in one of the following ways:
  1. Specifying the z/OS UNIX (C/C++ only) RESOLVER_TRACE environment variable or a SYSTCPT DD allocation. Specification of the RESOLVER_TRACE environment variable or allocation of the SYSTCPT DDname will dynamically activate Trace Resolver output regardless of the TCPIP.DATA or the _res structure's resDebug specification. This dynamic activation of Trace Resolver can be useful when you are not sure where the TCPIP.DATA statements will be found.
  2. Specifying the TCPIP.DATA statement 
         TRACE RESOLVER
    or 
         OPTIONS DEBUG
  3. Setting the debug option (resDebug) in an application's _res structure

The search order used by the resolver to determine if Trace Resolver output is desired follows:
  1. Environment variable RESOLVER_TRACE (z/OS UNIX C/C++ only)

  2. SYSTCPT DD allocation

  3. TRACE RESOLVER or OPTIONS DEBUG statement stdout or SYSPRINT must be allocated or no trace data is generated. If using either of these TCPIP.DATA statements, they should be the *first* TCPIP.DATA statement so that all trace information can be collected.

  4. resDebug bit that is set on in the options field of the _res structure stdout or SYSPRINT must be allocated or no trace data is generated.

Trace Resolver output can be written to
  • A TSO user's terminal screen
  • z/OS UNIX standard out
  • JES sysout
  • An MVS sequential data set (member of a PDS is not supported). The data set must already exist or be allocated as NEW with the following DCB characteristics. An LRECL in the range 80 - 256 with an RECFM of Fixed Block. For an LRECL of 128 or larger, the last six print positions will be the storage address of the MVS TCB that issued the resolver call. This can be helpful with multitask applications.
  • An HFS file. The file can either be an existing file or it will be dynamically allocated by the resolver when needed. The maximum line length used in the file is 255 characters. The last six print positions will be the storage address of the MVS TCB that issued the resolver call. This can be helpful with multitask applications.

    If the Trace Resolver output is using an MVS data set or HFS file, the output will be for the resolver services invoked by the last command or UNIX process. If possible, it is recommended to use SYSOUT=* or z/OS UNIX standard out so that multiple resolver service invocations (for example, a multitask environment) can be traced.

How to specify the Trace Resolver output location for the TSO environment
  • To use the user's terminal 
      • alloc dd(systcpt) da(*)
    When directing Trace Resolver output to a TSO terminal, the screen size must be defined to be only 80 columns wide. Wider screen sizes will cause trace output to be wrapped and hard to read.

  • To use an existing MVS data set:
      • alloc dd(systcpt) da(appl.restrace)
    The userid will be used as the first qualifier for the data set. To specify a fully qualified data set name, use single quotation marks: 
      alloc dd(systcpt) da('user3.resolver.trace')
  • To disable the Trace Resolver output:
      • free dd(systcpt)

How to specify the Trace Resolver output location for the z/OS UNIX shell environment
    • To use standard out:

      • export RESOLVER_TRACE=stdout

      If desired, stdout can be redirected when the z/OS UNIX command is issued. If your application was compiled with the z/OS C/C++ Language Environment Native ASCII support, do not use STDOUT. If you use STDOUT with ASCII programs, the trace data will not be readable. Instead, send the trace data to an MVS data set or HFS file as described below.

    • To use a new HFS file or existing MVS data set:
        • export RESOLVER_TRACE=/tmp/myjob.resolv.trace
export RESOLVER_TRACE="//appl.restrace"
        The userid will be used as the first qualifier for the data set.
        export RESOLVER_TRACE="//'user3.resolver.trace'"
    • To disable the Trace Resolver output:
        • set -A RESOLVER_TRACE

How to specify the Trace Resolver output location for the MVS batch job environment
    • To use the recommended JES SYSOUT
        • //SYSTCPT  DD  SYSOUT=*
//SYSPRINT DD  SYSOUT=*
      The SYSPRINT DD must be allocated if the TCPIP.DATA statements TRACE RESOLVER or OPTIONS DEBUG are specified. If the DD does not exist, no trace output will be written.

How to specify the Trace Resolver output location for the z/OS batch environment
    • If the application resides in an HFS file, you should use BPXBATSL to run the program because DD allocations will be passed to the application. If the application does forks, the DD allocations will not be passed to the new process so the Trace Resolver output can no longer be collected for these type applications.

    • To use the recommended JES SYSOUT
        • //SYSTCPT  DD  SYSOUT=*
    • Because BPXBATSL cannot allocated stdout to SYSOUT=*, either of the following STDOUT DD JCL statements could be used. Note for the HFS file that OTRUNC is not specified on the PATHOPTS so that Trace Resolver output will be appended to the HFS file. You will need to ensure that the file does not fill the specified directory (/tmp/).
        • //STDOUT   DD DISP=SHR,DSN=USER3.APPL.RESTRACE
//STDOUT  DD PATH='/tmp/appl.stdout',
//  PATHOPTS=(OWRONLY,OCREAT),
//  PATHMODE=SIRWXU
    • The STDOUT DD must be allocated if the TCPIP.DATA statements TRACE RESOLVER or OPTIONS DEBUG are specified. If the DD does not exist, no trace output will be written.

    • The RESOLVER_TRACE environment variable can be passed by BPXBATSL or BPXBATCH using a //STDENV DD JCL statement. Below is an example:
        • //STDENV   DD  DISP=SHR,DSN=USER3.APPL.ENVIRON
      The STDENV data set can be of a variable (nonspanned) or fixed record format. It can contain multiple environment variables. Sample contents are shown below. Note that environment variables must start in column 1 and the data set must not have any sequence numbers in it because sequence numbers will be treated as part of the environment variable.


      For the RESOLVER_TRACE environment variable, any blanks from a fixed format STDENV data set will be removed. This might not be true for all variables so a variable record format data set is recommended. 
        RESOLVER_TRACE=//'USER3.APPL.RESTRACE' 
_BPXK_SETIBMOPT_TRANSPORT=TCPCS
      For applications that fork, it is recommended than an MVS data set is used as usage of an HFS file might result in a C03 ABEND when the forked process ends.

[{"Product":{"code":"SSSN3L","label":"z\/OS Communications Server"},"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Component":"All","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"1.6;1.7;1.8;1.9;1.10;1.11;1.12;1.13;2.1;2.2","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}}]

Document Information

Modified date:
15 June 2018

UID

swg21316784

Page Feedback