Troubleshooting RFC Connections for Information Server Pack for SAP BW and Applications

Resolving The Problem

The steps described in this technote should be the first troubleshooting action for BW issues. If still encountering difficulties, see also technote 1441967  -  Troubleshooting hanging BW Load or Extract jobs that might terminate with a "timeout" message

The SAP Packs consist of 4 components:

• Client tier Graphical user interface (used to edit the stage properties in a Datastage job)
• Client tier Datastage Administrator for SAP - used to edit SAP Connections and setup Source Systems and IDOC types
• Engine tier Runtime (used on the server side when the job is run)
• Engine tier Listener / RFC Server which runs as an independent process to receive:
  - notifications arriving from the SAP BW Server
  - IDOCs sent from the SAP ERP system

Contents

Generic Listener troubleshooting
BW Load Runtime troubleshooting

Source system connection
InfoSource and InfoPackage properties
InfoPackage log in SAP
Process Chain

BW Extract OpenHub Runtime troubleshooting

Settings checkup
Monitoring the InfoSpoke and Process Chain
Resetting a previous request
Additional OHD request status considerations

Other notes

Enabling the RFC Trace
Manual cleanup of listener temporary files

• Generic Listener troubleshooting
  
  This is common between the Pack for SAP BW and Applications.
  First, use a Program ID in the form: <DataStage Server box name>.<SAP box name>.<usage>
  where usage can be idoc (for an IDOC listener) and bw (for a BW source system).
  Example: vdraganx.bocasap5.bwextract
  
  Important: the Program ID needs to be unique, no two listeners can share the same id. This is the single most often cause of errors. See the steps below on how to determine if an existing listener uses a unique Program ID.
  Instead of this potentially lengthy troubleshooting the user may choose to use a new Program ID (for example adding a prefix or suffix to the previous id). Using a new id is recommended whenever possible.
  
  Steps to troubleshoot listener issues

  1. Check the listener logs -  usually contains an error indicate of the issue.
  Applications: $DSSAPHOME/DSSAPConnections/[Connection]/IDocListener.log, dsidocsvr.log
  BW: $DSSAPHOME/DSBWConnections/[ConnectionName]/SourceSystems/[SourceSystemName]/Data/RFCServer*.log
  For example if this error is shown it means registration of RFC destination is not allowed from the SAP host. Check SAP Profile parameter 'gw/acl_mode' using t-code RZ11. If its value is 1 make it 0(zero) to allow Data Stage to register the RFC destination.
  Fatal: RFC server function 'rfc_getline' failed, returned '765:ERROR: RFC error has occured with the following details: %1:Error Details: BAPI: RfcListenAndDispatch Code: RFC_COMMUNICATION_FAILURE Key: RFC_COMMUNICATION_FAILURE Message: LOCATION SAP-Gateway on host SRVBRA153.tods.com / sapgw00
  ERROR registration of tp ZRFC_ZRT_TP_CALC_IVA from host srvibm4bids.tods.com not allowed

  2. Check the connection from the SAP side -  after making sure the listener is started and there are no further errors in the log:

  • In SAP user interface, run transaction sm59 and test the RFC destination involved in the connection.
  • If the connection test fails, you have no connection. See note below.
  • If the connection test succeeds, but you do not get the expected results when running InfoSphere DataStage, such as IDOCS not arriving, jobs hanging with "Waiting for BW " or aborting with " Timeout (...) " you may have duplicated Program IDs. Go to step 4.
  • Stop the listener on the InfoSphere DataStage side (IDOC Manager, RFC Manager). Ensure that that all processes are stopped. Otherwise use the kill command to stop a process: dsrfcmgr, dsrfcsvr, dsidocmgr, dsidocsvr
  • Logon to SAP, sm59 and test the connection: you want the test to fail as shown below.
  • If the test is successful, then there is another listener sharing the Program ID; remove the duplicate
    

  
  To find out what system is using the listener, from the menu, click System Information/Target System . Read the Host name/Network address to identify the system. You need to stop the listener on that system or change the Program ID that you are using. Then repeat these steps.
  

  • If the tests above indicate no duplicate listeners between different DataStage systems, there can still be a duplicate listener within the same system. To verify that:
    - For IDOC: open each connection properties in DS Administrator for SAP and verify the Program ID. It should be unique. You can also disable individual Connection listeners for testing purposes: Connection properties/IDOC Listener Settings tab/disable "Listen for IDOCs"
    - For BW: open each connection properties in DS Administrator for SAP then click on Source Systems. For each Source System click Properties and verify the Program ID on the "RFC Server Configuration" tab. It should be unique. You can also disable individual Source system listeners for testing purposes by disabling "Listen for BW Load requests" on the same tab.

  
  3. Investigate the DS Server side  

  • Stop the Listener component
    • Turn on the RFC trace (so that the environment variables are visible to the listening component). For more information, see “Notes/Enabling the RFC Trace”
    • Start the Listener component
    • Investigate the logs the component offers (in DataStage SAP administration: for BW inspect Source System/Monitor RFC, for IDOC inspect IDOC log)
    • Investigate the RFC trace: there should be a dev_rfc.trc modified at the time when you started the listener. Look at the end of the file for most recent entries.

  BW Load Runtime troubleshooting

  Source System connection

  1. Open the job in InfoSphere DataStage Designer and edit the SAP BW Load stage properties.

  
  
  
  2. In Transfer Structure tab, edit the Source System properties:
  
  
  
  3. Open a SAP user interface session (SAP Logon) and run the transaction sm59 .
  In the TCP/IP Connection category find the the Source System (as RFC Destination) and open it:
  
  4. Perform a Test connection . If it fails, verify whether the Datastage RFC Manager is running and check the DataStage Source System log; if the connection succeeds, then we need to make sure the link between SAP DataStage identified here is unique.
  5. Go to DataStage user interface (where you have the Source System properties open) and change the Program ID (or, alternatively, shut down the listener)
  Click OK.
  6. Go to SAP user interface and repeat a Test connection (it should fail. If it doesn’t, then there is another Source System using the same Program ID – you should identify it and remove it or change the Program ID).
  7. Revert the Program ID in the DataStage user interface, Source system properties . Make sure the connection test succeeds.

  InfoSource and InfoPackage properties

  In SAP user interface, run transaction rsa1; in the InfoSource list search for the component used (in our example 0ABCPROCESS); make sure your Source System (in our example VD.BW35) is listed as DataSource (see below). If it’s not, then try selecting again the InfoSource in DS user interface (manual assignment in SAP user interface is not recommended).

  
  
  
  When the InfoSource is attached to the Source System in DataStage, the following confirmation message displays:
  
  
  Double-click the DataSource and check the properties (ensure the Transfer Method is set to PSA).
  
  
  Go to the InfoPackage (in this example VD.120420) and verify whether it is listed under the DataSource (see step 1); if not, create it again by using the DataStage user interface (by clicking New in the InfoPackage tab).
  
  Note: The InfoPackage name may not display in the stage user interface in certain conditions. See technote 1697910 for more details.

  InfoPackage log in SAP

  In rsa1, when displaying the hierarchy at step 1 above, double-click the InfoPackage to display the properties (Maintain InfoPackage); now click on Monitor to display the log (You can also do a Step-by-step analysis):

  
  

  Process Chain

  If a Process Chain was used in load, you can use transaction rspc to see the logs (see Extract next)

  
  BW Extract OpenHub Runtime troubleshooting
  
  Settings checkup
  
  Perform the following checks:
  
  Source System connection
  Run the steps from generic troubleshooting section
  
  InfoSpoke settings
  In DataStage\stage properties, InfoSpoke tab, click UpdateBW .
  
  
  The following message displays:
  

  Process Chain settings

  1. Open SAP and run transaction rspc ; open the Process Chain used in the job; make sure the InfoSpoke used in the job is included here (the Infospoke is responsible for notifying the DataStage job about the status of the extraction):
  
  2. Double-click InfoSpoke to view the properties (ensure that in Destination tab, the check box Notification to third party tool is selected and the RFC destination and Parameters correspond to the Source system selected in the job).
  
  
  
  Monitoring the InfoSpoke and Process Chain
  
  InfoSpoke
  From transaction rpsc, displaying the InfoSpoke details, you can monitor it.
  
  
  
  Double-click any entry to get more details about that run.
  
  Process Chain
  From transaction rspc you can monitor the logs for a Process Chain (click the Scroll icon):
  
  
  
  You can then drill down in logs for a specific date and identify the component that failed.
  
  
  Resetting a previous request
  If you see the red or yellow requests as shown in the above screenshot, that means those red or yellow requests can possibly block your next request. This is also the case when the job fails with the following error message:
  Previous request status has not been set
  
  In those cases, you will need to change the status of the red or yellow requests to green using the transaction se37 and run the function RSB_API_OHS_REQUEST_SETSTATUS:
  
  
  
  Next, press F8 – Single Test:
  
  
  Enter the RequestId and press F8 – Execute.
  
  When the above method fails (the request status is still not green or getting the same error message), you can try a second method :
  Use transaction se16 in BW user interface. Edit the table RSBREQUID3RD and search for the request either by ID (15629) or by status (it will be either N - New or R - Red). Then delete that line or set the status to G - Green.
  Also, check the table RSBKREQUEST and delete any entries for the specific request ID.
  
  
  Additional OHD request status considerations

  • ODH of type "3rd party" cannot have "Overall Status of Request Option" set to "Automatic", it's always set to "Manual" in SAP GUI (grayed out, so it cannot be changed)
  • The Pack is setting the status by calling the RFC API RSB_API_OHS_REQUEST_SETSTATUS (which should be equivalent with the "Manual" setting)
  • There's a known issue in SAP with Kernel release 7.00 - 7.11 where the status for 3rd party OHD is not set correctly (when the user sets the status manually or via the API); the solution is documented in SAP note 1243168 . It's strongly recommended the users of affected SAP versions install the correction

  Other notes

  Enabling the RFC Trace

  The RFC Trace is comprised of all files generated as logs by the SAP RFC library. The files are named:

  • rfc<NNN>.trc or .log  (e.g. rfc00192_00568.trc , rfc00195_00765.log )
  • dev_rfc.trc

  
  The trace is generated by default in the application folder, but the destination can be changed by setting the environment variable RFC_TRACE_DIR=folder. The trace is enabled by setting the environment variable RFC_TRACE=1. These environment variables can be set at Project level by using DataStage Administrator or in the dsenv file (in which case Datastage needs to be restarted). Also set RFC_NO_COMPRESS so the table content will be visible.

  • RFC_TRACE=1
  • RFC_TRACE_DIR=folder
  • RFC_NO_COMPRESS=1

  Manual cleanup of listener temporary files

  When the following symptoms are met, it may be necessary to do a manual cleanup of the temporary listener files:

  • Job hangs and then stops with "Timeout (...)" error message (or it hangs indefinitely if timeout=0)
  • The following message displays in the job log for a DataStage-initiated extraction: "Starting BW initiated extraction"
  • Any other unexpected behavior.

  
  To manually cleanup the temporary files:

  1. Stop the BW RFC Server listener.
  2. Ensure no processes running. Use the kill command to stop a process:
     • ps -ef|grep rfcsvr
     • kill PID
  3. Delete the contents of the Data folder: <IIS Install>/Server/DSBWConnections/<Connection>/SourceSystems/<SourceSystem>/Data/opt/IBM/InformationServer/Server/DSBWConnections/BI7/SourceSystems/AIX53B/Data
  4. Restart the RFC Server.
  5. Ensure that the listener processes are running:
  • ps -ef|grep rfcsvr