Image Services: Tuning Best Practices

Technote (troubleshooting)


Problem(Abstract)

This document describes common configuration settings that should be investigated when a performance issue is suspected within FileNet Image Services.

Resolving the problem

This document describes the most common items to check when reviewing the current configuration.

    • Performance Reports
      The FileNet Image Services performance reports can be created at any time after the software has been started by running the command perf_report -a. The performance reports are in the /fnsw/local/logs/perf directory on UNIX servers and the DRIVE:\fnsw_loc\logs\perf directory on Windows servers.

      By default, the perf_mon process that gathers performance data, stores performance data every 15 minutes. If a performance issue is suspected, IBM suggests changing the interval from 15 minutes to 5 minutes. The perf_mon.script file can be located in one or two locations. Both files must be modified if they exist.

      UNIX: /fnsw/lib/perf/perf_mon.script
      /fnsw/local/sd/perf_mon.script

      Windows: DRIVE:\fnsw\lib\perf_mon.script
      DRIVE:\fnsw_loc\sd\perf_mon.script

      Change:
      From:
      schedule 0 0:00:00 2:00:00
      schedule 0 6:00:00 0:15:00
      schedule 0 19:00:00 2:00:00
      To:
      schedule 0 0:00:00 2:00:00
      schedule 0 6:00:00 0:05:00
      schedule 0 19:00:00 2:00:00
      After editing the perf_mon.script file or files, restart FileNet Image Services to put the change into effect.

    • Verify Shared Memory Segment Size

      The preferred shared memory segment size is different for each hardware platform. It is important to configure the correct shared memory segment size to avoid running out of the resources that can cause critical errors. See the IBM Tech Note below for detailed information about troubleshooting shared memory issues and setting the correct shared memory segment size.


    • Set Document & Directory Buffers to Maximum Values
      Generally, the number and sizes of the document and directory buffers should be set to the maximum size and number in the fn_edit -> Performance Tuning -> Server memory tab.

      The maximum sizes and counts are:

      Document Buffer Count: 256 Directory Buffer Count: 256
      Document Buffer Size: 1024 Directory Buffer Size: 256

      Document Buffer Count - Each process that accesses cache should have a document buffer associated with it. These processes include all of the dtp processes, the number of CSMs processes configured as well as the committal processes (bes_commit, fbc_commit. rmt_commit, etc). If a FileNet Image Services server does not have enough document buffers configured, overall system performance is affected because the processes that access cache must wait for a buffer to become available..

      Document Buffer Size - A document buffer is used to transfer an object (document page) to and from cache and an optical/MSAR surface. By setting the document buffer size to the maximum size, it requires fewer transfers, which translates into better performance because less time is required for the transfer to complete.

      To determine the number of processes currently configured that use document buffers, obtain the total of the following three items:

      • The total number of CSMs, DOCs and PSMs request handler processes configured in the /fnsw/etc/serverConfig file. If the /fnsw/etc/serverConfig.custom file exists, use the total number from this file instead.
      • The number of optical and MSAR drives configured in the Storage Library tab in fn_edit. All MSAR libraries have 12 drives each. The number of optical drives can vary depending upon the model of the optical library.
      • The total number of FileNet Image Services committal processes is in the /fnsw/local/sd/as_conf.g file. If the number of processes for a specific committal process is not shown, use 1 which is the default.
          processes {
          notify ds_notify 2
          scheduler dsched
          dtp dtp
          dtp_tran dtp_tran 1
          rmt_commit rmt_commit 2
          fbc_commit fbc_commit
          del_commit del_commit
          osi_migrate osi_migrate
          }
      Directory Buffers Count and Size - Directory buffers are used to break up a Fast Batch Object, and they are also used by the Integral SDS_worker processes.

      The dbp -s command can be run to monitor the buffer configuration when the FileNet Image Services software is running at peak load or when there is a performance issue. See the FileNet Image Services System Tools manual got information about interpreting the dbp output.

    • Turn off MKF Verify Disk Writes
      This feature is configured in the fn_edit -> MKF databases tab. This feature should only be turned on if the System Administrator suspects that there is some type of hardware or network error on the device where the MKF databases reside. When Verify Disk Writes is turned on, the FileNet Image Services software reads back and verifies everything it Writes. This verification process causes two transactions to occur for every write operation. Verify Disk Writes slows down performance. By turning it off, only the write transaction occurs and this improves the server performance.

    • Separate MKF databases and cache to different disk drives
      Each of the MKF databases (Permanent, Security and Transient) and Cache should be on different physical disk drives to prevent disk contention with the disk read write heads on very active systems. Having cache and the transient database on the same physical disk drive would slow down performance when objects are moving in and out of cache. Both the transient database and cache are updated at the same time as documents are processed. The same thing holds true between the transient and permanent databases.


    • Separate MKF database RL partitions from their databases

      Each of the three MKF databases have one or more database datasets (permanent_db0. permanent_db1, etc) and redo logs (permanent_rl0, permanent_rl1). The database datasets are where data is stored and the redo log is used to maintain log files as sequential records of database changes. These logs are flushed as information is written to the actual database.

    • Set the MKF buffers to get as close as possible to a 100% cache hit ratio

      The MKF buffers can be increased in the fn_edit -> Performance Tuning -> Server Memory tab. By increasing the size of the buffers, more of the database is held in memory and avoids having to read out of the database on the physical disk drive. The cache hit ratio for each of the databases can be obtained by creating the performance reports and looking at the MKF I/O reports.
          cmb1.permdb_io.Apr15.txt
          cmb1.secdb_io.Apr15.txt
          cmb1.transdb_io.Apr15.txt

      Tuning the MKF buffers should be done gradually. MKF buffers are stored in shared memory. If the buffers are made too large, performance problems could result.

    • Tune cache for best performance

      In the fn_edit -> System Application Services -> Cache tab, the System Administrator can allocate the minimum percentage of cache to devote to the main types of cache.

      The minimum percentages for all four caches should add up to 100%.

      Allocate most of the cache percentage to page cache so more documents are kept ageable in cache before they are aged out by the CSM_daemon process. This improves performance by preventing the documents in cache from having to be retrieved from an optical surface, MSAR surface, or Integral SDS device.

      The current cache hit ratio can be obtained by looking at the performance reports (Client Page Request Report)

    • Configure the locked threshold for BES

      Configure the locked threshold for BES cache to leave sufficient free space to hold the largest batch that could be created plus 10%.

      For example, for an environment where:
        1. The Minimum Allocation for BES cache is 20% and the total cache size is 1 GB
          or 200 MB
        2. The largest batch size is five 1000 page documents
        3. The size of each page 50 KB
        4. The largest batch size would be 25 MB (5 x 1000 x 50 KB)
        5. The free space required would be 27.5 MB (25 MB + 10%)
      Set the Locked Threshold % in fn_edit for BES cache to at least 86% since 27.5 MB is almost 14% of the Minimum Allocation of BES cache (200 MB).

    • Turn on Fast Batch Committal and Fast Batch Breakup

      If a server uses Fast Batch Committal, Fast Batch Breakup should be turned on in the fn_edit -> System Application Services -> Other tab. Typically, after a Fast Batch Object has been migrated, the documents contained in Fast Batch Object do not reside in cache. By turning on Fast Batch Breakup, the documents remain in page cache after they are migrated to the storage device. Most users access new documents immediately. By allowing the Fast Batch documents to remain in cache, they do not have to be migrated back into cache if they are immediately requested by a user.

      Applications that use fast Batch Committal are COLD, HPII and Capture. A custom application can also be written to use Fast batch Committal.

      The current cache hit ratio can be obtained by looking at the performance reports (Client Page Request Report).

    • Set the number of Integral SDS_worker processes

      If Integral SDS performance issues are suspected, there are several things that can be used to investigate problems.

      1. DOC_tool - All ISDS statistics are kept in memory. When the Image Services software is recycled, all of the information since the software was started is lost. There are no performance reports that track ISDS committals or retrievals. DOC_tool has an “SDS” option that can be used to display the current statistics that are kept in memory. This command should be used infrequently in a production environment because it causes all of the SDS_worker processes to pause while the performance information is collected and displayed.

      Here is an example of the DOC_tool SDS output for an SDS Unit. It shows a high AVG requests queue wait time which indicates the number of SDS_worker processes should be increased.

      The AVG requests queue wait time shows how long the requests are in the queue before they are processed by SDS_worker. Requests are put in the queue by SDS migration background job.

        DOC_tool
        DOC_tool> SDS
        Summary info, Detailed, Worker info, All info, Find object, or list?
        ('s', 'd', 'w', 'a', 'f', 'l') : d

        The current time is Fri Feb 5 10:25:31 2010

        SDS info: ALL option
        All SDS units mode (y/n) [y]: : n
        SDS unit ID: 2
        ****** SDS unit = cen_kirin (2)
        SYSTEM state = SYSTEM ENABLED (0x0)
        USER state = USER ENABLED (0x0)
        Worker = 'SDS_worker' Number Instances = 4
        info = 'Centera2.usca.ibm.com?/fnsw/local/sd/1/QAImport.pri'
        SDS priority = high
        DEBUG Setting = MAX
        dynamic repository lib = 'SDSw_centera'
        retention default offset (1 days)
        SDS content delete setting=YES
        SDS supports: EBR=YES HOLDS=YES Retention Extension=YES

        Total Accumulated counters from all workers(4):
        ** Configured workers =4 active workers=2
        TOTAL WORKER COUNTERS (sds_id=2):
        Read Requests processed: 0
        Write Requests processed: 20
        Copy Requests processed: 0
        Errors: 0
        Requests processed = 20
        Successful requests processed = 20
        Errors = 0
        AVERAGE ACCUMULATED ELAPSE TIMES:
        Up time: 274.098067 secs/workers (4.568301 mins)
        Idle time: 263.780673 secs/workers (4.396345 mins) (96.24%)
        Total processing time: 7.472040 secs/workers
        (0.373602 secs/reqs)
        (0.373602 secs/image page)
        (0.013004 secs/KB)
        AVG requests queue wait time: 3.466658 secs/reqs

        *****Total READ REQUEST PERFORMANCE (sds_id=2)
        Total retrieval requests = 0
        Images retrieved from SDS = 0
        Data retrieved = 0.000000MB
        Number of read requests where the whole blob fits
        into the internal image_buffer (1024K): 0
        Number of read requests where the whole blob does not fits
        into the internal image_buffer (1024K): 0
        Cache hits: 0
        Number of redirection: 0
        Number of redirection errors: 0

        Total Time to process read requests: 0.000000 secs (0.000000 mins)

        *****Total WRITE/COPY REQUEST PERFORMANCE (sds_id=2)
        Total write requests = 20
        Total copy requests = 0
        Documents written = 20 (FBC=0, MSAR reads=0, Cache=20, Copy=0)
        Images written = 20
        Data written = 0.561123MB
        AVG Image Size = 28.729492K
        Cache hits in copy: 0
        Total Time to process write requests: 14.943911 secs (0.249065 mins)
        Total Time to process copy requests: 0.000000 secs (0.000000 mins)
        Time in SDS device create and write object: 0.423189 secs (0.007053 mins)
        (0.021159 secs/reqs)
        (0.021159 secs/image page)
        (0.000737 secs/KB)
        Time in SDS device write only: 0.053464 secs (0.000891 mins)
        (0.002673 secs/reqs)
        (0.002673 secs/image page)
        (0.000093 secs/KB)
        Time in cache(CSM) to process write/copy: 0.025077 secs (0.000418 mins)
        (0.001254 secs/reqs)
        (0.001254 secs/page)
        (0.000044 secs/KB)
        Time in MSAR read to process write/copy: 0.000000 secs (0.000000 mins)

      2. Number of SDS_worker processes – The default number of SDS_worker processes is 3 when the SDS unit is configured in fn_edit. Each of the three default worker processes take on a different function.

      SDS_worker # 1 - The first worker process performs only reads.
      SDS_worker # 2 - The second worker process performs only copies.
      SDS_worker # 3 - The third worker process performs only copies.

      If the number of worker processes is not increased from the default of three, performance can be degraded. Only one process performs all of the ISDS writes and one process performs all of the read operations.

      When more than three SDS_worker processes are created, SDS_worker #4 and higher, perform all three operations (read, copy and write). The maximum number of SDS_worker processes that can be configured is 99 for each SDS unit. Setting the number of SDS_worker processes too high can also have an adverse effect. When excessive SDS_worker processes are configured, they can use resources and may never be accessed.

      The number of initial SDS_worker processes should be set to around 20 and then monitored using the DOC_tool SDS option. The SDS_worker processes can be monitored to see if they are active or idle (too many configured).

      The DOC_tool SDS statistic, AVG requests queue wait time, can be monitored for the SDS unit to determine if the number of SDS_worker processes needs to be increased for the SDS Unit.

    • Configure the TCP and Ephemeral Ports Network Settings

      It is important to set the TCP and ephemeral port settings to avoid ports being in a time wait state or ports being blocked. When the ephemeral port settings are not correct, 15,16,17 errors might be written on the error log. This type of error can cause performance and network connection issues. The preferred settings for each hardware platform are provided below.


      AIX
        Preferred settings:
          tcp_keepidle 80
          tcp_keepintvl 20
          tcp_ephemeral_high 65535
          tcp_ephemeral_low 42767
          udp_ephemeral_high 65535
          udp_ephemeral_low 42767
        Verify current settings:
          The no -a command can be used to verify the current settings.

        Resolution:
          /usr/sbin/ no -p -o tcp_keepidle=80
          /usr/sbin/ no -p -o tcp_keepintvl=20
          /usr/sbin/ no -p -o tcp_ephemeral_high=65535
          /usr/sbin/ no -p -o tcp_ephemeral_low=42767
          /usr/sbin/ no -p -o udp_ephemeral_high=65535
          /usr/sbin/ no -p -o udp_ephemeral_low=42767

      HP-UX & HP Integrity
        Preferred settings:
          udp_smallest_anon_port 42767
          udp udp_largest_anon_port 65535
          tcp tcp_smallest_anon_port 42767
          tcp tcp_largest_anon_port 65535
          tcp tcp_time_wait_interval 30000

        Verify current settings:
          ndd -get /dev/udp udp_smallest_anon_port
          ndd -get /dev/udp udp_largest_anon_port
          ndd -get /dev/tcp tcp_smallest_anon_port
          ndd -get /dev/tcp tcp_largest_anon_port
          ndd -get /dev/tcp tcp_time_wait_interval

        Resolution:
          ndd -set /dev/udp udp_smallest_anon_port 42767
          ndd -set /dev/udp udp_largest_anon_port 65535
          ndd -set /dev/tcp tcp_smallest_anon_port 42767
          ndd -set /dev/tcp tcp_largest_anon_port 65535
          ndd -set /dev/tcp tcp_time_wait_interval 30000


      Solaris
        Preferred settings:
          udp_smallest_anon_port 42767
          udp udp_largest_anon_port 65535
          tcp tcp_smallest_anon_port 42767
          tcp tcp_largest_anon_port 65535
          tcp tcp_close_wait_interval 30000 (for Solaris 2.x only)
          tcp tcp_time_wait_interval 30000 (for Solaris 8 and above)

        Verify settings:
          ndd -get /dev/udp udp_smallest_anon_port
          ndd -get /dev/udp udp_largest_anon_port
          ndd -get /dev/tcp tcp_smallest_anon_port
          ndd -get /dev/tcp tcp_largest_anon_port
          ndd -get /dev/tcp tcp_close_wait_interval (for Solaris 2.x only)
          ndd -det /dev/tcp tcp_time_wait_interval (for Solaris 8 and above)

        Resolution:
          Make a backup copy of the /etc/rc2.d/S69inet file before you modify it. If the file does not exist, create it.
          As root user, make sure you have write permission on the file by entering:
            chmod 754 /etc/rc2.d/S69
          Use your preferred text editor (such as vi) to modify the /etc/rc2.d/S69inet file.
          Add the following lines somewhere near the end of the file:
            ndd -set /dev/udp udp_smallest_anon_port 42767
            ndd -set /dev/udp udp_largest_anon_port 65535
            ndd -set /dev/tcp tcp_smallest_anon_port 42767
            ndd -set /dev/tcp tcp_largest_anon_port 65535
            ndd -set /dev/tcp tcp_close_wait_interval 30000 (for Solaris 2.x only)
            ndd -set /dev/tcp tcp_time_wait_interval 30000 ndd -set /dev/udp
          Save your change and exit from the file.
          Reboot the server or servers.

      Windows
        Use the Registry Editor (regedt32.exe) to make the modifications.

        MaxUserPort
          Description: Determines the highest port number TCP can assign when an application requests an available user port from the system. Typically, ephemeral ports (those ports that are used briefly) are allocated to port numbers 1024 through 5000.

          Note: Windows does not add this entry to the registry. You can add it by editing the registry or by using a program that edits the registry.
          Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
          Data Type: REG_DWORD
          Range: 5,000-65,534 (port number)
          Default Value: 5000
          Recommended value: 65534 (65534 DEC)

        TcpMaxConnectTransmissions
          Description: Determines how many times TCP retransmits an unanswered request for a new connection. TCP retransmits new connection requests until they are answered or until this value expires.
          Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
          Data Type: REG_DWORD
          Recommended value: 5 (5 DEC)

        TcpMaxConnectRetransmissions
          Description: Determines how many times TCP retransmits an unanswered request for a new connection. TCP retransmits new connection requests until they are answered or until this value expires.
          Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
          Data Type: REG_DWORD
          Range: 0–255 (retransmission attempts)
          Default Value: 2
          Recommended value: 5 (5 DEC)

        TcpTimedWaitDelay
          Description: Determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. This interval between closure and release is known as the TIME_WAIT state or twice the maximum segment lifetime (2MSL) state. During this time, reopening the connection to the client and server costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster and provide more resources for new connections. Adjust this parameter if the running application requires rapid release, the creation of new connections, or an adjustment because of a low throughput caused by multiple connections in the TIME_WAIT state.

          Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

          Data Type: REG_DWORD

          Default Value: 0xF0 (240 DEC)

          Recommended value: 0x1F (30 DEC)

        FN_COR_QLEN
          Description: The 15,16,17 error indicates a process is not able to connect to the COR_Listen process due to the unavailability of COR queue space.

          To resolve this issue an environmental variable named FN_COR_QLEN should be created.

          The default COR queue length is 5. The environmental variable must be set for the user that starts the Image Services software and named FN_COR_LEN. The value set for the environmental variable should initially be between 20 to 25.

Rate this page:

(0 users)Average rating

Document information


More support for:

FileNet Image Services
Image Services

Software version:

4.1.1, 4.1.2, 4.2

Operating system(s):

AIX, HP Itanium, HP-UX, Solaris, Windows

Software edition:

All Editions

Reference #:

1634425

Modified date:

2013-11-01

Translate my page

Machine Translation

Content navigation