Exporting and importing FileNet P8 object instances

Technote (FAQ)


Question

How do you use FileNet Deployment Manager instead of FileNet Enterprise Manager to export and import data between object stores?

Cause

The topics under "Administering Content Platform Engine" in IBM Knowledge Center do not describe how to export and import FileNet P8 object instances by using the FileNet Deployment Manager tool. Note that use of FileNet Deployment Manager for export manifest creation, as well as the export and import operations, replaces the use of FileNet Enterprise Manager for those operations.

Answer

This article provides an overview of how to use FileNet Deployment Manager for the export and import of Content Platform Engine objects to move them from one environment to another.


Content
Level: Intermediate
Updated: March 15, 2014


Goals for exporting and importing

Objects that are stored in object stores can be exported and imported to move them from one FileNet P8 (source) environment to another (destination) environment. The environments can be in different FileNet P8 domains or within the same FileNet P8 domain. This data movement can be performed with one of several possible goals in mind:

  • Application packaging and deployment where metadata (such as class definitions, property templates, choice lists, and workflow definitions and configuration) and application design-time objects (such as folders, documents, custom objects, event actions, subscriptions, code modules, custom launch and step processors, and search templates) are moved from the environment where the application was developed or tested into the next environment to deploy and configure the application in that destination environment.
      Note that for a workflow system, configuration information can be exported and imported, but not workflow items. For more information about exporting and importing workflow system configuration information, see Workflow system assets in IBM Knowledge Center.
  • Extraction of business data from one system to populate another system for testing purposes.
  • Consolidation of the business data into a single object store as part of a merge.
  • Migration of a subset of an object store from one FileNet P8 domain to another or one object store to another for system rebalancing or in reaction to organizational changes.

For information about application deployment as described in the first bullet in the previous list, see the Deploying applications topic in IBM Knowledge Center. The remainder of this topic focuses the movement of business data stored in a FileNet P8 object store. This movement is often called data migration.

Using FileNet Deployment Manager
This is an overview of how to use FileNet Deployment Manager for the export and import of Content Platform Engine objects to move them from one environment to another. The information in this article is intended to act as a guideline only and the product documentation can be consulted for greater understanding. References to appropriate topics in IBM Knowledge Center are provided to assist in locating relevant information.

Process summary
Although there are a number of options inherent to the process for migrating data from one environment to another by using FileNet Deployment Manager, typically the process is as follows:
  • Define the source and destination environments in FileNet Deployment Manager. See Creating or editing an environment definition and Checklist: Environment definitions for more information.
  • Export the necessary data from the source environment by creating a list in an export manifest and then exporting the object instances to create a deployment data set. For information about creating an export manifest, see Creating or updating an export manifest. For information about creating a deployment data set, see Creating a deploy dataset.
  • Retrieve the object store, security principal, connection point, and other information to create environment half maps. This step is in preparation for later creation of the data maps that are used when the data is converted for acceptance in the destination environment. For information about creating and working with half maps, see Manage an environment definition.
  • Use the half maps for the source and destination environments to construct data maps. The data maps direct FileNet Deployment Manager how to prepare the deployment data set for import into the destination environment. See Manage a source-destination pair.
  • With the data maps, convert the environment-specific attributes in the deployment data set to produce the data that is suitable for import into the destination environment. Note that after the data maps are constructed for a particular source-destination pair, they can be used for multiple conversion operations without repeating the previous steps unless changes occur in the source or destination environments that affect the half maps or data map. For more information, see Converting objects for import.
  • Analyze the converted deployment data set to assess the impact on the destination environment and the potential for the import operation to encounter issues. Assessing the impact on the destination environment might be of little concern when the destination is a test environment that is easily reconfigured. However, when data is imported into a live, deployed environment, performing this analysis is critical. For more information, see Analyzing objects for import.
  • Ensure that a current backup of the destination system is available. For a description of the backup of a FileNet P8 system, see Backup and recovery.
  • Import the converted deployment data set to the destination environment. For more information, see Importing converted objects.

Export include options: data design components
The include options for FileNet Deployment Manager control individual assets or objects. The include options that are specified for an asset are additionally propagated to any related objects that are added to the export data set as a result of the include options settings for the original asset. Thus include options cascade from one object to the next as the export operation finds associated objects to add to the export data set based on the specified include options.

With this behavior and the appropriate include options in the FileNet Deployment Manager export manifest, the data design components can be automatically included with the instance data deployment data set. However, performance is better when the data design components are imported separately from and prior to the instance data. This order of separating and importing data design components before instance data reduces the overall processing time because FileNet Deployment Manager does not have to search for the metadata within the deployment data set and sort the import to get the metadata into the environment first as an implicit step.

Use the following general guidelines for include options when you export data design components:
  • For associated metadata that was created in the source system by an add-on feature, install the same add-on feature into the destination object store instead of moving the metadata by using FileNet Deployment Manager.
  • For class definitions, explicitly include the user-defined property templates and implicitly include the choice lists for the custom property templates. Do not use the Data Design include options for the class definitions themselves. This approach reduces potential side effects that are caused by the export of property templates that were created by add-on features. Typically, the Include modified system classes option is not selected and is used only when necessary.
  • The Event and Lifecycle include options can always be checked for both data design and object instance components because there are events and subscriptions that might be tied to both types of components.
  • The following groups of include options are typically omitted:
      Folders and Contained Objects
      Document-Related
      Social Collaboration

When data design components are exported, the suggested include options selection is as shown in this graphic:


Export include options: object instance components
For the object instance data, many combinations of include options are possible. Which combination will provide the best export depends on the instance data and its relationships to other data in the object store. However, as a starting point:
  • When you use FileNet Deployment Manager to gather business data, specify include options that narrowly target the objects to export. This narrow focus is intended to avoid implicit inclusion of unwanted objects that can be caused by propagation of the include options.
  • In the Include Options window in FileNet Deployment Manager, click None to clear all the include options that are selected. If you clear the include options, only the asset that is listed in the export manifest will be exported and placed into the deployment data set.

In certain cases, this narrow focus might need to be broadened:
  • Moving all content in a folder
    If all content in a folder is to be moved, export the folder and implicitly include the contents and subfolders with the relationships. In this case, keep the include options that are selected by default in the Folders and Contained Objects section.
    For more control over the export, add the parent folder explicitly to the export manifest instead of by using the Include parent folders option. Doing so avoids the export of an application-generated folder that is best moved with the application design-time objects and not the business data. Doing so also avoids inclusion of unexpected folders that might not relate to the desired export if an object is filed into more than one folder and thus has more than one parent folder.
  • Associated objects that an asset depends on
    Explicitly specify in the manifest other associated business data an object instance depends on. For example, select the Include relationship to containing folders option and maintain relationships to the parent folder. Without this include option, a document or custom object will appear as unfiled after the import.
  • Documents that are revised over time
    Some documents can be revised over time, so multiple versions might exist. Use the Document-Related include option Include all document versions to ensure that subsequent exports with the same export manifest extract the latest version. If you do not want to export all versions, but you want to ensure the most current version is exported, you must add the reference to the current version into the export manifest. Before reusing the manifest, edit it to delete the older document reference and then add the current one to the manifest.
  • Compound documents, annotations, and thumbnails
    Use the other document-related include options for compound documents, annotations, and thumbnails on an as-needed basis.
  • The Event and Lifecycle and General include options can typically be left with their default values.

Import Options
  • When the import step is executed, it is important that the user that is specified in the FileNet Deployment Manager connection to the destination environment has sufficient privileges on the destination object store to perform all necessary actions during the import. It is best to execute the import step when the connection information for the destination environment specifies as the user an object store administrator.
  • If the import option Use Original Create/Update Timestamps and Users is chosen, then modifications to Content Platform Engine system properties might occur. To modify system properties, the FileNet Deployment Manager import user must additionally have the Modify certain system properties privilege on the destination object store or the import operation returns errors. This object store privilege is not assigned by default.
  • When the data is imported through FileNet Deployment Manager, some unavoidable data validation exceptions can occur. One example occurs because Content Platform Engine allows you to remove a choice list element even if existing instance data contains that choice list element value. Subsequent import of the instance data with the deleted choice list value results in an exception because the choice list definition no longer contains that element.

Handling data caused import errors
Review any data errors to determine the root cause and remediation steps. Two examples of how a data error on import might be managed are as follows:
  • For the choice list example given previously where an object's property contains a now deleted choice list value, a possible workaround is to import the choice list definition first and then temporarily add the previously deleted choice list element before execution of the FileNet Deployment Manager import for the object instances. The deleted value can then be removed from the choice list definition at a later point to ensure that the choice list definition on the destination matches with the choice list definition on the source.
  • As another example, the import script mechanism can be used. For example, an import pre-save script runs on each object in the converted deployment data set before the object is saved in the destination object store. A script can be written to detect when an object has a data integrity issue and modify the object to correct the issue before it is imported. In the choice list example, the object instances with the invalid choice list value can be modified so that a valid choice list value is present on import. The script must have logic to determine what a valid value is.

Consideration of possible data integrity issues is a part of the planning and test phases for the data migration process. Including a test phase with realistic data is invaluable to ensuring a smooth import process by allowing for the development and testing of remediation steps.

Using the logs
The FileNet Deployment Manager logs contain diagnostic and summary information. Always consult the logs after any FileNet Deployment Manager operation. The log files that are used for FileNet Deployment Manager are created by using Apache log4j and are, by default, in the current working directory that is used to start FileNet Deployment Manager. For example, if FileNet Deployment Manager is started from the Windows Start menu, the log file is under the installation directory for the Content Platform Engine tools installation, by default:
C:\Program Files (x86)\IBM\FileNet\ContentEngine\tools\deploy\deployment.log

However, at times, it might be necessary to enable detailed logging of FileNet Deployment Manager operations to gain insight into correctable issues such as data integrity violations. To enable debug levels of FileNet Deployment Manager and Content Engine Java API tracing, modify the log4j.properties file found under the installation directory for the Content Platform Engine tools installation, by default:
C:\Program Files (x86)\IBM\FileNet\ContentEngine\tools\deploy\log4j.properties

For information about enabling Java logging and other important Java runtime options to control FileNet Deployment Manager runtime behavior, see Specify Java runtime options.
The following example shows a FileNet Deployment Manager log4j.properties file with debug enabled for FileNet Deployment Manager and the Content Engine Java API.

################################################################################
# Appenders
################################################################################
#=== ConsoleAppender with simplified layout to improve legibility on the screen
log4j.appender.ConsoleAppender=org.apache.log4j.ConsoleAppender
log4j.appender.ConsoleAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.ConsoleAppender.layout.ConversionPattern=%5p [%t] %c{2} - %m%n
log4j.appender.ConsoleAppender.encoding=UTF-8
#=== FileAppender, uses a more complete layout for better problem determination via the logfile
log4j.appender.FileAppender=org.apache.log4j.FileAppender
log4j.appender.FileAppender.File=./deployment.log
log4j.appender.FileAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.FileAppender.layout.ConversionPattern=%d %5p [%t] %c %x - %m%n
log4j.appender.FileAppender.encoding=UTF-8
################################################################################
# Set Root logger level and appenders
################################################################################
log4j.rootLogger=ERROR, ConsoleAppender, FileAppender
################################################################################
# Loggers
# Set log level to either one of OFF/DEBUG/INFO/WARN/ERROR/FATAL/ALL
# Child logger's value overwrites parent logger's one.
# If a logger is not specified, it inherits its value from its parent logger.
################################################################################
log4j.logger.com.filenet.deployment = debug, FileAppender
# the line above enables FDM tracing
log4j.logger.filenet_error.api = debug, FileAppender
log4j.logger.filenet_error.api.com.filenet.apiimpl.imex.LoggingStream = debug, FileAppender
log4j.logger.filenet_tracing.api = debug, FileAppender
# the line above enables the Content Engine Java API tracing

Planning concepts
Connected or disconnected environments
Object instances can be moved between connected or disconnected environments.
    Connected: One FileNet Deployment Manager instance can access the source and destination at the same time and performs both export and import.
    Disconnected: Separate instances of FileNet Deployment Manager perform the export and then the import. An example is where environments are separated by firewalls or the object instances are being moved from an external source such as might be the case when test data is being provided to a separate development organization.

The typical process must be adjusted depending on whether the environments are connected or disconnected.

Process overview when the environments are connected:


If the environments are connected, only one deployment tree is needed. If the environments are disconnected and more than one instance of FileNet Deployment Manager is used, then more than one deployment tree must be created. To facilitate a disconnected deployment, FileNet Deployment Manager can create a compressed Deploy Package that is easier to move between systems. This package contains the information about the source environment that is needed to convert the assets as well as the selected deployment data set. If the systems are disconnected, the source environment information is typically extracted from the deployment package into the deployment tree for the FileNet Deployment Manager instance that is performing the import on the destination. For more information about the creation and management of a deployment package, see Managing deploy packages.

Process overview when the environments are disconnected:


Identify object instances and dependencies
The IBM Knowledge Center topic FileNet P8 assets contains important information such as object dependencies and specific export and import processing that can be studied as a part of planning any data migration. The following general concepts might be useful when you identify what to export.

FileNet Deployment Manager uses a browse approach to navigate to the object instances in the source object store. To build the export manifest, you must know the location within the source environment of the object instances that you want to export. For filed objects, typically the parent folder is selected and the export include options are used to export all contained objects and subfolders. For unfiled objects, FileNet Deployment Manager offers a virtual navigation path that uses the Unfiled Documents and Uncontained Objects nodes as a part of its Add Assets window.

Additional information that can be collected about the FileNet P8 objects to move includes:

Object dependencies:
    A dependency exists between two objects when one object references the other. When such objects are imported, the object that is referenced must exist in the object store before the object that references that object is imported.

    For dependent relationships defined by a FileNet P8 application, FileNet Deployment Manager understands the dependencies and ensures that the objects are imported in the correct sequence. However, if object dependencies are known only to a third-party product, it might be necessary to create multiple deployment data sets to ensure that the objects will be sequenced correctly by managing the order in which they are imported.

Folder references:
    Path-based or GUID-based: Some folder references use a path-based rather than GUID-based scheme. In general, path-based references are used for object-valued properties that are single cardinality and have a required class that is Folder or a subclass of Folder. Object-valued properties that do not have a required class or have a non-Folder required class are referenced by GUID, even if they reference a folder instance.

      For example, path-based references are used to reference the folder's parent folder, the security folder of an object, the containing folder in a Referential Containment Relationship, or a folder in a system or user-defined property. Migration of these assets must be preceded by the creation or import of the correct folder hierarchy so that the complete folder path is retained.

      GUID-based folder references are often used by other components whose metadata directly reference a folder such as an entry template. Migration of these assets might include the referenced folder object instance if the folder was not already migrated to the destination to ensure stable GUIDs.

Source and destination environment-specific information:
    To assist in the creation of the data maps, this information can be useful:
    • The object store in the destination environment that corresponds to a specific object store in the source environment.
    • If a workflow system is referenced, the destination environment connection point that will be used.
    • The user or group in the destination environment that will be assigned the access privileges of a specific principal in the source environment.
    • The service data in the destination environment that will replace the service data values from the source environment.

Ensure source and destination environment compatibility
Inherent to preparing to migrate any FileNet P8-based application between environments is ensuring that the source and destination object stores are compatible. Ensuring compatibility includes the following activities:
  • Ensure that add-on features that define classes or properties in the destination object store are the same as in the source object store.
      Consider add-ons that modify the metadata for the default Document class definition. Any document stored in the Document class or a subclass of the Document class will be affected by that modification even if the document is not used by the product that installed the add-on. For example, if the object store in the source environment is also configured to be an IBM Enterprise Records records object store, a document that is not intended to be managed as a record will still have the additional IBM Enterprise Records properties when it is exported. If that document is imported into a destination environment that does not have the add-on for a records object store applied, an import error will occur.
  • Consider the release levels of the software and deployment tools that are used. FileNet Deployment Manager must be at the same release level as the server it is connecting to. Also, if the IBM FileNet products in the source environment are not at the same release level as the products in the destination environment, import errors can occur if the difference causes metadata or data incompatibilities between the two environments.
  • Verify all content to be deployed is checked in before starting the import process.
  • Plan the corresponding administrative groups in the source and destination environments so that appropriate data maps can be created.

For more information, see the IBM Knowledge Center topic About deploying to the Content Platform Engine server.

Preparing the destination environment can be an iterative process if complex dependencies between assets are present. The FileNet Deployment Manager change impact analysis, or analyze, operation reports on potential import errors in the destination environment and the impact of the changes on the destination environment.

The information in the change impact report must be examined to identify add-on features or configurations that are missing from the destination environment. The report also helps to point out missing items or unexpected extra items that are present in the converted deployment data set. These issues must be corrected before the import can proceed. Corrective measures might include the following actions:
  • Installing and configuring third-party components that are a part of the application that originally generated the object instances being migrated.
  • Following the suggested practice to export and import the data design components (such as class definitions, property templates, and choice lists) first before migrating the instance data.
  • Re-exporting the assets from the source with a more focused set of include options to ensure that the deployment data set contains only the objects you want exported to eliminate the unexpected extra items.

If large amounts of data will be imported and time to perform the analysis for all the data is not available, the analysis can be performed on a representative subset of the data. If this approach is taken, ensure that a backup of the system is available before starting the import in case errors do occur and the system must be reset to retry the import after the issues are addressed. If the import is a part of the migration of production data, follow the entire migration process including the analysis by using a test environment before performing the actual migration of production data. Doing so provides an opportunity to gain an understanding of potential errors and to develop procedures to correct them. For more information, see About change impact analysis.

Perform one-time configuration and setup tasks
When business data is migrated into a system for the first time, steps preceding the export and import might be required to configure a new destination environment to receive the business data. These additional steps will be referred to collectively as system configuration. A system configuration can be roughly divided as follows:
  • Containers: FileNet P8 domain, object store, Content Search Services server, workflow system, isolated region, and other containers.
  • FileNet configuration constructs: storage areas, index areas, isolated region configuration, definitions of connections (database, Rendition Engine, isolated region), fixed content device, content federation services configuration, Content Search Services configuration, and other constructs.

The types of system configuration tasks and their procedures are specific to the objects being moved and the destination environment. For details based on the types objects being moved and information about object types requiring special consideration, see FileNet P8 Assets.

Conclusion
FileNet Deployment Manager is a powerful tool that can be used effectively and efficiently to move business data from one FileNet P8 environment to another. Planning and testing are key elements to a successful migration process. The final step in the process is verification that the objects are imported as expected. As a part of the planning and testing, procedures to verify the destination environment after the imports are complete are important to develop and document.


Related information
IBM Knowledge Center
IBM FileNet P8 Version 5.2 Information Center
IBM Content Foundation Version 5.2 Information Center
IBM Redbooks IBM FileNet Content Manager Implementation Best Practices and Recommendations
MustGather: FileNet Content Engine FileNet Deployment Manager (FDM)


Cross reference information
Segment Product Component Platform Version Edition
Enterprise Content Management FileNet Content Manager AIX, Linux, Windows 5.0, 5.1.0, 5.2.0 All Editions

Rate this page:

(0 users)Average rating

Document information


More support for:

Content Foundation
Content Platform Engine

Software version:

5.2

Operating system(s):

AIX, Linux, Windows

Software edition:

All Editions

Reference #:

1667406

Modified date:

2014-04-01

Translate my page

Machine Translation

Content navigation