IBM Support

New features and changes in InfoSphere Information Server, Version 11.5

Question & Answer


Question

What new functionality has been added with rollup patches or fix packs since the release of InfoSphere Information Server, Version 11.5?

Answer




    Latest rollup information:


    General updates
    Governance updates
    Data Integration updates

    General updates


    New in 11.5.0.2
    • Support for Oracle 12c with Multitenant architecture (Container Databases (CDB) and Pluggable Databases (PDB)).
    • Ability of installer to access the Oracle databases by using ServiceName.
    • Support for the Google Chrome web browser.

    Governance updates


    The following components have new Governance features:
    InfoSphere Information Analyzer
    InfoSphere Information Analyzer Thin Client
    InfoSphere Information Governance Catalog
    InfoSphere ISALite
    InfoSphere QualityStage
    Managing metadata
    Managing exceptions and events
    Deprecated features from the Information Analyzer Workbench



    InfoSphere Information Analyzer


    New in 11.5.0.2 Service Pack 3
    • You can use different credentials for running analysis jobs than the ones you use to import metadata in InfoSphere Metadata Asset Manager. You can set the credentials for a specific project or at the Global level which is applicable for all projects.
    New in 11.5.0.2
    • Ability to run analysis on data sources imported by using the JDBC connector and the Hive connector. The supported analysis types are Column Analysis, Key Analysis, Cross Domain Analysis, and Data Rules Analysis.
    • Improve performance when running an analysis by using the new -batchAnalysis option with the IAAdmin command. Use this new option when running an analysis on a large number of data sets to prevent your system from being overloaded with jobs.
    New in 11.5.0.1

    InfoSphere Information Analyzer Thin Client


    The thin client, a browser-based interface for data analysts, has the following new or changed features:
    New in 11.5.0.2 Service Pack 2
    • You can export the results of a data set analysis to a CSV file.
    • By using a new IAAdmin getDataClasses command, you can export your data classes to an XML file.
    • When you run analysis, you can choose to run only column analysis, or only data quality analysis, or both.
    • Apache Parquet files are supported as the source for column analysis, data quality analysis and data rules.
    • You can run overlap analysis from Information Analyzer Thin Client, which is known as cross-domain analysis in Information Analyzer workbench. This analysis searches for columns that have the same values in a table or in multiple tables.
    New in 11.5.0.2 Service Pack 1
    • You can export entity relationship diagrams (ERD) to a PDF format and print them.
    • You can set Workspace settings on UI. Four tabs are added: Column Analysis, Data Quality, Sampling and Engine.
    • You can use global variables in your data rules.
    • You can specify variables, columns, statistics, attributes, and expressions when you define how to display the output of your data and quality rules.
    New in 11.5.0.2
    • Ability to run analysis on data sources imported by using JDBC connector and Hive connector. The supported analysis types are Data Quality Analysis, Relationship Analysis, and Data Rules Analysis.
    • Ability to view virtual tables and columns in Information Analyzer Thin Client. You can use them to run analysis and display the analysis results.
    • Ability to filter catalog and workspace data sets by Database and Schema names.
    • New data classes added. See the Data classification topic.
    New in Governance rollup 7
    • Run relationship analysis to automatically identify relationships between two or more data sets in a given workspace. Visualize and review these relationships graphically through an entity-relationship diagram. Understand the strength of each relationship based on displayed statistics and confidence with the ability to mark relationships as 'selected', 'rejected' or 'candidate'.
    • Ability to edit existing data rules from a hover menu next to each rule. This saves time and reduces errors when rules need modification.
    • New supported data sources. Netezza and Teradata are now supported data sources. Import metadata by using InfoSphere Metadata Asset Manager and then add data from these sources to your workspaces.
    • A Project Administrator can set up workspace-level sampling settings used by all subsequent quality analysis (column analysis) and data rule runs. Using sampling functionality can dramatically improve the speed of the analysis.
    • Improved error messages for easier problem determination. In some cases, even including the ability to directly/easily view a more detailed message log.
    • Also refer to the 'Deprecated Features' section below.
    New in Governance rollup 6
    • Ability to perform sampling during analysis.  Set sampling settings at the Workspace level in the thin client by using a command line interface command.
    • Improved user experience.  There are redesigned views of charts and grids, as well as redesigned views of activities, governance, classification, data quality, and data rules on the Workspace Overview screen.
    • Support for deleting data quality rules and data rules in the thin client.
    • New features for InfoSphere Information Analyzer data rules and quality rules, including:

    • o Support for creating data rules with output table definitions
      o Drag and drop capabilities for rule binding
      o Better search for columns and rule definitions
      o Updated rule binding layout 
      o Support for literals and global variables for rule binding
      o Integrated Information Governance Catalog service for rule suggestions based on data
      class and term  relationships that are defined in Information Governance Catalog
    New in Governance rollup 4
    • Bind columns in a data set to multi-variable or single variable rule definitions to create quality rules. Quality rules quickly and easily run checks on specific aspects of your data. With quality rules, you no longer need to create and name an individual rule for each condition you want to validate. You can simply bind rule definitions to columns with a few clicks, and then run the new quality rules as part of the data quality analysis.
    • Analyze data sets, work with analysis results, and view data quality scores for data sets from S3 data connections created in InfoSphere Metadata Asset Manager.
    • Execute data rules for files with a single click from the Rules grid.
    • View data quality trends for your data sets.
    • View data quality score results for your data sets using the command-line interface.
    New in Governance rollup 3
    • Create single variable quality rules from column properties of a dataset.
    • View and remove quality rules from the rules accordion of a dataset.
    • Rule status and rule quality range charts on workspace overview.
    • Automatic creation of the datasets collection.
    New in Governance rollup 2
    • "File Connector - Engine tier" connection types support. Users of the InfoSphere Information Analyzer thin client can now create "File Connector - Engine tier" connections using InfoSphere Metadata Asset Manager and then view and browse those connections from the thin client to add and analyze files on the engine tier.
    • New advanced search filters: number of data rules, first imported data, last analyzed date, and last published.
    • View data rule and rule set execution results at workspace overview and column-level drill-down screens in the thin client.
    • Multiple selection of data sets in the Find Data and Workspace Data Set Browse screens to allow for a common action (for example, adding data sets to a workspace, running analysis, and so on) to be applied on more than one data set at a time.
    • View row-level details of tables in the data set drill-down.
    • Common file delimiters and an option for "first row contains column names" when previewing and importing HDFS delimited files.
    • Bar chart of ten most and least frequent values for frequency distributions.
    New in Governance rollup 1
    • Create quality rules that quickly and easily run checks on specific aspects of your data.
    • Analyze data sets, work with analysis results, and view data quality scores.
    • View data rules and rule sets that are bound to columns.
    • Evaluate data rules as part of the data quality score for a data set or column.

    InfoSphere Information Governance Catalog


    New in 11.5.0.2 Service Pack 2
    • You can run a new istool sql query command to run SQL queries from the command line. The results are saved in a CSV file.
    • New REST API actions are added. You can now manage stewards and labels by using REST API. All new actions are located in the administration section in Information Governance Catalog REST API Explorer.
    • The verification of the syntax of an imported CSV file is enhanced. The usage of double quotation marks must be compliant with the RFC 4180 CSV standard. For detailed information about the supported format, see the 'Special characters and language support' section in the CSV file format to import information asset values topic.
    New in 11.5.0.2 Service Pack 1
    • You can automatically promote detected term assignment suggestions, which means that such terms are assigned to an information asset. You can configure the feature in Administration > Catalog Options > Term Assignments. New supported asset types: Database, Database Schema, Data File, Data File Folder.
    • Data Visualization is extended to support terms with relationship type Related Terms. Also, you can display relationships of other terms that are in the graph.
    • A new data class attribute Column Name Match is added. A column is analyzed against the data class only if the name of the column matches the filter.
    • When you create or edit a data class, you can set new attribute Column Name Match, which defines a filter for column names. The data inside the column is analyzed only when the column name matches the filter.
    • The display of data quality score, as calculated in Information Analyzer, and of output results for data rules and data rule sets is enhanced.
    • Logging is improved for lineage, VRQL layer, starting and updating, REST API, and workflow.
    • Output messages that are displayed when using REST API are improved.
    • New REST API actions are added. You can now enable or disable workflow and add comments to assets.
    • The process for exporting CSV files is improved by adding byte order mark (BOM) characters so that the output file can be opened in Microsoft Excel.
    New in 11.5.0.2
    • HIVE views can be imported as Database Views and displayed in Information Governance Catalog.
    • Data visualization is supported for terms with relationship types Has A / Is Of, Has Types / Is A Type Of and Synonyms.
    • You can detect suggested term assignments to information assets of types Database Column, Database Table, Database View, Data File Field, and Data File Record.
    • REST API supports generating lineage on a column level.
    • Introduction of SQL Views for Development Glossary.
    • Improved logging performance in the Import/Export area.
    • Support Data Class of type Valid Value, where values are read from an external file.
    New in Governance rollup 7
    • Switching glossary context - With workflow enabled, when you view, or edit terms, categories, information governance rules and information governance policies, you can switch between Catalog and Glossary Development. For details, see Browsing the catalog.
    • Logging improvements - The process of collecting log files has been enhanced. You can view new files to troubleshoot Information Governance Catalog problems. You can also configure the logging level in WebSphere Application Server. For details, see Log files for Information Governance Catalog.
    New in Governance rollup 6
    • New user interface - Introduction of a new and enriched user experience to support the ability to search, display and analyze information within Information Governance Catalog. Enhancements include the ability to better narrow search results according to facets, and to scroll and pan the information details for an asset.
    • Optimization of search terms performance - Searching for terms is faster and the way of sorting search results is different. In the new method, diacritic characters are listed after 'z', for example 'a, b, c, (...), z, ą, ć', in comparison to the old method 'a, ą, b, c, ć, (...), z'. Such ordering method makes searching for terms faster. However, if you want to go back to the old method, you can set the com.ibm.iis.gov.vr.setting.orderInDatabase=true flag to false.
    New in Governance rollup 5
    • Data visualization - The feature has been extended to display graphical depiction of relationships for Database Columns and Data Rules as well. The feature is now supported on both IBM WebSphere Application Server ND (Network Deployment) and IBM WebSphere Application Server Liberty Profile.
    New in Governance rollup 4
    • API for integration of Information Governance Catalog and InfoSphere Information Analyzer - The Information Governance Catalog service fetches DataRuleDefinitions of the provided DataField RIDs from IA in the repository.
    • Enhanced views for Cognos report assets - BI Hierarchies and BI Levels displayed within the Business Intelligence Hierarchies view.
    • Data visualization - The new feature introduces the ability to graphically depict the relationships and binding between Information Governance Policies, their referenced Information Governance Rules, their implemented Data Rules, and their bound Implementation Data Resources. The graphical depiction may be viewed when displaying the details (dossier) of an Information Governance Policy or Information Governance Rule. Further, this feature is currently restricted only when the Services Tier is IBM WebSphere Application Server ND (Network Deployment).
    New in Governance rollup 3
    • The ability to customize and save lineage reports.
    • Custom Relationships - Allow for the setting of a relationship between two or more Assets of the Catalog, defining and publishing dependencies, linkages or usages (such as Trusted Source, Business Owner, etc.)
    • Managing: Glossary Workflow - Allow for a workflow actor, an Editor, Approver or Publisher, to query upon and more easily complete workflow actions for approving and publishing draft content of the Development Glossary.
    • Semantic Mapping, Supporting HIVE - Allow for the seamless flow of Data Lineage thru the jobs and processes that access a Hive and HDFS storage as sources of information.
    • List assets in the development glossary by using the REST API - Use /GET/assets/{id} and select workflowMode.
    • Query the development glossary - In the development glossary, you can query the assets, create published and user queries, import and export query results, and delete queries. The queries that you create can also be used in the catalog. In addition, you can run command-line queries of the development glossary.
    • Use custom attributes to define relationships between assets
    • Trace data flow across HDFS/Hive abstractions by setting the same as relationship between database schemas and data file folders from HDFS.
    New in previous rollups
    • Compliance Reporting - Allow for the continued support of Data Lineage and Risk Data Reporting requirements and their customization and distribution.
    • Data lineage reports for compliance reporting - Design lineage report templates that focus on critical data attributes and facets and customize lineage report information for external audiences.
    • Export information asset values - Export information asset values and then modify term and steward assignments, descriptions, and custom attributes. see Exporting information asset values.
    • Term history migration - Term history is now included when you import and export the glossary.
    • Integration with InfoSphere Information Analyzer - The Details page for assets in InfoSphere Information Governance Catalog displays the following information that is generated in InfoSphere Information Analyzer:
      • Published data profiling results for data files that were imported by using the file connector.
      • Data metrics
      • Data quality scores
    • XML schema library - Display and manage XML schema definition libraries are created during the import of XML Schema Definitions (XSDs) by InfoSphere Metadata Asset Manager.

    InfoSphere ISALite


    New in 11.5.0.2
    • Added diagnostic tests to validate the connection to the IBM InfoSphere QualityStage standardization rules designer database (SRDB) and the Exceptions stage database (ESDB).
    • Added a test to verify the connection to all InfoSphere Metadata Asset Manager connectors and bridges.
    • Added a collection of log file that contains the output of the istool event listEvents command.
    • New Information Governance Catalog log files collected by the collector.
    New in Governance rollup 7
    • New InfoSphere DataStage tests in the General Health Checker.
      • InfoSphere DataStage tests verify the InfoSphere DataStage administrator (dsadm).
      • InfoSphere DataStage tests verify that the dsadm user has read, write, and execute permissions to the UVTEMP folders and others.
      • InfoSphere DataStage tests to verify the number of files in the TMPDIR + /tmp folders. Warnings are issued if there are more than 7500 files.
      • Tests to verify that the Server/DSEngine and Server/DSEngine/bin directories do not have orbtrc/javacore/heapdump files.
      • InfoSphere DataStage Version, 11.3.x fails the Information Goverance Catalog (IGC) test if the IGC RUP23 or RUP24 patches are installed.
      • New Information Governance Catalog test to report the IGC configuration properties in the Information Server (IS) registry.
    • New support for Zookeeper, Kafka, Solr (Shared Open Source).
      • Verify If existing InfoSphere Information Analyzer thin client data sets are indexed or not.
      • Verify and report the status and configuration of SOLR.
      • Verify that the SOLR collection (index) "da-datasets" and "dqecExceptionSets" are created and operational.
      • Add verification and status of KAFKA+ SOLR + ZOOKEEPER services.
      • Add collection of files and logs for Kafka, Solr and Zookeeper.
    • Fixes for the Xmeta Diagnostics.
      • XMeta Diagnostic: The user can now choose the JVM memory maximum heap size in the user interface to avoid out of memory (OOM) issues.
      • XMeta Diagnostic: The user can select which diagnostic probes to run.

    InfoSphere QualityStage


    New in Governance rollup 3
    • Added importable Standardization rule sets for Taiwan and Hong Kong. These rule sets were developed by a third party vendor.

    Managing metadata


    New in 11.5.0.2 Service Pack 1
    • You can use command line interface to compare new events with events that are already imported.
    • Analyze and Import stages are integrated. It means that you no longer need to select option to analyze the metadata while importing it.
    • You can save password for data connections in command line interface in the similar way as in UI.
    • The selection window for data connection is enhanced to help in choosing the correct connection.
    New in 11.5.0.2
    • When you run an express import of metadata, you can skip the preview of the results. It is possible when the administrator properties do not allow the express import to be stopped after the preview.
    • Support for IBM Cognos Analytics 11.
    • Support for MITI 9.1.
    • Support for Tableau 10, 10.1, and 10.2.
    New in Governance rollup 6
    • Support for importing multiple databases in a single area for Informix.
    • The InfoSphere Metadata Asset Manager business intelligence (BI) bridges now have multi-database support.
    • The ability to configure query batch sizes for custom attributes in the service API's in order to improve bulk CSV imports.
    • The ability to export metadata from a Hive metastore and import it into the metadata repository. The CSV bridge in InfoSphere Metadata Asset Manager now supports BI assets.
    New in Governance rollup 4
    • Added the cleanup CAValSets option for the xmetaAdmin command.
    • Enable imports from Tableau through InfoSphere Metadata Asset Manager.
    • Enable imports from MicroStrategy version 10 through InfoSphere Metadata Asset Manager.
    New in Governance rollup 3
    • Support for MITI 9.0.2.
    • Asset interchange of custom attributes now includes custom relations.
    • New import bridge - The QlikView Files Import bridge supports import of business intelligence metadata and related implemented data resources, such as database tables, from QlikView files and databases.
    New in Governance rollup 2
    • Added a new InfoSphere Metadata Asset Manager command-line interface (CLI) option to list out import areas without formatting so that this list can be used in a script to re-import multiple import areas using InfoSphere Metadata Asset Manager CLI.
    • Added support to filter schemas during InfoSphere Metadata Asset Manager import and reimport using a regular expression.

    Upgraded import bridges
    All non-IBM import bridges that are used in InfoSphere Metadata Asset Manager are upgraded. Among the changes are the following:
    • Import PowerCubes from Cognos PowerPlay Transformer by using the IBM Cognos Content Manager bridge.
    • Name changes for bridges: ERwin is no longer owned by Computer Associates (CA), ER/Studio is now owned by IDERA, PowerDesigner is now owned by SAP.
    • Support for import from MicroStrategy 10.
    • Changes in import of dependent objects with IBM Cognos Content Manager bridge. Transfer contract libraries and XSDs between instances of InfoSphere Information Server.

      Managing exceptions and events


      New in Governance rollup 7
      • The ready-to-use Apache Kafka installation in InfoSphere Information Server has been upgraded. The new version is Kafka 0.10.0.1.
      • The istool command has been extended to list all subscriptions of all users.
      New in Governance rollup 3
            

      Deprecated features from the Information Analyzer Workbench


      As IBM transitions Information Analyzer functionality from the Information Analyzer Workbench to the Information Analyzer thin client, we plan to deprecate certain functionality from Information Analyzer Workbench as it becomes available in the thin client or as it becomes obsolete.

      Updated in Governance rollup 7- The Contacts and Policies have been removed from the 'Home->Metadata Management' section of Information Analyzer.
      Similar features are available other places in InfoSphere Information Sever. You can view contacts in the User Management area in the Administration Console. You can view policies in Information Governance Catalog if you have the appropriate licenses.

      Planned for deprecation in the future – Over the next few releases, the following Information Analyzer Workbench features will be deprecated as we continue toward the goal of supporting only the Information Analyzer thin client. Our intent is to provide advanced notice (one or two Governance rollup releases) before a feature is deprecated from the Information Analyzer Workbench. The following features might be deprecated in the Information Analyzer Workbench:
      • Column Analysis – When a few more of the remaining column analysis features move to the Information Analyzer thin client, column analysis functionality will be removed from the Information Analyzer workbench. Note: Today, the InfoSphere Information Analyzer thin client offers nearly all the column analysis functionality that’s available in the workbench.
      • Primary and Foreign Key Analysis – A lot of this functionality has been introduced in the Information Analyzer thin client in rollup 7. You should begin testing and using this capability within the thin client as soon as possible.
      • Data Rule Definitions and Data Rules – When the remaining data rule definition and data rule features are added to the thin client, the functionality will be removed from the workbench. The Information Analyzer thin client already allow you to create and edit data rules, but does not yet allow you to edit data rule definitions. Soon the thin client will have all this functionality. Note: Today there are extensive command line APIs that support creating and modifying data rule definitions and data rules. Many clients who do large scale data rule work use the command line almost exclusively to perform these tasks.
      • Information Analyzer Built-in Reports (invoked from within the Information Analyzer workbench and from the Reporting tab of the Administration Console) We continue to enhance the provided SQL Views and command line APIs to facilitate the query and extract of Information Analyzer results for display and reporting within your choice of Business Intelligence reporting tools, dashboard, or spreadsheet.
      • Baseline Analysis - Alternatives are being considered within the Information Analyzer thin client.
      • Cross-Domain Analysis – This functionality will soon be included in the Information Analyzer thin client.

      Data Integration updates


      The following components have Data Integration updates:
      InfoSphere DataStage
      Connectivity
      InfoSphere Information Server on Hadoop
      InfoSphere Information Services Director

      InfoSphere DataStage


      New in Data Integration rollup 1
      • Improved performance and scalability of the parallel engine TCP port allocation.  Set the new environment variable APT_CONNECTION_PORT_RANGE to 0 for significant improvements in job startup time on busy systems.
      • New option for the Peek stage to produce output in hexadecimal format.  The Peek stage now prints hexadecimal values for the complete RAW data type record when the variable APT_RAW_DISPLAY_HEXADECIMAL is defined. 
      • Added option to change the behavior of APT_STRING_PADCHAR on NLS InfoSphere Information Server systems.  The new environment variable APT_STRING_ALLPADS_NOT_EMPTY changes the rule of string comparison to match a specific requirement.  
      • The most common features in the InfoSphere DataStage Designer client now support Microsoft Windows high contrast themes (Control Panel-->Appearance and Personalization).      

      Connectivity


      New in 11.5.0.2
      • You can use the DB2 Connector to access DB2 database systems for dashDB.
      • You can use MDM Connector stage to run the delete, drop, search and score operations on data on an MDM server.
      • Salesforce Connector supports the Bulk API to extract the data in Bulk Mode for Query Operation.
      • Salesforce Connector supports Salesforce API version 39.0.
      • Support for Teradata 16.0.
      • File connector supports Kerberos keytabs distribution to data nodes when working with Parquet file format.
      New in Data Integration rollup 1
      • Hive Connector support.  The Hive Connector is used to connect to the supported Hive data sources and perform the following operations:
        • Read data from or write data to Hive data sources.
        • Import metadata from Hive data sources by using InfoSphere Metadata Asset Manager.
      • Kafka Connector support. The Kafka connector can be used to connect to the Kafka cluster and perform the following operations:

      • o Read messages from the topics in the Kafka clusters.
        o Write messages into the topics configured in the Kafka clusters.   

    InfoSphere Information Server on Hadoop


    New in 11.5.0.2
    • Support for the following Hadoop distributions: Cloudera Version 5.8.0 and Hortonworks HDP 2.5.
    New in 11.5.0.1
    • Multiple users can run InfoSphere DataStage jobs. Configure InfoSphere DataStage jobs so they can be submitted by individual users.
    • New supported versions of Hadoop Hortonworks Version 2.3, Cloudera Version 5.5, and IBM BigInsights Version 4.1 are now supported.
    • Link to view the status of InfoSphere DataStage jobs in the Operations Console - When you run InfoSphere DataStage jobs by using the YARN client, click a URL in the Resource Manager to monitor jobs in the Operations Console. 


    InfoSphere Information Services Director


    New in 11.5.0.2

    [{"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Not Applicable","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"11.5.0.1;11.5;11.5.0.2","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}},{"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Not Applicable","Platform":[{"code":"","label":""}],"Version":"11.5","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

    Document Information

    Modified date:
    16 June 2018

    UID

    swg21977675