Directory Integrator, Version 7.1.1

Creating your first AssemblyLine

Returning to the example scenario outlined in the introduction, you will now create an AL that migrates information from D1 to D3, ignoring for the moment the joining of data from D2.

Figure 1. Simplified scenario diagram with just two data sources

The 'Tutorials' folder (that you should have copied from TDI installation directory/examples to your Solution Directory) contains a file named People.csv:

First;Last;Title
Bill;Sanderman;Chief Scientist
Mick;Kamerun;CEO
Jill;Vox;CTO
Roger
Gregory;Highpeak;VP Product Development
Ernie;Hazzle;Chief Evangelist
Peter;Belamy;Business Support Manager

You can see from the above listing that this is in character separated value format (CSV). This file represents our D1 input data source. Your AL will extract this data and transfer it to an XML document which will be our D3 output target.

Click on New AssemblyLine in the topmost toolbar and call the new AL 'CSV2XML'.

Figure 2. New AssemblyLine dialog box

Now press the Finish button to open the AL in an AssemblyLine editor tab.

Figure 3. Empty AssemblyLine editor

The left part of the AL editor contains the list of components that make up this AssemblyLine and is empty right now except for the section names: Feed and Data Flow. The right-hand area displays all Attributes being mapped in and out of the AL.

To understand these AssemblyLine sections, consider for a moment what we want this new AL to do: For each line in the CSV file, create a new node in the XML document. This looping behavior is provided for you automatically by the Tivoli® Directory Integrator kernel, driving components listed under the AL Data Flow section as long as there is input data coming from Connectors in the Feed section¹.

Let's take advantage of this functionality by adding a Connector to the Feed section to read in our CSV input file. Do this by right-clicking on the Feed section folder and selecting Add Component...

Figure 4. Inserting a new component

Inserting a new component to the Feed section

You will be presented with the Choose Component wizard.

Figure 5. Choosing the component

This dialog gives you a couple of options to find and select the component you want:

Start typing any part of the name of the component in the text field and the selection list to the right is filtered accordingly. For this example, type "file".
You can optionally limit the selection list to include only a single type of component – Connectors, Parsers, Scripts, and so forth.
Locate and select the component you want from this list. In our example this will be 'File Connector'.

The new Connector is automatically named 'FileSystemConnector' for you. Change this to 'Read_CSV_File' so that it has more meaning in the context of your solution² and then select Iterator from the Mode drop-down.

Figure 6. Renaming the Connector and changing its mode

It's the Mode setting of a Connector that tells the built-in AL execution logic what role this component plays in the flow. Iterator Mode results in the for-each behavior you need in order to drive the data from the CSV file, one entry at a time, to the components you will add to the Data Flow section.

Now press the Next button to continue on to the configuration panel for the selected Connector.

Figure 7. File Connector Configuration panel

Each component provides its own set of configuration parameters. The ones shown onscreen now are for the File Connector and it has only one required parameter: File Path. Type in the path to the People.csv file – either the full path, or the relative path from your Solution Directory as shown in the screenshot above³ – or press the Select button to bring up a file browser to locate this file.

Because a formatted text file is a byte stream and not a structured data source like a database or directory, you must set up a Parser to interpret the formatting of the stream as it is read. Tivoli Directory Integrator provides a powerful and versatile Data Browser feature for interactively testing your Connector/Parser selection and configuration.

We'll take a look at this in a moment, but first you need to complete this wizard by pressing Next again and proceeding to Parser Configuration. Here you click on the CSV Parser to select it.

Figure 8. Selecting Parser during Insert new object

Selecting Parser using 'Select Parser' button during Insert new object

Once you have selected the CSV Parser then press Finish to close the wizard. You will now see the Parser Configuration panel. Since you don't need to change the default settings, simply press Finish again to complete the wizard.

The next step is to have your Connector discover the schema of your input source in order to map these values into your AssemblyLine. This is where the Data Browser comes in handy⁴. Start with the Data Browser by right-clicking on your new Iterator Connector in the AssemblyLine Components tree and selecting Browse Data from the context menu.

Figure 9. Browse Data context menu selection

This will open the Data Browser in a new editor tab.

Figure 10. The Data Browser

The area labeled 1 in the above screenshot is for choosing – and changing – the selected Parser. Area 2 provides a Details tab that shows you the raw byte stream that will be parsed. There are also tabs for changing the Connector's Connection parameters, as well one for configuring the chosen Parser.

The last section of this dialog (#3 in the screenshot) is for connecting to the data source and discovering which Attributes are available. Do this now by first pressing Connect and then the Next button.

Figure 11. Interactively discovering schema by browsing live data

Interactively discovering schema by browsing live data, clicking Next button

You have now discovered the schema of this file. Select the Attributes you want to map in, which in this case is all of them, by either selecting the checkbox next to each one, or using the Select All button.

Use the Ctrl-W shortcut to close the Data Browser tab, or simply click on the 'Close' symbol (X) at the right edge of the tab and return to the AssemblyLine editor where your AL should look like the screenshot below⁵.

Figure 12. AL with Iterator Connector in place

Details for the selected component are shown to the right of the AL component list, including the three mapping rules you just set up in the Input Map. Each Attribute Map item has an Assignment, which is a snippet of script that is evaluated in order to set the value (or values) of the target Attribute.

Before continuing, take a moment to reflect on these Assignments: You will recall from the Entry-Attribute-value data model section that the AssemblyLine has a globally available Work Entry that carries all data being transported down the AL. This object is referenced in script code by using the pre-registered script variable work. In addition, the Interface of every Connector has its own Conn Entry that is used as a cache for reads and writes. This component-specific object is accessed from script through the pre-registered variable conn⁶. To illustrate, consider the first mapping rule. It creates an Attribute in the Work Entry named 'First'. Its value is derived from the following assignment:

	conn.First

This shorthand notation references the Attribute called 'First' that was just read into the conn Entry, and its values are used to populate the new Work Entry Attribute. A comparable assignment script would be:

return conn.getAttribute("First");⁷

Getting back to the exercise, you now you need to add your output Connector for creating the target XML document (data source D3). This time try using the Add component button at the top of the AssemblyLine Components panel.

Figure 13. Add component button

Again choose the File Connector, renaming it to 'Write_XML_File'. Leave the Mode setting as AddOnly and then press Next.

In the Connector Configuration panel, set the File Path parameter to write to a file called Output.xml in the Tutorial folder. Then choose 'XML Parser' in the next Wizard panel. Now you can press Finish since you don't need to change the XML Parser configuration. Note that in the case of your output Connector, you can't do Schema Discovery since there is no Output.xml file to discover from.

Figure 14. AL with two Connectors in place

You may have noticed that when you select a component, its details appear in the right part of the editor screen. Whenever you select either the 'Feed' or the 'Data Flow' folder, you are presented with the overview of all Attribute Maps for this AssemblyLine. This is a handy display for copying your input Attributes to the Output Map of your latest Connector, so bring up this screen now by clicking on either 'Feed' or 'Data Flow'.

Here you see the list of Attributes (three in total) that are being brought into your AL by the Iterator-mode Connector. Select these Input Map Attributes⁸ and drag them down to the Output Map of the 'Write_XML_File' Connector, completing the data flow.

Figure 15. Dragging Attributes to the Output Map

Notice how the Assignment is automatically converted from input format to output. For example, the first map item in the Input Map of your 'Read_CSV_File' Connector will create an Attribute in the Work Entry named 'First' to hold any values found in conn.First (that is, the Attribute called 'First' that was read into the Conn Entry). When you drag this input mapping rule to an Output Map then its assignment is changed so that the value now comes from the Work Entry instead, and it is creating a target Attribute in the Connector's cache (the Conn Entry).

If you want to change the source for any mapping rule then edit the assignment. In order to change the name of the Attribute being mapped to, simply right-click and rename it. Do this now for the first two Output Map rules.

Figure 16. Renaming an Attribute Map rule

Change 'First' to 'FirstName' and 'Last' to 'LastName'.

Now add a new map item to this Output Map by right-clicking on the 'Write_XML_File' Output Map itself and choosing Add Attribute.

Figure 17. Adding the 'FullName' Attribute to the Output Map

Call the target of this new mapping rule 'FullName', press OK and then double-click on it to edit its assignment.. This opens up the Script editor panel and presents you with a default assignment script: work.FullName. Of course there is no 'FullName' Attribute in the Work Entry, so this map will not be able to set any values. Instead, you must compute this value by changing the script so that it concatenates the First and Last Attributes, leaving a single space between these values:

Figure 18. Editing the assignment

The script should read as follows:

work.First + " " + work.Last

Note that no terminating semi-colon is required for one-liner Attribute Map assignment scripts like this⁹. Press the Close button at the top-right of the Script editor panel when you are done.

¹ Note that only one Feeds Connector will be delivering data to the AL at a time. If you put more than one Iterator Connector here then the topmost one will empty first before the next one in line begins reading from its source.

² Although you can name Connectors as you like, it is recommended that you name them in the same way that you would a script variable: start with a letter, followed with any number of letters, digits and underscore characters. This is because all AL components are automatically registered as script variables, making it easier if you later want to reconfigure and drive them from your script code.

³ This technique makes your solution easier to move and share since all you have to do is specify the Solution Directory you want and all relative paths will work unaltered.

⁴ Since you know the file is in CSV format, the quickest approach would be to just click on the Connect and Next button in the Schema area of the Iterator Connector. Then you drag discovered Attributes into the Input Map as you want. The Data Browser is useful when you are unsure of the format. But I still thought you ought to try it :)

⁵ If for some reason your Connector is in the Data Flow section, simply drag it up to Feed. If the mode setting is not Iterator then right-click on the Connector, select Mode and then choose Iterator.

⁶ The conn variable is only available for limited periods, as shown in the Tivoli Directory Integrator Hook Flow Diagrams. Outside this scope it is still accessible by querying a component for its conn Entry.

⁷ For users familiar with version 6.x and earlier, you can also use the pre-7.0 syntax:

  	ret.value = conn.getAttribute("First")

⁸ You can Control-click to select multiple, or use Shift-click to select a range.

⁹ Pre-7.0 syntax is also supported so map assignment scripts can still start with "ret.value =".

[ Terms of use | Feedback ]

IBM Tivoli Directory Integrator 7.1.1