Pacbase futures: Part 2 – Pattern-driven programming in RDz software

By Hervé Le Bars
Chief Architect, Pacbase Convergence tools

The maintenance of applications originally developed by using IBM® VisualAge® Pacbase will be supported in the IBM® Rational® Software Delivery Platform (SDP), specifically by Rational Developer for System z (RDz). This capability is provided by a set of tools implementing the pattern-driven programming paradigm. Hervé Le Bars, Chief Architect of the Pacbase Convergence Tools, presents the main characteristics of this technology and how it is used for the maintenance of Pacbase applications.

This article, the second in a series, describes new features to be added to the Rational SDP that will ensure a high level of productivity of a developer maintaining Pacbase applications. The current article introduces the Pattern Driven Programming (PDP) technique, the use of PDP for Pacbase application models and an overview of the code reconciliation techniques used by the tool. Parts 3 and 4 of this series of articles describe additional features to support enterprise-wide reuse of programming artifacts and to transfer Pacbase assets to the new tool.

A- Pattern Driven Programming

The Pattern Driven Programming (PDP) technique has been supported by the Pacbase tools for years and has been proven to facilitate efficient application development and maintenance. This programming technique enhances a developer’s productivity in both the creation and maintenance phases of an application’s development lifecycle. Other benefits of this programming technique are enhanced reliability and predictability of the coding activity and enhanced maintainability of code.

Definition of a programming pattern

A programming pattern is an implementation template for the solution to a recurring class of problem in an application domain.

An implementation template needs parameters to derive an actual implementation. A set of pattern parameters describe an actual problem to solve in the class of problems addressed by the pattern.

A Domain Specific Model (DSM) or Domain Specific Language (DSL) can be used to describe the parameters specific to a particular programming pattern. In the rest of this article, only the DSM term is used.

The DSM can be seen as a programming language dedicated to a particular class of problems. An instance of DSM is a particular program in this dedicated language. A DSM instance can be transformed into another program in a target programming language, which is known as the implementation. The implementation conforms to a standard of implementation (or implementation template) for this class of problem.

It is important to understand that both the DSM and the implementation template are part of the definition of a programming pattern. The implementation must follow a standard structure, based on a functional decomposition of the problem to be solved. The documentation of a programming pattern includes the documentation of the DSM and the documentation of the generated code (how it is structured). If you take the DSM from an existing pattern, but change the code generator, you have defined a new pattern.

Description of how a programming pattern emerges

Programming patterns emerge by practice.

A community of programmers in an application domain isolates a recurring class of problems and proposes a reference implementation and a reference vocabulary. The implementation template or the functional decomposition for this class of problems is derived from the reference implementation.

The variation of problems in the class is captured in a model (DSM).

Interestingly, in the process of creating a new programming pattern, the implementation generally emerges before the model. The model is defined from the variations of the implementation.

The target of a programming pattern

A programming pattern is generally tied to a set, possibly a large set, of underlying languages and technologies (for example, COBOL and CICS) because a programming pattern includes an implementation template.

For this reason, Pattern Driven Programming techniques are generally used by programmers in the coding phase of application development.

The link to Model Driven Development

Model Driven Development (MDD) uses Unified Modeling Language (UML) models to define the global architecture and high-level processes of an application. The modeling tools are often used by analysts, architects or project managers, including persons who prefer not to deal with implementation concepts or underlying programming languages. Tools that support MDD can often generate skeletons for implementation but are not dedicated to generate a detailed implementation, including algorithms solving a specific class of problem.

Pattern Driven Programming (PDP) can generate implementation details from a DSM instance. Tools that support PDP are mainly used by programmers in the coding phase of an application’s development. Use of PDP techniques is directly applicable only for application domains that are covered by a set of pattern instances. To address a new domain using PDP, you must first define specific patterns for this domain. MDD, using more abstract models, is directly applicable to any domain.

MDD and PDP can be complementary approaches. UML models can be used to define global application architecture and high-level processes. In the coding phase, these models can be used to generate initial DSM instances, which can be used to generate an actual implementation.

Programming patterns versus programming languages

A programming pattern defines a Domain Specific Model (DSM). A DSM can be seen as a new programming language, dedicated to a particular class of problems. In particular situations, the use of a set of well-chosen programming patterns can produce a fully functional application, and thus discard the need for coding at a lower level with a general purpose programming language. But this is generally not the case.

Generally, DSMs cannot describe all the required functionalities of a particular application. DSMs capture recurring needs in an application domain, but cannot express specific business logic. For this reason, a general purpose programming language is required to complete the code produced from the pattern. The user must introduce specific business logic by using a general purpose programming language.

Because the implementation derived from a DSM instance obeys a standard documented structure, it is easier for the developer to know where to add specific code.

The coexistence of two levels of programming leads to interesting challenges for the tools that support Pattern Driven Programming.

Tools required for Pattern Driven Programming

To support Pattern Driven Programming, the following set of tools are required:

• Tools to support the creation of an application based on patterns:
  - Dedicated editors to create DSM instances
  - Code generators to generate an implementation from a DSM instance
  - Enhanced code editors to add specific business rules (user code) in the generated code to fulfill requirements that are not captured by the DSM
• Tools to support maintenance of an application based on patterns:
  - Automated code reconciler (reconciliation engine) to support the regeneration of a new implementation from a modified DSM instance, while preserving existing user code
• Tools to support authoring of new patterns:
  - Model editor to define a new DSM
  - Tool to produce a dedicated editor for the new DSM
  - Generation technology to define the transformation of a DSM instance into code

An editor for a DSM instance can be a textual or graphical editor. The editor allows the user to create and modify a DSM instance.

The enhanced code editor is an editor showing the implementation and allowing you to add new code or remove and replace existing code. This editor should make a clear distinction between the generated code and the user-specific code.

The reconciliation engine is a centerpiece of these tooling elements. Its role is to reapply changes made by the user whenever the code is regenerated.

The process of Pattern Driven Programming

When practicing Pattern Driven Programming, the developer iterates on the two following activities:

1. Create or modify a DSM instance and generate or regenerate the implementation, and then go to activity 2.
2. Modify the generated code (add specific code and remove or replace generated code), and then go back to activity 1.

Activities 1 and 2 are two different kinds of coding activities at two different levels. Activity 1 is sometimes called high-level coding because the developer uses a programming language dedicated to the class of the problem to solve. Activity 2 is called low-level coding because the developer uses a general purpose programming language (typically, a 3GL).

The developer builds the application incrementally, switching from activity 1 to activity 2 and vice versa. Each time the implementation is regenerated, the reconciliation engine is invoked.

Benefits of using Pattern Driven Programming

The main benefits of using tools supporting the Pattern Driven Programming techniques are the increased reliability and maintainability of code and the predictability of coding activities.

• Reliability: code generation is supposed to use best practices for quality and efficiency.
• Speed: code generation and reconciliation speeds up the coding process – both during the creation of new applications and in the maintenance of existing applications.
• Maintainability: code is readable and documented.
  - The implementation and vocabulary are homogeneous among all the variations of the same pattern.
  - Ask 10 developers to develop the same application. You might obtain 10 working applications that are written with 10 different algorithms or styles that make them hard to understand by another developer. Generation from a DSM produces variations over the same pattern of code. Developer creativity is not expressed in the generated code.
• Predictability: generated applications are variations of the same pattern. The main effort is to enhance it with business logic. It is thus easier to assess:
  - The cost of writing and testing a new application
  - The skills needed to write a new application

These benefits are obtained during the initial implementation of an application and during its maintenance.

B- Example: the Pacbase batch program pattern

The size of a pattern can vary from very small patterns that generate only a few lines of code in a program, to very large patterns that generate several programs or a complete application.

A concrete example can be used to illustrate the concepts that were introduced in the previous chapter: the Pacbase batch pattern. This pattern generates a single COBOL (or PLI) program but uses a rich DSM.

Batch operations are used to work with and manipulate files. In this domain, recurring operations are easy to isolate. You can filter, merge, update, join, control, sort files and write reports:

• Filter: extract data from a file, based on a specific criterion, convert the data and store the result in another file.
• Merge: merge two sorted files into a new sorted file, based on a sort criterion.
• Update: updates file content based on a list of movements (create, modify, delete).
• Join: join records coming from different input files, based on specific join criteria.
• Control and Report: check input files content based on specific criterion and print a report.
• Sort: sort a file according to a sort criterion.
• Report: print formatted reports based on content of input files.

Here is a simplified overview of the corresponding DSM:

 
The batch programming pattern introduces a dedicated vocabulary: file synchronization, break detection, multi-structure files, movement files, and other terms. Note that the term file has a particular meaning in the context of the batch pattern. The term file signifies any list of 0 to N records, stored for example in a table in a RDBMS, a DL1 node, a message queue, a memory array or, a real file (sequential, indexed, text or binary).

A tool supporting the batch programming pattern allows the developer to easily define a batch program that combines batch operations on files. In a DSM instance, the user declares the files that are to be read or written by the program, and the operations that are applied to them. Then, the generator for batch pattern can generate the standard implementation of the batch program from the DSM instance. In Pacbase, the implementation is in the COBOL or PLI programming language.

All batch programs generated from a pattern have the same general structure. Here is a simplified overview of the standard implementation of a batch program:

    For all declared files
        Open file

    While there are more records to read
        For each file to read
            If read condition is true for this file
                Read next row

        For each declared operation on files
            Take operation on selected rows

        For each file to write
            If write condition is true for this file
                Write next row

    For all declared files
        Close file

Precisely, the COBOL implementation of a Pacbase batch program is structured into functional blocks of code (functions). Each function has a dedicated role and is identified by a dedicated tag. Here are some of the functions of a Pacbase batch program in the procedure division:

    F01: Start of program, initializations.
    F01xx: Open file xx.
    F05: Beginning of main loop.
    F05xx: Read next row of file xx.
    F10: Read files with break detection.
    F10xx: Read next row of file xx, if break detection is active.
    F20: Test exit conditions, if true close files and exit.
    F20xx: Close file xx.
    F24: Compute synchronization flags.
    F26: Break detection.
    F30: Check validity of movements.
    F76: Perform update logic based on a movement
    F90: Write files.
    F90xx: Write file xx.
    F9099: Loop to F05.

C- Overview of the support of a pattern in the RDz product

The following screen capture shows a Pacbase batch program being edited in RDz. The program shown here (JOINCO) is a very simple batch program that merges two input files. The program reads two input files CU and OR. These two files contain respectively a list of customer records and a list of order records, sorted by increasing customer ID. The program writes a single file on output, whose records contain both customer and order information, sorted by customer ID.

 
Design View

The Design View (on the left) shows the DSM instance of the batch program being edited. The content of this view is always synchronized with the COBOL editor (displayed in the middle). It always shows the DSM instance that has been used to generate the code displayed in the source editor.

The design view and the editor are shown side by side to allow the user to quickly navigate from the DSM instance to the associated implementation code.

When the user has changed the DSM instance, he can regenerate the implementation using a button in the toolbar of the design view (). Each time the code is regenerated, the reconciliation engine is invoked to mix the newly generated code with any existing user specific code.

Generated Code Structure View

The Generated Code Structure View (on the right) shows the functional decomposition of the program. In the example shown, you can recognize the standard structure of a batch program, driven by the Pacbase batch pattern.

Grey nodes in this view correspond to generated functions. Black nodes indicate changes made by the user to the generated code (user specific code). In the example shown, the user has inserted code in the function F01, removed code in the function F05CU and replaced code in the function F05OR.

Using this view, the user can quickly and easily retrieve all the modifications made to the generated code.

Enhanced source code editor

The enhanced source code editor is based on the COBOL editor of RDz called System z LPEX. The LPEX editor is enhanced to support pattern-related highlighting and to link the editor to the design view and the generated code structure view. The generated code, coming from the DSM instance, is shown in grey, while the user specific code uses the default syntax highlighting of LPEX.

Each time the user makes a change in the implementation code, the highlighting in the editor and the content of the Generated Code Structure View are refreshed accordingly.

D- The reconciliation engine

The code reconciliation engine is a key technology that is required to support Pattern Driven Programming. Its role is to preserve user-specific code after a regeneration of an implementation from a DSM instance. The reconciliation engine operates transparently to the user each time the user regenerates an implementation from a DSM instance.

External view of the reconciliation agent

 
The reconciliation agent takes three descriptions as input:

• The current implementation code, including user-specific code
• The previously generated implementation code
• The new generated implementation code

On output, the reconciliation agent produces two other descriptions:

• The new implementation code, including user code
• A list of reconciliation warnings that indicate potential issues detected during reconciliation

Main reconciliation techniques

To perform its task, the reconciliation agent combines two different code reconciliation techniques: textual reconciliation and functional reconciliation.

Textual reconciliation operates on texts. As input, it takes three pieces of text, one of them playing the role of the reference, the two others being two modified versions of the reference. As an output, it produces another piece of text based on the reference text but including the modifications made in the two modified versions. When modifications made in the two modified versions overlaps, a conflict is detected producing a reconciliation warning. Textual reconciliation is frequently used in SCM (Software Configuration Management) technologies, to merge changes made by two developers in the same file in parallel. Generally it is based on an algorithm called diff3. Textual reconciliation is not specific to the reconciliation of code produced from a pattern.

Functional reconciliation is an original technique for the support of Pattern Driven Programming. This reconciliation technique is dedicated to the reconciliation of code produced from a programming pattern. It takes advantage of functional decomposition induced by a programming pattern.

Enhanced textual reconciliation

The textual reconciliation mechanism implemented in the reconciliation engine is enhanced in two different ways compared to the standard diff3 algorithm.

First, you use a refined diff3 algorithm to reconcile cases that are rejected by the standard diff3 algorithm.

Second, the enhanced textual reconciliation takes into account reformatted code. Changes in the formatting of the code are distinguishable from actual changes to the code content and are preserved after regeneration only if the generated code did not evolve.

Functional reconciliation

Textual reconciliation can give disappointing results in some instances, especially in the following cases:

1. A totally new portion of code is generated.
2. A portion of code is no longer generated.
3. The generated code has been reordered.

Moreover, it can be very time-consuming to work with a large amount of text by using a textual reconciliation mechanism based on the diff3 algorithm.

Functional reconciliation helps solve these issues. It is based on tags that are generated with the code, which means that the functional reconciliation does not only take text as an input but it also takes generated tags as input. Here is the revised external view of the reconciliation agent to support functional reconciliation:

 
A reconciliation tag is just a name that is attached to a portion of the generated code. Tags are set by the code generator. Each tag identifies the role of a particular portion of code.

There are three fundamental rules for dealing with tags:

Rule 1:
If a tag is generated with the new generated implementation, but was not generated with the previous one, the corresponding portion of code is considered as new code. The reconciliation engine excludes this new portion of code of the textual reconciliation.

Rule 2:
If a tag is generated with the previous generated implementation, but is not generated with the new one, the corresponding portion of code is considered dead. The reconciliation engine excludes this dead portion of code of the textual reconciliation.

Rule 3:
If a same tag is generated with both generated implementations, the two corresponding portions of code are reconciled together using textual reconciliation.

Using generated tags and applying these three core rules, the cases where textual reconciliation gives disappointing results can be handled elegantly.

In addition, tags isolate portions of code to be treated individually using textual reconciliation based on the diff3 algorithm. Instead of passing large pieces of text to textual reconciliation, you successively pass a set of smaller text portions. This activity leads to much better scalability of the reconciliation algorithm.

Example of textual reconciliation

Here is a portion of code that has been initially generated from a DSM instance:

Fig1. Initial generated code 
You can see that the code is shown in grey in the editor because it is pure generated code. From this portion of generated code, the developer can insert specific business logic. Here is the same portion of code, after the user inserts specific business logic:

Fig2. code modified by the user 
You can see that the user has modified the first line and has added a new line (AND MV00-FNAME …) and has removed another line (IF MV00-NAME …).

After that, the user can go back to the editor of the DSM instance, make a few changes and regenerate the code. Here is an example of the new generated code after some changes have been made to the DSM instance:

Fig3. new generated code 
You can see that the MV00-CMD element has been replaced with the MV00-ACTION element. Also, two new lines have been generated at the end of the portion. Note that the user cannot see this newly generated code in the editor. The user sees the result of the textual reconciliation, as shown in this case:

Fig4. new code after textual reconciliation 
This is a simple example. In some cases, conflicts can happen. In case of conflict, the reconciliation engine sets a warning on the conflicting lines. Each warning holds a set of quick fixes, to help the user solve the conflict manually.

Example of functional reconciliation

The previous example shows how a portion of code is reconciled using textual reconciliation. When using functional reconciliation, the code is split into a set of distinct portions. A set of tags that are not visible in the code are defined by the code generator. Each tag has a unique name and refers to a particular portion of the generated code. The tags are organized into a hierarchy where the bigger tags include the smaller ones.

Functional reconciliation consists in the analysis of the way the tree of tags evolves from one generation to the next. Here’s a simplified example where tags are organized in a small list. The following piece of generated code serves as an example:

Fig5. initial generated code 
You can’t see tags in this generated code because the generator does not store tags in the code. Assume that the generator has defined three tags on this piece of code, splitting this piece of code into three named sections: READ-CU-FILE, READ-MV-FILE and READ-OR-FILE.

 
Starting from this initial generated code, the user can insert specific business logic. For example, here is the code after the user has made changes:

Fig6. code modified by the user 
You can see that the user has modified a line in the first portion of code, deleted a line in the second, and inserted a line in the last portion. Now, assume the user goes back to the DSM instance, makes a few changes and regenerates the implementation. For example, the generator generates the following newly generated code:

Fig7. new generated code 
The generator also generates a set of tags for the newly generated code. For this example, the new tags can be:

 
The user cannot see this new generated code in the editor. The user only sees the direct results of the reconciliation, which is, in this case:

Fig8. code after reconciliation 
To compute this result, the reconciler first reviews the old and the new list of tags.

Old list of tags:
READ-CU-FILE, READ-MV-FILE, READ-OR-FILE.

New list of tags:
READ-OR-FILE, READ-XX-FILE, READ-CU-FILE, READ-YY-FILE.

The READ-MV-FILE tag is not generated with the new generated code. So the user changes made in this portion of code are not injected in the new implementation.

The READ-XX-FILE and READ-YY-FILE tags did not exist in the original generated code. So these two portions of newly generated implementation are not modified by the reconciler.

The READ-CU-FILE and READ-OR-FILE tags exist in both generated implementations. To reconcile code lying in these two tags, the textual reconciliation mechanism is applied. Note that in this example, the textual reconciliation is straightforward for these two portions of code, which is not a general rule.

Without the help of generated tags, the reconciler cannot obtain the same result in this very simple example. Combining functional and textual reconciliation makes it possible to obtain much better results than textual reconciliation alone.

Summary

This article introduced the Pattern Driven Programming (PDP) technique, inherited from the Pacbase experience. It quickly introduced how PDP is supported in the Rational Software Delivery Platform. Finally, it described a key technology of these new tools, the reconciliation engine.

Other new features, addressing team productivity through the support of enterprise-wide reuse of programming artifacts and ensuring the transfer of Pacbase assets to the new environment are presented in Part 3 and 4 of this series of articles.