IBM Streams 4.2

Mixed-mode SPL/Perl sources

You often must parameterize application composition when you create parallel subgraphs, conditional segments, and variable length pipelines. The SPL compiler provides a Perl-based preprocessor for this purpose. SPL files that are authored with the preprocessor directives are named with the .splmm extension and are called mixed-mode SPL programs.

As a preprocessing step, the SPL compiler converts .splmm files into .spl files before it starts the actual compilation process. It first creates a code generator (in Perl) out of the .splmm file and then runs that code generator to produce the final .spl file. The generated .spl files are put into read-only mode.

An .splmm file is a mixture of SPL and Perl code. The Perl code is used to generate SPL code that is based on the command-line parameters that are specified and the variables that are defined in the Perl code. Any Perl code that is used in an .splmm file is enclosed between one of the following two scoping pairs:

<% [Perl-code-here] %>: This code is directly inserted into the Perl code generator that is derived from the .splmm file. Any print statements inside the scoping pairs generate SPL code that ends up in the generated .spl file.
<%= [Perl-expression-here] %>: This shorthand notation is for: <% print [Perl-expression-here]; %>. The shorthand notation automatically includes the print keyword and the trailing semicolon.

Note: Use the character sequences <% and %> to enclose only Perl code in the source. If either character sequence is needed in a string, one of the characters in the sequence must be specified with its Unicode equivalent.

Any SPL code that appears in the .splmm file goes into the final .spl file after it is interpreted by the Perl code that might be surrounding it. This process is best described with an example:

namespace sample;

<%  # switch to Perl scripting mode
    # parse arguments, do error checking
    use integer;
    my $numLevels = $ARGV[0];
    unless(defined($numLevels) and $numLevels>0) {
       print STDERR "ERROR: Please specify the number ",
                    "of tree levels (a positive number).\n";
       exit 1;
    }
%>

composite Main {
graph
    stream<rstring data> Src = Beacon() {
        logic state : mutable int32 i = 0; 
        param iterations : 10u;
        output Src : data = (rstring)(++i);
    }

    // the root node
    stream<rstring data> Node_0_0 = Filter(Src) {}

    <% # create levels of the tree
       my @leaves; # collect leaf streams
       my $n = 1; # num. of nodes at a level
       for(my $i=1; $i<$numLevels; ++$i) {
           $n *= 2; # num. of nodes at this level
           for(my $j=0; $j<$n; ++$j) {
               my $p = ($i-1)."_".($j/2); # parent
               my $s = "${i}_${j}"; # self %>

               stream<rstring data> Node_<%=$s%> = Filter(Node_<%=$p%>) {}

               <% # collect last level's streams
               push @leaves, "Node_${s}" if($i==$numLevels-1);
           }
     }
     my $sinkStreams = join(",",@leaves);
  %>

  () as Writer = FileSink(<%=$sinkStreams%>) {
     param file : "Results.txt";
  }
}

The example code generates a binary tree of the Filter operators with mixed-mode support. The result can be visualized as follows:

Figure 1. Binary tree of the Filter operators with mixed-mode support Binary tree topology that is generated by the use of mixed-mode SPL.

The first part of the code is Perl only and it ensures that the command-line arguments passed to the script are valid. It also stores the first command-line argument into a variable named $numLevels to use later during the tree generation phase.

The second part of the code deals with the creation of the tree. The first Perl loop iterates over the levels and the second for loop iterates over the nodes in each level. The SPL code is used for creating the Filter operators as nodes in the tree. The SPL code is repeated within the for loops, except for the parts that are specialized based on the loop indexes. An example is:

stream<rstring data> Node_<%=$s%> = Filter(Node_<%=$p%>) {}

When preprocessed, the line that is defined in the excerpt translates into the following SPL code (assuming that the number of levels is 3):

Compilation command:
  sc -M my.sample::BinaryTree 3

Segment from the preprocessed output:
  stream<rstring data> Node_1_0 = Filter(Node_0_0) {}
  stream<rstring data> Node_1_1 = Filter(Node_0_0) {}
  stream<rstring data> Node_2_0 = Filter(Node_1_0) {}
  stream<rstring data> Node_2_1 = Filter(Node_1_0) {}
  stream<rstring data> Node_2_2 = Filter(Node_1_1) {}
  stream<rstring data> Node_2_3 = Filter(Node_1_1) {}

When there are errors in the generated SPL code, the SPL compiler reports the error using both the .spl file location and the .splmm file location. An example is given as follows:

my.sample/BinaryTree.spl:10:12: CDISP0053E ERROR: Unknown identifier ‘rstringd'.
my.sample/BinaryTree.splmm:19: CDISP0053E ERROR: Unknown identifier ‘rstringd'.

When there are errors in the generated Perl code, the SPL compiler reports the error using the splmm file location, if possible. When it is not possible, errors reported from the Perl compiler are listed. An example of the former is given as follows:

my.sample/BinaryTree.splmm:32:0: CDISP9012E ERROR: (Perl) No such class n at
   my.sample/BinaryTree.splmm line 32, near "my n"
my.sample/BinaryTree.splmm:32:0: CDISP9012E ERROR: (Perl) syntax error at
   my.sample/BinaryTree.splmm line 32, near "my n ="

When there are errors in the matching of the scoping pairs, the SPL compiler reports the error using the .splmm file location. An example is given:

my.sample/BinaryTree.splmm:37:0: CDISP9010E ERROR: The separator ‘%>' is not paired.

To get error messages that include the .splmm locations, any print statements used in the Perl code must appear as the first token in the line. For instance:

<%
   my $a = "value";
   print "stream <int32 $a> = ";              # right
   $a = "length"; print "stream <uint32 $a>"; # wrong
%>

The use of print statements inside the Perl code within mixed-mode applications is discouraged.

Multiple mixed-mode files: If an application contains multiple mixed-mode files, they are preprocessed using the same set of command-line arguments. These arguments are specified on the sc command line. Any arguments that are not used by the compiler are passed as is to the preprocessor. If different mixed-mode files require accessing a different set of arguments, then named arguments can be used by the developer to resolve any ambiguities that might arise

Mixed-mode files in toolkits: Mixed-mode files can also appear in toolkits. These mixed-mode files are preprocessed during toolkit indexing. For more information on toolkit indexing, see the spl-make-toolkit utility. Command-line arguments to the preprocessor are specified during toolkit indexing as well. If a different set of arguments are wanted, the toolkit must be reindexed with the new set of arguments