IBM Streams 4.2

Operator MapStore

SPL standard and specialized toolkits > com.ibm.streams.geospatial 3.0.2 > com.ibm.streams.geospatial > MapStore

Primitive operator image not displayed. Problem loading file: ../../image/tk$com.ibm.streams.geospatial/op$com.ibm.streams.geospatial$MapStore.svg

The MapStore operator stores geometry data for the edge network in a shared map store, which the PointMapMatcher operator can access. Many PointMapMatcher operators can access the same shared map store. Therefore, the amount of memory is less than the memory, which would be required if every PointMapMatcher instance has its own copy of the geometry data.

Shared Memory Segments

The shared map store consists of following shared memory segments:

  • <storeName>

    Holds the map data.

    Multiple PointMapMatcher operator instances running on the same host, can access the same map data, thus reducing the application's memory footprint.

  • <storeName>.control

    Holds some control data like the mutex and state information to detect malfunctions.

Access permissions for shared memory segments are handled by the operating system like file permissions. To ensure that the MapStore and the PointMapMatcher operators can access the segments, it is recommended to run the Streams jobs containing the operators as the same operating system user.

Operator States & Events

The MapStore operator stores some state information in the control shared memory segment and produces event tuples on state changes, on which the application can and should react. The operator can be in one of three states:

  • modifying: A modification of the map data is in progress.
  • ready: The map data is valid. No modification is in progress.
  • waitingForData: The map data is either not available, or invalid. No modification is in progress.

The following graphic shows the states and events.

MapStore states and events

During each startup phase the operator sends a started event. Operator starts or restarts can happen under following conditions:

  • A job is submitted.
  • The PE that contains the MapStore operator is manually restarted with the streamtool restartpe command.
  • The PE that contains the MapStore operator is automatically restarted because the operator is configured to be restartable and the operator failed.

If the shared map store does not exist so far, the operator creates it and sends the waitingForData event. The lock that prevents the PointMapMatcher operator from reading is locked because the MapStore operator waits for map data.

If the shared map store exists at startup time, a state information is already available.

If the existing state is modifying, which means that the last time the MapStore operator was running a modification was in progress, the new state is either waitingForData if the faultHandling parameter is set to strict, or ready if the permissive mode is enabled. A corresponding event is sent.

If the existing state is either waitingForData or ready, the operator sends a corresponding event. The lock that prevents the PointMapMatcher operator from reading is unlocked only in the ready state.

For every geometry data tuple, the operator either stays or goes into the modifying state. The modifying state is left with a punctuation that indicates the end of the map data modification.

The only state for which the mutex that prevents the PointMapMatcher operator from reading is released, is the ready state.

The MapStore operator sends events only if the TupleType() and Event() custom output functions are used in the output clause.

Fault handling and exceptions

During startup the MapStore operator tries to create (or attach to) all needed resources. In this phase the following errors can happen and are handled as follows:

  • The shared map objects provided by the ST library cannot be opened. The operator throws an SPL::SPLRuntimeOperator exception in these cases.

  • The shared control segment exists but is corrupted. The MapStore operator tries to recreate the control segment.

  • The control segment cannot be removed or created, maybe caused by permission problems on the operating system level. The operator logs an error message and aborts in these cases.

During modifications of the shared map data the following errors can happen and are handled as follows:

  • The write lock cannot be acquired, either due to an error in the lock function, or due to a timeout. The internal timeout is set to 5 seconds, this is enough time for all readers to finish their current queries and release the lock. If the MapStore operator cannot get the lock, and the fault handling is strict the operator aborts. If fault handling is permissive the operator tries to recover the lock by forcing out readers in a loop. If this is not possible, the operator aborts.

  • If one of the modification commands or data types is invalid, the operator throws an SPL::SPLRuntimeOperator exception.

In the unlikely case that the error situation cannot be remedied by the operator, you have to manually perform a clean restart of the involved applications. See the troubleshooting section in the PointMapMatcher operator for instructions.

In case the MapStore operator is started twice with the same storeName, the first MapStore will create the control segment (because it does not exist), and will store its own process id (PID) in the segment. The second MapStore will open the control segment (because it already exists) and read the PID from the segment. It will check if a process with this PID is still running on the host, and refuse to start if this is the case. If the process is not running or the status cannot be determined, it will take ownership of the segment.

Behavior in a consistent region and checkpointing

The MapStore operator can be neither an operator within the reachability graph of a consistent region, nor the start of a consistent region.

The operator supports neither periodic, nor operator-driven checkpointing.

Summary

Ports
This operator has 1 input port and 1 output port.
Windowing
This operator does not accept any windowing configurations.
Parameters
This operator supports 12 parameters.

Required: storeName, storeSize

Optional: endNodeId, faultHandling, isOneWay, latitude, longitude, objectId, removeSharedMapStoreOnShutdown, startNodeId, updateFlags, updateType

Metrics
This operator does not report any metrics.

Properties

Implementation
C++
Threading
Always - Operator always provides a single threaded execution context.

Input Ports

Ports (0)

Receives geometry data for the edge network.

The following parameters are used to specify the expressions that are used to get the geometry data, for example, from the input tuple:

  • updateType

    Specifies whether the received tuple holds node or edge data.

  • updateFlags

    Specifies the action that is performed. Valid actions are: add, remove, and clear.

  • objectId

    Specifies the node/edge id, which is used to add or remove nodes and edges.

  • latitude

    Specifies the latitude of a coordinate that is used while adding a new node.

  • longitude

    Specifies the longitude of a coordinate that is used while adding a new node.

  • startNodeId

    Specifies the node id that is the start of an edge. This value is used while adding or removing edges.

  • endNodeId

    Specifies the node id that is the end of an edge. This value is used while adding or removing edges.

  • isOneWay

    Specifies whether the edge is going from start to end or in either direction. This value is used while adding or removing edges.

If data is invalid or problems occur while performing the required action, the tuple is ignored, and an error tuple is sent to output port 0.

A window punctuation indicates the end of a geometry update, which means that the edge network is recalculated after receiving a window punctuation.

The CLEAR action clears the map immediately and resets the error indicator for the shared map mode. This means that errors that occurred before the CLEAR action are ignored.

Hint: Nodes that are referenced by edges must be added before the referencing edge is added, otherwise the tuple that adds the edge will be considered invalid.

Properties

Output Ports

Assignments
This operator allows any SPL expression of the correct type to be assigned to output attributes. Attributes not assigned in the output clause will be automatically assigned from the attributes of the input ports that have the same name and type. If there is no such input attribute, an error is reported at compile-time.
Output Functions
ResponseFunctions
<any T> T AsIs(T)

Returns the parameter value.

MapStoreTypes.TupleTypes TupleType()

Returns event in case of a state change or failure in case that a failure occurred. See the Event() function for more details about the supported events.

Either you use the TupleType() and the Event() functions together, or none is used. This means, you cannot use only one of these functions in your output clause.

MapStoreTypes.Events Event()

Returns details about occurring events. See also the MapStoreTypes.Events type. Supported events are:

  • started

    This event is emitted in any case after startup of the MapStore operator. Normally this happens only once during the lifetime of the SPL application. If this event is send multiple times it means the operator was restarted due to some error or manually triggered PE restart. The application can use this event to react on the restart and take measures to remedy the problem, for example, by aborting an active map load operation, or by clearing and loading the map data again. If this is not needed, the application can simply ignore this event. But, the next event, which is either waitingForData or ready, has to be taken into account, too.

  • waitingForData

    Indicates that the operator waits for map data and has, therefore, locked the mutex that prevents the PointMapMatcher operator from reading. You can use the OSMXMLGeometrySource operator to provide geometry data. This event occurs under the following conditions:

    • The MapStore operator created the shared map store.

    • The MapStore operator detected the existing shared map store during the startup phase, and the existing state was either waitingForData or modifying with faultHandling set to strict.

      The application can use this event to determine if it should take measures to correct and reload the map data, or to notify the administrator that loading failed.

    • A window punctuation is received but one or more erroneous geometry data tuples were processed (= failed modification) with faultHandling set to strict.

  • ready

    This event is send once the geometry data is loaded into the shared memory if either no errors occurred during the load process, or if the faultHandling parameter is set to permissive. The application can use this event to determine that loading was successful. After this event is send, PointMapMatcher operators can access the map data.

If TupleType() indicates a failure, the Event() function returns waitingForData.

Either you use the TupleType() and the Event() functions together, or none is used. This means, you cannot use only one of these functions in your output clause.

Ports (0)

Sends failure tuples in case of failures while modifying the store, or events in case of state changes.

The MapStore operator sends events only if the TupleType() and Event() custom output functions are used in the output clause.

Possible error scenarios are:

  • An internal error when adding or removing a node
  • An internal error when adding or removing an edge
  • An internal error when clearing the map

Properties

Parameters

This operator supports 12 parameters.

Required: storeName, storeSize

Optional: endNodeId, faultHandling, isOneWay, latitude, longitude, objectId, removeSharedMapStoreOnShutdown, startNodeId, updateFlags, updateType

endNodeId

Specifies the node id that is the end of an edge. This value is used while adding or removing edges. If not specified, the endNodeId input attribute is mandatory.

Properties

faultHandling

Specifies how to handle faults. Valid values are:

  • strict (default)

    A runtime problem like a failed node/edge insertion causes the operator to invalidate the received geometry data completely. As soon as the punctuation is received, the waitingForData event is generated and the lock that prevents the PointMapMatcher operator from reading is not released.

  • permissive

    A runtime problem results in runtime log entries, sent tuples indicating the problem per tuple, a dropped input tuple, and - as soon as the punctuation is received - the MapStore operator continues with the so far received valid data. The lock that prevents the PointMapMatcher operator from reading is released.

Properties

isOneWay

Specifies whether the edge is going from start to end or in either direction. This value is used while adding or removing edges. If not specified, the isOneWay input attribute is mandatory.

Properties

latitude

Specifies the latitude of a coordinate that is used while adding a new node. If not specified, the latitude input attribute is mandatory.

Properties

longitude

Specifies the longitude of a coordinate that is used while adding a new node. If not specified, the longitude input attribute is mandatory.

Properties

objectId

Specifies the node/edge id, which is used to add or remove nodes and edges. If not specified, the objectId input attribute is mandatory.

Properties

removeSharedMapStoreOnShutdown

Specifies whether the shared map store is kept or removed on application shutdown.

If the parameter is set to false, the shared map store is kept on application shutdown, which can be used to avoid the time-consuming reload of the map data. But, any manipulation of the shared memory segments outside of the application, for example, removing the map data segment but keeping the control segment, results in inconsistent states and unpredictable behavior.

If this parameter is set to true the operator removes the shared map store on application shutdown. On the next application start, the shared map store is recreated, which means it is empty and the application is supposed to load the map data again.

The default is false.

Properties

startNodeId

Specifies the node id that is the start of an edge. This value is used while adding or removing edges. If not specified, the startNodeId input attribute is mandatory.

Properties

storeName

Specifies the unique name for the store. Different MapStore operator instances must use different store names.

Properties

storeSize

Specifies the size (in bytes) for the store within the range of 1 GiB and 1 TiB. The specified size is the maximum allowed size of the store. The actual memory usage depends on the amount of data that has been inserted.

Hint: The average storage (node, line, and index) for a given roadnet line is approx. 2.4 KB. For example, the storage for 45M road segments is about 92 GB.

Attention: A changed store size takes effect only if an existing store is removed. Stop the application, switch to the /dev/shm directory, and remove the shared memory segment that has your store name and the corresponding control segment. You will see bad_alloc exceptions in the operator trace if the store size is too small.

Properties

updateFlags

Specifies the action that is performed. Valid actions are: add, remove, and clear. If not specified, the updateFlags input attribute is mandatory.

You can use the com.ibm.streams.geospatial::UpdateFlags type to specify the SPL attribute.

Properties

updateType

Specifies whether the received tuple holds node or edge data. If not specified, the updateType input attribute is mandatory.

You can use the com.ibm.streams.geospatial::UpdateType type to specify the SPL attribute.

Properties

Libraries

ST library
Library Name: st
Library Path: ../../impl/lib
Include Path: ../../impl/include/st
MapController
Library Path: ../../impl/lib/control
Include Path: ../../impl/include/control
PROJ4 library
Library Name: proj
Library Path: ../../impl/lib/external/proj4
Include Path: ../../impl/include/proj4