KafkaProducer node
Use the KafkaProducer node to connect to the Apache Kafka messaging system, and to publish messages from a message flow to a topic on a Kafka server.
- Developer
- Application Integration Suite
- Standard
- Advanced
- Express
- Scale
- Adapter
This topic contains the following sections:
Purpose
You can use a KafkaProducer node in a message flow to publish an output message from a message flow to a specified topic on a Kafka server. You set properties on the KafkaProducer node to define how it will connect to the Kafka messaging system, and to specify the topic to which messages are sent.
For more information, see Processing Kafka messages.
The KafkaProducer node handles messages in the following message domains:
- DFDL
- XMLNSC
- JSON
- BLOB
- MIME
- XMLNS
- MRM
The KafkaProducer node is contained in the Kafka drawer of the palette, and is represented in the IBM Integration Toolkit by the following icon:
Using the KafkaProducer node in a message flow
Use the KafkaProducer node to publish messages that are generated from within your message flow to a topic that is hosted on a Kafka server. The published messages are then delivered by the Kafka server to all topic subscribers (consumers).
The KafkaProducer node publishes messages non-transactionally to the Kafka server, and they are available to be read by consuming applications as soon as they are published. As the publish operation is non-transactional, if the flow is rolled back to the input node after the message has passed through the KafkaProducer node, the publication of the message to the Kafka server is not rolled back.
You can use the Acks property on the KafkaProducer node to configure synchronous processing of the message, by specifying that the KafkaProducer node must wait for confirmation that the message has been successfully received by the Kafka server before continuing in the flow.
For more information about how to use the node, see Producing messages on Kafka topics.
For information about the supported versions of Kafka, see IBM Integration Bus system requirements. For more information about Kafka version compatibility, see the Apache Kafka documentation.
Terminals and properties
The KafkaProducer node terminals are described in the following table.
| Terminal | Description |
|---|---|
| In | The input terminal that accepts a message for processing by the node. |
| Failure | The output terminal to which the message is routed if a failure is detected during processing in the node. |
| Out | The output terminal to which the message is routed if it is successfully published to the Kafka topic. |
The following tables describe the node properties. The column headed M indicates whether the property is mandatory (marked with an asterisk on the panel if you must enter a value when no default is defined); the column headed C indicates whether the property is configurable (you can change the value when you add the message flow to the BAR file to deploy it).
The KafkaProducer node Description properties are described in the following table.
| Property | M | C | Default | Description |
|---|---|---|---|---|
| Node name | No | No | The node type, KafkaProducer | The name of the node. |
| Short description | No | No | A brief description of the node. | |
| Long description | No | No | Text that describes the purpose of the node in the message flow. |
The KafkaProducer node Basic properties are described in the following table.
| Property | M | C | Default | Description | mqsiapplybaroverride command property |
|---|---|---|---|---|---|
| Topic name | Yes | Yes | The name of the Kafka topic where the message will be published. Only one topic name can be specified. The name can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). | topicName | |
| Bootstrap servers | Yes | Yes | A list of host/port pairs (separated by spaces) to use for establishing the initial connection to the Kafka cluster. | bootstrapServers | |
| Client ID | No | Yes | The client name to be used when connecting to Kafka. This value can be up to 255 characters in length, and can include the following characters: a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). | clientId | |
| Add IIB suffix to client ID | No | Yes | Yes | If selected, this property suffixes the provided
client ID with the following string: |
useClientIdSuffix |
| Acks | No | Yes | 0 | The number of acks (acknowledgements)
to request from the server before the publication request is sent.
Possible values are:
If this property is set to 0, the KafkaProducer node does not wait for any acknowledgement that the publish request has been processed by the Kafka server. This is equivalent to a 'fire and forget' mode of operation. If this property is set to 1, the KafkaProducer node waits for a single acknowledgement from the Kafka server. If this property is set to All, the KafkaProducer node waits for acknowledgements from all replicas of the topic. This option provides the strongest available guarantee that the record was received. |
acks |
| Timeout (sec) | No | Yes | 60 | The time (in seconds) to wait for the request to complete. | timeout |
The KafkaProducer node Security properties are described in the following table.
| Property | M | C | Default | Description | mqsiapplybaroverride command property |
|---|---|---|---|---|---|
| Security protocol | Yes | Yes | PLAINTEXT | The protocol to be used when communicating with integration nodes. Valid
values are:
If you are using IBM Event Streams, this property must be set to SASL_SSL. If either SASL_PLAINTEXT or SASL_SSL is selected, you must configure the user ID and password that will be used to
authenticate with the Kafka server, by running the mqsisetdbparms command with a DataSource name of
|
securityProtocol |
| SSL protocol | Yes | Yes | TLSv1_2 | The SSL protocol that is used when the selected
value for the Security protocol property
is either SSL or SASL_SSL. You can select one
of the following values from the editable list, or you can specify
an alternative value:
|
sslProtocol |
Alternatively, to configure the 
The KafkaProducer node Validation properties are described in the following table.
| Property | M | Default | Description |
|---|---|---|---|
| Validate | No | Inherit | This property controls whether validation takes place. Valid values are None, Content and Value, Content, and Inherit. |
| Failure action | No | Exception | This property controls what happens if validation fails. You can set this property only if you set Validate to Content or Content and Value. Valid values are User Trace, Local Error Log, Exception, and Exception List. |
For a full description of these properties, see Validation properties.
| Property | M | C | Default | Description |
|---|---|---|---|---|
| Events | No | No | None | Events that you have defined for the node are
displayed on this tab. By default, no monitoring events are defined
on any node in a message flow. Use Add, Edit,
and Delete to create, change or delete monitoring
events for the node; see Configuring monitoring event sources by using monitoring properties for details. You can enable and disable events that are shown here by selecting or clearing the Enabled check box. |
Local environment overrides
You can dynamically override set values in the local environment in the same way as setting values in other elements of a message. For more information, see Using local environment variables with Kafka nodes.