This article will help you, the integration flow developer, to get a better understanding of the Kafka sender adapter. You can learn how the Kafka sender adapter works with respect to exception handling upon polling. Please note, this is a complementary article to the Kafka article “Cloud Integration – What You Need to Know About the Kafka Adapter” which explains some important general considerations when using the Kafka adapter. Check it out!

The Road So far

When you deploy an integration flow which uses Kafka inbound adapter (sender adapter), the adapter starts to poll messages from the Kafka broker. If an exception occurs during the polling process (for example, because of invalid credential or because of temporary connectivity issues), the integration flow will be deployed, and its status will be set to “Started”.

As such exception occurs before any message exchange is created, there is no message processing log available. Therefore, the user is not aware of a potential misconfiguration in the integration flow or any temporary connectivity issues and will assume that the integration flow is configured properly; yet no messages are being consumed and no message processing log is generated. This is a confusing status and the users need to investigate the problem by looking at the log traces (if allowed to do) and do connectivity tests to find the potential problems.

Improvements

With Cloud Integration release in September 2021 we have provided an update to improve this problem. With this update, a new Kafka polling monitor is added to Kafka sender adapter. When an integration flow with Kafka sender endpoint is deployed, the polling status is also monitored and checked for any exceptions during polling. In case of an exception, it is propagated to the integration flow monitor and the deployment status is set to “Error”. The corresponding error details are then shown in the “Error Details” pane of the manage integration content page of Cloud Integration UI. The following screenshot shows the status of a simple integration flow using the Kafka sender adapter with invalid credential. The integration flow status is set to “Error” while the Error Details pane indicates that a SaslAuthenticationException has caused the error.

Deployment status in Error when polling exception occurs

The polling monitor is actively listening if any exception occurs during the polling. This exception might be a transient one and it could vanish or be resolved after a certain amount of time. In such a case, the deployment status of the integration flow would automatically go to “Started” and it would start consuming the messages properly. This means that if the polling monitor does not detect any exception for a certain amount of time, the deployment status would be set to “Started”. With this mechanism, we can ensure that whenever an exception occurs during the polling, the integration flow status is set to “Error” and if it does not occur again, the status would be set automatically to “Started”.

Specific Considerations

You have seen that the new Kafka polling monitor helps you to identify exceptions that can happen due to adapter misconfigurations as well as temporary connectivity issues after deployment. However, not all the misconfigurations may lead to an exception during polling, meaning not all of them are identified by polling monitor. For instance, if you set an invalid port number in the host address, no exception is thrown by polling. We have identified all these cases. In the following table you can see a summary of different misconfigurations and their coverage with kaka polling monitor. We will try to apply further improvements for them in the future.

Finally, as mentioned above, it is possible that some temporary connectivity issues e.g., network problems cause the deployment status to be in the “Error” state. In such a case, it is recommended to use the Kafka connectivity test to investigate the problem. This function is specifically added to do connectivity test against Kafka brokers and ensure that the brokers are reachable from Cloud Integration. You need to provide the same connection configuration and credential as in Kafka sender adapter to do the connectivity test. If the connectivity test is successful, it will provide a list of existing topics in the broker. For better performance, the number of shown topics is limited to 200 topics.

Kafka connectivity test

Conclusion

We have seen how kafka sender adapter works with respect to error handling while polling. We have learned what are the potential problems. I have then proposed a potential solution which we have implemented and is available since September 2021 in Cloud Integration. Additionally, I have introduced you to a recently added functionality, Kafka Connectivity Test, which can be used for checking connectivity issues. Finally, we have seen the state of the polling monitor after the proposed solution and further possible improvements in the future. As an integration designer it is important to have a deeper understanding how the polling in kafka adapter is exactly working with respect to error handling. Polling monitor helps to detect the potential problems which may occur during polling and therefore save the effort required to investigate such problems.

Any feedback from you would be appreciated. Please do not hesitate to ask, if there were any questions. You can also use this link to submit questions in the SAP Community.