In last week’s blog Secure and Governed Microservices with HDF/HDP Kafka Streams Support, we walked through how to build microservices with the new Kafka Streams support in HDF 3.3 and HDP 3.1 that is fully integrated with Ranger, Schema Registry and other platform services. This blog is all about monitoring these microservices with Hortonworks Streams Messaging Manager (SMM).
In a microservices architecture, you will see a proliferation of stand-alone decoupled services. Hence, monitoring and managing these services becomes extremely critical. SMM provides users a powerful tool to monitor and visualize their microservices and understand how data flows across these services.
Working off the trucking fleet use case example from the previous blog, you can view each of the three microservices as a consumer and producer as depicted in the below diagram.
Lets first focus on monitoring the stream between MicroService 1 and MicroService 2 where MicroService 1 is a Kafka producer into the driver-violation-events topic and MicroServce 2 is a consumer from that topic. The below video showcases how to monitor the interactions between these two Kafka Streams microservices.
As the above video showcased, SMM cured the Kafka blindness for the streams app between the two microservices shedding light on some important information including the rate at which MicroService 1 was producing data, the lag of MicroService 2 and understanding the details of the internal Kafka changelog topics created by the Kafka streams join operator.
Another common use case for monitoring Kafka streams application using SMM is the following:
The below video showcases how SMM address this use case.
The addition of Kafka Streams to HDF 3.3 and HDP 3.1 integrated with platform services like Ranger, Schema Registry and other platform services provides app developers a comprehensive platform to build secure and governed microservices. With SMM, devops and platform operations teams have enterprise tools to debug, monitor, and troubleshoot these microservices built using Kafka Streams.
In the next installment of the Kafka Analytics blog series, we walk through the new Hive and Kafka integration for the SQL access pattern.