DEPLOYING IOT AND REAL-TIME WEB APPS
WITH AWS MSK AND TENEFIT.CLOUD
Robin Zimmermann, Leonid Lukyanov
tenefit.cloud is a managed API Gateway that facilitates connecting mobile, web and IoT clients to Apache Kafka without proprietary SDKs, intermediary brokers or “Push” services. It's an easier, more manageable, and better-performing alternative to Kafka Connect and Kafka Proxies.
tenefit.cloud can be VPC peered with AWS MSK enabling a range of real-time web and IoT use cases at any scale with minimal DevOps burden.
In this post, we walk you through a sample IoT demo where you'll connect MQTT and web clients to MSK without writing a single line of code.
tenefit.cloud is built on top of a protocol transformation engine that enables it to directly bridge the Kafka protocol with a range of other protocols such as SSE (Server Sent Events), REST, MQTT and AMQP. As a result, it removes the need for architectural components such as Kafka Connect or a cluster of MQTT brokers. It has no proprietary SDKs and so leaves client applications untouched.
As a secure gateway solution, tenefit.cloud supports SSL/TLS encryption and offers built-in authorization via JWT + OAuth 2.0 support.
Deploying an MSK-based IoT Setup
There are five components to the demo setup:
- A Java-based MQTT client that’s simulating IoT temperature sensors.
- An AWS MSK sandbox with topics for publishing and subscribing to IoT sensor data.
- tenefit.cloud, which exposes the MSK IoT sensor topics over MQTT, SSE and REST endpoints.
- An MSK microservice that subscribes to, transforms and then publishes modified sensor data to a new topic.
- A sample web application that’s subscribed to the modified sensor data topic through tenefit.cloud.
These instructions also contain an optional section that demonstrates integrating MQTT clients with MSK via a physical MQTT broker and Kafka Connect. Prior to tenefit.cloud, this would have been one of the main options considered by architects.
If you want to test out tenefit.cloud with your own MSK instead of using the MSK sandbox, you’ll need to VPC Peer the two services. Instructions on how to do this are covered at the end.
These are the major steps in the procedure:
- Step 1: Configure tenefit.cloud with the MSK sandbox
- Step 2: Add tenefit.cloud SSE and REST endpoints
- Step 3: Start the temperature convertor microservice
- Step 4: Install the temperature sensor MQTT client
- Step 5: Open demo browser portal and test the SSE and REST endpoints
- (Optional) Step 6: Use your own Amazon MSK broker cluster
- (Optional) Step 7: Compare with Kafka Connect
Parts of this demo will build and run Java projects, so you will need Java 1.8 or higher, and Maven.
Step 1: Configure tenefit.cloud with the MSK sandbox
- Open the tenefit.cloud console in a browser: https://console.tenefit.cloud/
- Login, or sign up for free if you don't already have an account.
- Click the Add a new app button:
- Add a new app dialog, select the Amazon MSK sandbox option.
Name of your broker field, such as "msk", then click Test connection.
- Once the setup steps are complete, press Done to return to the main screen.
- Make a note of the Kafka broker address as it will be used later. In these steps, it is using the TLS/SSL protocol and the address is msk.demo.tenefit.cloud:9093:
Step 2: Generating tenefit.cloud SSE and REST endpoints
The new configuration has no endpoints, so let's add the endpoints needed by the application.
Click the Add a new endpoint icon (the + icon):
Use the table below to add all of the endpoints required for the demo. Taking the first row as an example, when you click the + icon to add a new endpoint, choose MQTT over TLS from the select list. In the field after the domain, type /sensors/$key. From the Topic selection list, choose sensors. Then click the checkmark () icon to save the endpoint:
Repeat for each row in the table.
|MQTT over TLS||/sensors/$key||sensors|
|MQTT over TLS||/control/$key||control|
|MQTT over TLS||/state/$key||state|
The final result will look like this:
At this point, the endpoints listed are ready to be used by applications, which we'll see shortly.
Make a note of the tenefit.cloud domain you were allocated, you will need it in the next step:
Step 3: Install the temperature sensor MQTT client
This microservice subscribes to the sensors topic, transforms the messages so they are suitable for the client GUI application, and publishes them to the readings topic.
- In a terminal window, fork or clone the microservice-temperature-converter.java GitHub repo. For example:
$ cd microservice-temperature-converter.java
- Build it:
$ ./mvnw clean install
- Start the microservice and leave it running:
$ java -jar target/microservice-temperature-converter-develop-SNAPSHOT.jar -b msk.demo.tenefit.cloud:9093
Notice that the broker address is msk.demo.tenefit.cloud:9093, which you got from Step 1, earlier.
Also note that the two algorithm parameters deliberately have nothing after the equals (=).
At this point, the microservice is listening for messages on the sensors topic to convert them, but nothing is publishing to that topic yet.
Step 4: Install the temperature sensor MQTT client
The sensor client, written in Java, simulates an IoT device that produces temperature readings and communicates using MQTT.
- In a terminal window, fork or clone the thing-temperature-sensor.java GitHub repo. For example:
$ cd thing-temperature-sensor.java
- Build it:
$ ./mvnw clean install
- Start the sensors. The following command will start 10 sensors distributed across 3 rows:
$ java -jar target/thing-temperature-sensor-develop-SNAPSHOT.jar \
-b mqtts://fwxwm5sg.streaming.tenefit.cloud \
--sensors 10 \
Replace fwxwm5sg.streaming.tenefit.cloud with the domain you noted from tenefit.cloud console at the end of Step 2.
At this point, there are 10 simulated IoT sensors publishing their temperature readings using MQTT, but instead of connecting to an MQTT broker, they are connected to the MQTT endpoints you defined in tenefit.cloud back in Step 3. This is transparent to the IoT sensors, who feel like they're talking to an MQTT broker.
tenefit.cloud publishes those temperature readings to Tenefit’s sandbox MSK that contains a sensors topic, which the microservice is subscribed to. The microservice transforms the messages and publishes them to the readings topic, which is used by the application in the next step.
Step 5: Open demo browser portal and test the SSE and REST endpoints
Open the following URL in a browser:
Replace fwxwm5sg.streaming.tenefit.cloud with the domain you noted from tenefit.cloud console in Step 2.
You will see the sensor portal GUI showing you temperature updates for each sensor. This data is coming from the SSE endpoints in tenefit.cloud that you configured in Step 2.
In the lower right corner, select a different temperature unit, like Celsius, and click Submit. This issues a REST request to one of the REST endpoints in tenefit.cloud that you created in Step 2. That REST request is published to a topic subscribed to by the microservice. In response to that request, the microservice transforms messages into the preferred temperature unit. Temperatures in the portal GUI will then change to the new unit.
You can also click on an individual sensor and turn its state on and off. The GUI does this with a REST request to tenefit.cloud, which publishes a message to a topic that is subscribed to by the sensor client, also through tenefit.cloud. Thus, the frontend application is interacting with the IoT sensor through tenefit.cloud.
(Optional) Step 6: Use your own Amazon MSK broker cluster
The instructions above used a sandbox MSK provided by tenefit.cloud. You can use your own MSK after your MSK VPC is peered with your dedicated teneift.cloud VPC.
Back in the tenefit.cloud console, note the email address you associated with your tenefit.cloud cloud account – shown at #1 in the screenshot below – then click the AWS MSK Peering button.
In the VPC peering form, enter the email address associated with tenefit.cloud, and the details for your MSK VPC:
When you submit the form, your dedicated tenefit.cloud VPC will initiate peering to your MSK VPC. This may take up to 24 hours, but will usually be faster. You'll receive an email when the request has been initiated, or you can monitor your own Amazon VPC console, in the Peering Connections screen and see the request arrive.
Approve the request. While optional, it is a good idea to give the peering connection in the Name column, such as tenefit.cloud-vpc.
Now that your VPC peering connection is active, you must add an entry to your VPC route table to enable traffic to be directed to the peer VPC. The Amazon documentation has a simple set of steps to do that: Updating your Route tables for a VPC peering connection.
Secondly, you will need to update your MSK security group to allow incoming traffic to port 9092 from the tenefit.cloud VPC CIDR. For example:
With the peering and routing complete, you can now use your MSK from the tenefit.cloud console.
Click the Add a new app button and click Select in the Bring your own Kafka box. Then fill in the details (see screenshot below).
- Use any name you like for Name of your broker
- In the Protocol field, choose plaintext or ssl based on whether you configured your MSK for clients to connect in plaintext or using TLS.
- Put one of the hostnames in the Kafka cluster domain name field. The other brokers will be discovered automatically.
- Enter the port you configured for your MSK brokers in the Kafka cluster port field. Typically it is 9092 for plaintext and 9093 for SSL/TLS.
Press the Test connection to create your configuration. Once it's done, you can now start adding endpoints for topics in your broker cluster:
Add an SSE endpoint, choose a topic, and enter a path, then save the endpoint:
Once the configuration is live, open the live preview. If data is being published to your Kafka topic, you will see it here:
To learn more about using tenefit.cloud, and how to incorporate endpoints into your applications, check out the tenefit.cloud tutorials.
(Optional) Step 7: Compare with Kafka Connect
Thanks to tenefit.cloud's cross-protocol translation, it is easy for IoT devices, say, to publish to tenefit.cloud using their native MQTT protocol, and have the data published into Kafka. Without tenefit.cloud, you'd need a more heavyweight solution like Kafka Connect in conjunction with standing up a cluster of MQTT brokers just to pass data through.
If you'd like to try those steps to get a sense of what is involved for comparison, they are available separately in the Sensors end-to-end demo instructions.
This post showed you how to connect IoT and Web clients to AWS MSK with tenefit.cloud, and how such an approach is faster and easier than the traditional one that relies on independent MQTT brokers and Kafka Connect.
With tenefit.cloud, you can save time, resources and effort when deploying Kafka-based IoT and real-time web use cases.