Getting started with Azure IoT Operations on Ubuntu

Introduction

With the recent announcement of the release of Azure IoT Operations, Microsoft has provided its customers with a unified data plane offering significant improvements in node data capture, edge-based telemetry processing, and cloud-ingress. 

Combining Azure IoT Operations with Ubuntu provides the perfect pairing to build secure and robust solutions straight out-of-the-box. 

This blog is a step-by-step guide to getting started with Microsoft’s Azure IoT Operations. By the end, you will have deployed the Azure IoT Operations Services to a local Azure Arc-enabled K3s Kubernetes cluster and configured secure communication with the cluster’s MQTT broker using X509 certificate authentication. To achieve this you will be executing commands, creating and editing files and issuing self-signed certificates in order to simulate the actions of a smart fridge providing monitoring information to the cloud.

In this tutorial we will be configuring three components. Firstly your edge node, then an Ubuntu Core device acting as a leaf node and data generator, and finally a cloud-based environment to visualise the data.

You will require a Microsoft Azure account to complete this tutorial. Let’s get started.  

Edge Node

1. Create your Multipass instance

Our first step is to create an environment in which to run our edge node. We will be using Multipass – a lightweight vm manager from Canonical designed to quickly launch and manage Ubuntu images on your local machine. We will use this to create an Ubuntu 22.04 VM for our edge node, and later a core device to operate as a leaf node. 

For our edge node, we will need to increase the default disk, memory and cpu sizes to make them more realistic – in this case 20Gb disk, 6GB memory, and 2 CPUs. 

Once launched, we will then enter our VM with the shell command:

$ multipass launch 22.04 --disk 20G --memory 8G --cpus 2 
Launched: cleansing-guanaco
$ multipass shell cleansing-guanaco

2. Install MicroK8s

Now we are working inside our edge node VM, we will install Microk8s – Canonical’s ultra-lightweight Kubernetes implementation. We will use this to run our Azure IoT Operations cluster.

$ snap install microk8s –-classic 

This install will automatically create a cluster for you with the default name microk8s-cluster.

To avoid having to use sudo with Microk8s, we will create a user group and then create a kubectl configuration file that can be picked up by Azure.

Note: to avoid being asked to use sudo for snap commands, you can login to the snap store with “snap login”. You will be asked for your credentials but this is not strictly required if you are happy to continue using sudo.

sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube

mkdir -p ~/.kube
microk8s config > ~/.kube/config

3. Deploy Azure IoT Operations

For our next step we will be deploying an Azure IoT Operations cluster. To do this we will need to install the Azure CLI. This can be achieved using the following command:

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

At this point, we will be following the Microsoft quickstart instructions to Deploy Azure IoT Operations on your local cluster. For this tutorial, you only need to follow the first page of the quickstart guide and do not need to progress onto the next step  (Add OPC-UA assets to your Azure cluster).

As we will be using our own cluster rather than GitHub codespaces, we need to set some environment variables before we start.

SUBSCRIPTION_ID="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # taken from your azure account, can be found under Subscriptions on Azure portal

LOCATION="westeurope" # must be a valid azure location with support for Azure IoT Operations which can be found here, list locations with `az account list-locations -o table`

RESOURCE_GROUP="azure_iot_operations_quickstart" # name for the resource group under which all azure resources will be created

CLUSTER_NAME="my-local-cluster" # name of your local cluster, replace with something unique to avoid name collisions

You may need to choose a unique CLUSTER_NAME to avoid name collisions, since it is being used in the creation of some Azure resources (e.g. Key Vault) that have a global namespace. Details on how to rename a Microk8s cluster can be found here.

Publish/subscribe to MQTT from our device with certificate authentication

In order to authenticate devices with our cluster we will need to generate some X509 certificates. As we are self signing, these can be created on your cluster machine, although this is not recommended for production use.

1. Install step-cli

You’ll use step-cli to create the certificates. Install it following the instructions here.

2. Create the required self-signed X509 certificates

Create a self-signed root CA certificate and private key.

step certificate create --profile root-ca "My Root CA" root_ca.crt root_ca.key

Create an intermediate CA certificate signed by the root certificate.

step certificate create --profile intermediate-ca "My Intermediate CA" intermediate_ca.crt intermediate_ca.key --ca root_ca.crt --ca-key root_ca.key

Create a leaf (client) certificate signed by the intermediate certificate. The –bundle flag bundles the intermediate certificate together with the client certificate. This is important to complete the CA chain when presenting the certificate to the MQTT broker, since the root certificate is the only certificate that will be imported to the Azure IoT Operations cluster.

step certificate create --profile leaf "Client A" client_a_intermediate_bundle.crt client_a_intermediate_bundle.key --ca intermediate_ca.crt --ca-key intermediate_ca.key --bundle

3. Configure certificate authentication

Import the root CA certificate as configmap under the “client-ca” key.

microk8s kubectl create configmap client-ca --from-file=root_ca.crt -n azure-iot-operations

Create an x509Attributes.toml file with the following contents. It is used for defining certificate-to-attribute mapping for authorization purposes. The content is dummy because we’re only concerned with authentication, but the file is required.

[root]
subject = "CN = Contoso Root CA Cert, OU = Engineering, C = US"

[root.attributes]
organization = "contoso"

[intermediate]
subject = "CN = Contoso Intermediate CA"

[intermediate.attributes]
city = "seattle"
foo = "bar"

[smart-fan]
subject = "CN = smart-fan"

[smart-fan.attributes]
building = "17"

Import the x509Attributes.toml file as a Kubernetes secret under the “x509-attributes” key.

microk8s kubectl create secret generic x509-attributes --from-file=x509Attributes.toml -n azure-iot-operations

Create a patch-authn-x509.json file with the following contents. It  will be used to add client certificates as the first authentication method referencing the imported root CA certificate under the “client-ca” key.

[
    {
        "op": "add",
        "path": "/spec/authenticationMethods/0",
        "value": {
            "method": "x509",
            "x509Settings": {
                "trustedClientCaCert": "client-ca"
            }
        }
    }
]

Apply the patch on the authn brokerauthentication.

microk8s kubectl patch brokerauthentication default -n azure-iot-operations --type json --patch "$(cat patch-authn-x509.json)"

4. Expose the MQTT broker endpoint to the outside world

Define a patch-listener-loadbalancer-ip-dns.json file with the following contents. It will be used to change the type of listener to loadbalancer and also define an ip and dns SAN (Service Alternative Name) for secure communication from outside the cluster.

[
    {
        "op": "replace",
        "path": "/spec/serviceType",
        "value": "loadBalancer"
    },
    {
        "op": "add",
        "path": "/spec/ports/0/tls/certManagerCertificateSpec/san",
        "value": {
            "dns": [
                "localhost"
            ],
            "ip": [
                "127.0.0.1"
            ]
        }
    }
]

Apply the patch on the listener brokerlistener.

microk8s kubectl patch brokerlistener default -n azure-iot-operations --type json --patch "$(cat patch-listener-loadbalancer-ip-dns.json)"

Check the assigned external ip and port on the MQTT frontend service. The patch might take a few moments to be applied.

microk8s kubectl get service aio-broker --namespace azure-iot-operations

5. Publish to MQTT with Demo Client snap

Create an Ubuntu Core Multipass instance. Make sure it can connect to the Multipass server instance then install snap:

$ multipass launch core24 -n CoreDev24
$ multipass shell CoreDev24

Install Demo Client snap.

snap install --edge azure-iot-operations-mqtt-demo-client

Get locally the root CA certificate to communicate securely with the MQTT broker.

microk8s kubectl get configmap aio-ca-trust-bundle-test-only -n azure-iot-operations -o jsonpath='{.data.ca\.crt}' > ca.crt

Configure broker.host, broker.port, broker.ca, client.cert and client.key.

snap set azure-iot-operations-mqtt-demo-client broker.host=localhost broker.port=31698 broker.ca="$(cat ca.crt)" client.cert="$(cat client_a_intermediate_bundle.crt)" client.key="$(cat client_a_intermediate_bundle.key)" # replace 31698 with the correct port

(!) If you have previously set client.username and client.password, unset them to avoid any ambiguity regarding the authentication method used.

snap set azure-iot-operations-mqtt-demo-client client.username! client.password!

Start publishing messages.

azure-iot-operations-mqtt-demo-client.mqtt-producer

Send and visualise MQTT data on Microsoft Fabric

With your MQTT Client running and publishing data, we will now try and visualise this data using Microsoft Fabric.

1. Create an Azure Event Hub

Create an Azure Event Hubs Namespace.

az eventhubs namespace create --name ${CLUSTER_NAME:0:24} --resource-group $RESOURCE_GROUP --location $LOCATION

Create an Azure Event Hub where the data will be sent.

az eventhubs eventhub create --name smart-refrigerator --resource-group $RESOURCE_GROUP --namespace-name ${CLUSTER_NAME:0:24} --retention-time 1 --partition-count 1 --cleanup-policy Delete

Get the name of the Azure IoT Operations Kubernetes extension on the cluster.

AIO_EXTENSION_NAME=$(az k8s-extension list --resource-group $RESOURCE_GROUP --cluster-name $CLUSTER_NAME --cluster-type connectedClusters -o tsv --query "[?extensionType=='microsoft.iotoperations'].name")

Define an event-hubs-config.bicep file that will be used to assign the roles of Event Hubs Data Receiver and Event Hubs Data Sender to the Azure IoT Operations Kubernetes extension with the following contents.

@description('Location for cloud resources')
param aioExtensionName string = 'azure-iot-operations'
param clusterName string = 'clusterName'
param eventHubNamespaceName string = 'default'

resource connectedCluster 'Microsoft.Kubernetes/connectedClusters@2021-10-01' existing = {
  name: clusterName
}

resource aioExtension 'Microsoft.KubernetesConfiguration/extensions@2022-11-01' existing = {
  name: aioExtensionName
  scope: connectedCluster
}

resource ehNamespace 'Microsoft.EventHub/namespaces@2021-11-01' existing = {
  name: eventHubNamespaceName
}

// Role assignment for Event Hubs Data Receiver role
resource roleAssignmentDataReceiver 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(ehNamespace.id, aioExtension.id, '7f951dda-4ed3-4680-a7ca-43fe172d538d')
  scope: ehNamespace
  properties: {
     // ID for Event Hubs Data Receiver role is a638d3c7-ab3a-418d-83e6-5f17a39d4fde
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', 'a638d3c7-ab3a-418d-83e6-5f17a39d4fde') 
    principalId: aioExtension.identity.principalId
    principalType: 'ServicePrincipal'
  }
}

// Role assignment for Event Hubs Data Sender role
resource roleAssignmentDataSender 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(ehNamespace.id, aioExtension.id, '69b88ce2-a752-421f-bd8b-e230189e1d63')
  scope: ehNamespace
  properties: {
    // ID for Event Hubs Data Sender role is 2b629674-e913-4c01-ae53-ef4638d8f975
    roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', '2b629674-e913-4c01-ae53-ef4638d8f975') 
    principalId: aioExtension.identity.principalId
    principalType: 'ServicePrincipal'
  }
}

Apply event-hubs-config.bicep to assign the roles.

az deployment group create --name assign-RBAC-roles --resource-group $RESOURCE_GROUP --template-file event-hubs-config.bicep --parameters aioExtensionName=$AIO_EXTENSION_NAME --parameters clusterName=$CLUSTER_NAME --parameters eventHubNamespaceName=${CLUSTER_NAME:0:24}

2. Create a dataflow

Define a dataflow.yml file that will be used to create a dataflow from the cluster’s MQTT broker to the Azure Event Hub with the following contents.

apiVersion: connectivity.iotoperations.azure.com/v1beta1
kind: DataflowEndpoint
metadata:
  name: mqtt-source
  namespace: azure-iot-operations
spec:
  endpointType: Mqtt
  mqttSettings:
    host: "aio-broker:18883"
    tls:
      mode: Enabled
      trustedCaCertificateConfigMapRef: azure-iot-operations-aio-ca-trust-bundle
    authentication:
      method: ServiceAccountToken
      serviceAccountTokenSettings:
        audience: aio-internal
---
apiVersion: connectivity.iotoperations.azure.com/v1beta1
kind: DataflowEndpoint
metadata:
  name: kafka-target
  namespace: azure-iot-operations
spec:
  endpointType: Kafka
  kafkaSettings:
    host: '<NAMESPACE>.servicebus.windows.net:9093'
    batching:
      latencyMs: 0
      maxMessages: 100
    tls:
      mode: Enabled
    authentication:
      method: SystemAssignedManagedIdentity
      systemAssignedManagedIdentitySettings:
        audience: https://<NAMESPACE>.servicebus.windows.net
---
apiVersion: connectivity.iotoperations.azure.com/v1beta1
kind: Dataflow
metadata:
  name: mqtt-to-eh
  namespace: azure-iot-operations
spec:
  profileRef: example-dataflow
  operations:
    - operationType: source
      sourceSettings:
        endpointRef: mqtt-source
        dataSources:
        - smart-refrigerator
    - operationType: destination
      destinationSettings:
        endpointRef: kafka-target
        dataDestination: smart-refrigerator
---
apiVersion: connectivity.iotoperations.azure.com/v1beta1
kind: DataflowProfile
metadata:
  name: example-dataflow
  namespace: azure-iot-operations
spec:
  instanceCount: 1
  diagnostics:
    logs:
      level: "debug"

Replace the <NAMESPACE> placeholder with the Event Hubs Namespace name and save it as a new my_dataflow.yml file.

sed 's/<NAMESPACE>/'"${CLUSTER_NAME:0:24}"'/' dataflow.yaml > my_dataflow.yaml

Apply the my_dataflow.yml file to create the dataflow.

microk8s kubectl apply -f my_dataflow.yaml

Verify that data is flowing to the Azure Event Hub, by going to Azure Portal and navigating to Event Hubs Namespace > Event Hub.

3. Create a Microsoft Fabric dashboard

Follow the quickstart instructions Get insights from your data to create a Microsoft Fabric dashboard for the OPC UA data. You should make the following adjustments to the instructions:

1. Use the smart-refrigerator Event Hub instead of destinationeh where required.

2. Create a KQL table named MQTT instead of OPCUA with the following columns.

3. Use the following KQL query to create a mapping for the table and use the created mqtt_mapping mapping where required instead of the opcua_mapping.

.create table ['MQTT'] ingestion json mapping 'mqtt_mapping' '[{"column":"RefrigeratorTemp", "Properties":{"Path":"$.refrigerator_temp"}},{"column":"FreezerTemp", "Properties":{"Path":"$.freezer_temp"}},{"column":"Timestamp", "Properties":{"Path":"$.timestamp"}}]'

4. Use the following KQL query for the dashboard.

MQTT 
| where Timestamp between (_startTime.._endTime)
| project Timestamp, RefrigeratorTemp, FreezerTemp

5. Use the following characteristics for the visual of the dashboard:

• Tile name: Refrigerator and Freezer Temperature
• Visual type: Line chart
• Data:
  • Y columns: RefrigeratorTemp (decimal) and FreezerTemp (decimal) (already inferred by default)
  • X columns: Timestamp (datetime) (already inferred by default)
• Y Axis:
  • Label: Temperature in C
• X Axis:
  • Label: Timestamp

This is what the dashboard should look like after completing the above steps.

Wrapping up

So, by now you should have a fully functioning Azure IoT Operations deployment simulating a smart refrigerator and sending data over MQTT up to the cloud where you can see it visualised in real time. From here you have a wealth of options available to you to enhance this data collection and aggregation. 

Building on this tutorial allows you to develop your own IoT solutions where data visualisation and real time telemetry reporting will be key to success.

In order to learn more about what you can do with your edge node – have a look at the Microsoft documentation available here.

Alternatively, this is also an OPC-UA simulator available as a snap package for experimenting with if that is the protocol of choice in your environment. 

If you want to learn more about how Ubuntu Core is the ideal platform for your leaf nodes when you want to expand into production, more information is available here.