MQTT & IoT Protocols

How It Works

MQTT (Message Queuing Telemetry Transport) was designed in 1999 by Andy Stanford-Clark (IBM) and Arlen Nipper for connecting oil pipeline sensors over unreliable satellite links. That origin story reveals everything about its design philosophy: minimal overhead, unreliable networks, and devices that cannot afford to waste a single byte.

The protocol is built on three concepts: topics, publish/subscribe, and quality of service levels.

The Pub/Sub Model

Unlike HTTP's request/response pattern, MQTT uses publish/subscribe. Devices never talk directly to each other. Every message goes through a central broker, which routes messages based on topic subscriptions.

Publisher (Sensor) → MQTT Broker → Subscriber (Dashboard)
                                 → Subscriber (Alert System)
                                 → Subscriber (Database Logger)

This decoupling is powerful. The sensor does not know or care who is consuming its data. New subscribers — analytics pipelines, alerting systems, data lakes — can be added without changing anything on the device side.

Topic Hierarchy and Wildcards

Topics are hierarchical strings delimited by forward slashes:

building/floor3/room301/temperature
building/floor3/room301/humidity
building/floor3/room302/temperature
fleet/truck-42/gps/location
fleet/truck-42/engine/rpm

Two wildcard characters make subscriptions flexible:

• + (single level): building/floor3/+/temperature matches all rooms on floor 3.
• # (multi level): building/floor3/# matches everything on floor 3 — all rooms, all sensor types.

import paho.mqtt.client as mqtt

client = mqtt.Client()
client.connect("broker.example.com", 1883)

# Subscribe to all temperature sensors on floor 3
client.subscribe("building/floor3/+/temperature", qos=1)

# Subscribe to everything from truck 42
client.subscribe("fleet/truck-42/#", qos=0)

def on_message(client, userdata, msg):
    print(f"{msg.topic}: {msg.payload.decode()}")

client.on_message = on_message
client.loop_forever()

QoS Levels: The Core Trade-Off

MQTT's three QoS levels are its most important design decision. Each level trades reliability for overhead:

QoS 0 — At Most Once (Fire and Forget)

The publisher sends the message. The broker does not acknowledge. If the network drops the packet, the message is lost forever. This sounds dangerous, but it is perfect for high-frequency telemetry where missing one reading out of a thousand does not matter.

Overhead: 1 packet.

QoS 1 — At Least Once

The publisher sends the message and waits for a PUBACK from the broker. If no PUBACK arrives within a timeout, it resends. This guarantees delivery but may cause duplicates — the broker might have received the first message and the PUBACK got lost.

Overhead: 2 packets (PUBLISH + PUBACK).

QoS 2 — Exactly Once

A four-packet handshake ensures no duplication and no loss:

1. Publisher → Broker: PUBLISH
2. Broker → Publisher: PUBREC (received)
3. Publisher → Broker: PUBREL (release)
4. Broker → Publisher: PUBCOMP (complete)

This is expensive but essential for operations like financial transactions or actuator commands where duplicates cause real problems (opening a valve twice).

Overhead: 4 packets.

# QoS 0: Temperature reading every second — loss is acceptable
client.publish("sensors/temp", "22.5", qos=0)

# QoS 1: Alert that needs guaranteed delivery — duplicates are okay
client.publish("alerts/high-temp", "WARNING: 85°C", qos=1)

# QoS 2: Command to open a valve — must execute exactly once
client.publish("commands/valve-3/open", "1", qos=2)

Retained Messages and Last Will

These two features solve the most common IoT pain points: late joiners and crash detection.

Retained Messages

When a message is published with the retain flag set, the broker stores the last message for that topic. Any future subscriber immediately receives this stored message upon subscribing — no need to wait for the next publish cycle.

# Publish current state as retained
client.publish("device/thermostat/status", '{"temp": 22, "mode": "heating"}',
               qos=1, retain=True)

# A subscriber connecting 2 hours later immediately receives this message

Without retained messages, a new dashboard loading at 3 AM would show blank values until each sensor publishes again. With retained messages, it immediately shows the last known state.

Last Will and Testament (LWT)

When a client connects to the broker, it can register a "last will" message. If the client disconnects ungracefully (network loss, crash, power failure), the broker automatically publishes this message on the client's behalf.

client = mqtt.Client()

# Register LWT before connecting
client.will_set(
    topic="devices/sensor-42/status",
    payload="offline",
    qos=1,
    retain=True  # Combine with retain so the offline state persists
)

client.connect("broker.example.com", 1883)

# Publish online status after connecting
client.publish("devices/sensor-42/status", "online", qos=1, retain=True)

Now the monitoring system can subscribe to devices/+/status and always know which devices are online. If sensor-42 loses power, the broker publishes "offline" automatically. Combining LWT with retained messages means the offline status persists for any future subscriber.

Why Not Just Use HTTP?

This is the question every web developer asks. Here is the concrete comparison:

Aspect	HTTP	MQTT
Minimum overhead	~300 bytes (headers)	2 bytes
Connection model	Short-lived (or keep-alive)	Persistent
Direction	Client-initiated request/response	Bidirectional publish/subscribe
Push to device	Requires polling or SSE	Native (subscribe)
Offline handling	None	QoS 1/2 + clean_session=false queues messages
Battery impact	High (TLS handshake per request)	Low (one connection, tiny packets)

For a sensor sending a 50-byte temperature reading every 10 seconds over a cellular connection, HTTP would add 300+ bytes of headers per request, require a TLS handshake per connection (or keep-alive management), and provide no way to push commands back to the device without polling.

MQTT sends that same reading in 52 bytes (2-byte fixed header + topic + payload) over a single persistent connection that also receives commands via subscriptions. Over 24 hours, that sensor sends 8,640 readings — the bandwidth savings add up to meaningful battery life on constrained devices.

MQTT 5.0: What Changed

MQTT 5.0 (2019) added features the IoT community had been requesting for years:

• Reason codes: Every acknowledgment now includes a reason code (success, quota exceeded, topic alias invalid, etc.), making debugging dramatically easier.
• Shared subscriptions: Multiple subscribers on $share/group/topic get round-robin delivery — built-in load balancing without a custom solution.
• Topic aliases: Replace long topic strings with short integer aliases after the first message, reducing per-message overhead.
• Message expiry: Set a TTL on messages. Stale retained messages auto-delete instead of lingering forever.
• Request/response pattern: Optional correlation data and response topic headers enable request/response over MQTT without application-level workarounds.
• User properties: Key-value metadata on messages — think HTTP headers for MQTT.

CoAP: The HTTP-Like Alternative

For devices that need request/response semantics (REST-style APIs for IoT), CoAP (Constrained Application Protocol, RFC 7252) is the alternative. It runs over UDP, uses compact binary headers, and maps cleanly to HTTP methods (GET, PUT, POST, DELETE).

CoAP is better for resource-oriented interactions (read sensor value, configure device settings). MQTT is better for event-driven pub/sub (sensor pushes updates, multiple systems consume). Many IoT architectures use both — CoAP for device management and configuration, MQTT for telemetry and events.

The choice between MQTT and CoAP often comes down to: does the system push data (MQTT) or does something poll for data (CoAP)? For most IoT telemetry, the answer is push — which is why MQTT dominates.