Manage cookies

We use cookies to improve our services. Read more about how we use cookies and how you can refuse them.

The necessary cookies help to make the website work properly.

Anonymous statistical cookies help to understand how visitors use the website.

  • Home Automation, Raspberry Pi
    12 min | 128334

    #Home Assistant: DIY Smart Home - tips and tricks (SSL, HAProxy & Let's Encrypt)

    Home Automation, Raspberry Pi | 12 min | 128334


    In the last years, I wrote several articles about Home Assistant (HA) and the possibility to replace the Zigbee hubs from different companies in your home using Zigbee2MQTT. Some of those articles are listed here (don't visit the links, they are outdated):

    Those articles were very popular at that time but, they need to be updated!

    I'm using HA for about 3 years now and in this article:

    • I'll describe my setup and learnings that I got during this time;
    • I'll include the steps to add an SSL layer to the HA setup.

    A Dockerized service adds the SSL layer (reverse proxy) and automatically renews the SSL certificate when needed. This setup enables you to seamlessly connect your HA to Google Assistant without using any external server or service in the middle.

    HA is officially compatible with the following hardware:

    generic-x86-64 odroid-c2 odroid-n2 odroid-xu qemuarm qemuarm-64 qemux86 qemux86-64 raspberrypi raspberrypi2 raspberrypi3 raspberrypi4 raspberrypi3-64 raspberrypi4-64 tinker khadas-vim3

    and more options (e.g. orange-pi, rock-pi, and more.). I'm using a Raspberry Pi 4 for my setup, so the steps below will be focus on this architecture, but they can be easily ported to other architectures.

    HA installation

    If you are newbie with HA, you should know that you can install HA in four different ways. I prefer the Supervised way. But, if you don't have enough experience in Linux and Docker, you can go straight ahead with the OS way - meaning hass.io. Why do I prefer the Supervised? Well, because I still have access to Raspberry Pi OS and I can install or start other things or services (e.g. HAProxy - continue reading). This means, I lose the possible advantages of a managed OS but I gain flexibility.

    To install HA using the Supervised way, you need to get Raspberry Pi OS (as I said, I focus on the Raspberry Pi architecture) running and you need to install Docker. This can be done after connecting e.g. over SSH to the Raspberry Pi and by typing the following on a Terminal:

    curl -sSL https://get.docker.com | sh
    sudo usermod -aG docker pi
    sudo reboot

    After installing Docker, you should install the following requirements:

    sudo apt-get install -y jq wget curl udisks2 libglib2.0-bin network-manager dbus libpcap0.8 libdbus-glib-1-dev apparmor ppp

    and the latest OS-Agent following the steps described here: OS Agent Installation.

    The next step is to install the HA supervisor by typing:

    wget https://github.com/home-assistant/supervised-installer/releases/latest/download/homeassistant-supervised.deb
    dpkg -i homeassistant-supervised.deb

    If you are running Raspberry Pi OS with Kernel >= 5.10 (type uname -a to check that info). Add the following at the end of the line inside the /boot/cmdline.txt file and reboot the Raspberry Pi.

    lsm=apparmor

    this activates AppArmor support of the Kernel.

    Then, you will have HA running on http://[RASPBERRYPI-IP]:8123. Thus, point your browser to that address and start the configuration.

    Note: If you connect your Raspberry Pi to your router via Wi-Fi, due to the network-manager package, the MAC address of your Wi-Fi adapter is going to change on every reboot -I don't know why-. This causes the IP to change as well, if are using DHCP, which is the default option of the interface. I didn't find any solution to that problem (obviously a fix IP would help), so I connected the Raspberry Pi using a LAN cable to the router, and reserved the IP to the MAC address of the LAN network adapter of the Raspberry Pi. Update @Fotios thanks for the comment. The problem is related with the MAC address randomization of the network-manager package. Describe for Fedora in this article.

    Zigbee2MQTT Hardware

    A setup with a Raspberry Pi and a Zigbee dongle or a Home Assistant Yellow can replace multiple Zigbee hubs/gateways that you may have running on your home. Xiaomi, Bosch, Philips Hue, IKEA, Osram, Sonoff, and others can be replaced with this setup. More important is that you can combine the devices (sensors and actors) from different companies using automation rules, scenes, scripts, and more. You save electricity because you only need a Gateway (Raspberry Pi) and you don't have to buy everything from the same company, which can be sometimes very expensive. These Zigbee devices are compatible with this setup.

    For the setup, I used to use the cheap CC2531 Zigbee dongle as I mentioned here. You need to flash the chip using a Raspberry Pi or a CC-Debugger. However, the dongle is no longer recommended because of the outdated chip and the performance. Last year, SONOFF has presented the SONOFF Zigbee 3.0 USB Dongle Plus. This dongle uses a CC2652P, which is pre-flashed with the right firmware (as Zigbee coordinator), and it comes with an aluminum housing and an antenna, which increases the Zigbee range (gain up to 5dB with default firmware). Thus, for my actual setup, I've chosen the SONOFF Zigbee 3.0 USB Dongle Plus.

    Note 1: FYI - Zigbee 3.0 devices can be added to an Echo Plus, Echo Show 2, Echo Studio, Echo 4G. It's a different setup as with HA, but not a lot of people know that those Amazon Echos have a Zigbee hub integrated.

    Note 2: If you want to update the firmware of the SONOFF Zigbee 3.0 USB Dongle Plus, follow the instructions provided in this tutorial. You should update the firmware if you have Philips HUE lights that don't have a reset button.

    Note 3: WI-FI/WLAN devices are also compatible with HA. Sonoff/eWeLink or Tuya/SmartLife can be added to the setup using cloud integrations or as I will explain, local integrations.

    Note 4: ZHA is another Integration that you can install to connect to Zigbee sensors. I prefer Zigbee2Mqtt because it is more flexible, and support more devices. But, ZHA is easier to set up without additional software (MQTT).

    Firmware Upgrade (optional)

    As I said, the SONOFF Zigbee 3.0 USB Dongle Plus comes pre-flashed with the Z-Stack 3.x.0 coordinator firmware. But, if you need to use Touchlink for resetting devices this only works with firmware newer as Z-Stack_3.x.0_20220103. Thus, you need to upgrade the firmware otherwise you'll get the error: SRSP - AF - interPanCtl after 6000ms

    To do that, I used this tutorial and followed Method 2 and the Manual mode option. That means I opened the housing and pressed the boot button while I connect the dongle in one of my Windows computers. Yes, you need a computer running Windows because you will use the SmartRF(TM) Flash Programmer 2 (v1.8.2) that runs only on Windows. The latest firmware is available here, and for the CC2652P you need the hex file inside the CC1352P2_CC2652P_launchpad_*.zip file.

    Philips HUE LightStrip resetted using Touchlink
    Fig. 1: Philips HUE LightStrip resetted using Touchlink

    Zigbee2MQTT Installation

    To install Zigbee2MQTT on a running HA, you need to add the repository:

    https://github.com/zigbee2mqtt/hassio-zigbee2mqtt

    to the add-on repositories. This installation can be only made following these steps if you installed HA as Supervised installed or the OS/hass.io. These two options support add-ons. Thus, add the above repository URL in the ADD field located under:

    • Configuration (menu on the left) →
    • Add-ons, Backups & Supervisor (option in the middle panel) →
    • Add-on Store (blue button on the right corner) →
    • (top right corner) →
    • Repositories (dropdown menu) →
    • Paste the URL in the ADD field (overlay popup) and click on the ADD button.

    The repository includes two add-ons:

    • zigbee2mqtt is the stable release (recommended),
    • zigbee2mqtt-edge is the dev branch, so you can install the edge version if there are features or fixes in the dev branch that are not yet released.

    These two add-ons will be listed in the add-on list (ConfigurationAdd-ons, Backups & Supervisor). You need to install only one of them by clicking on its box and selecting install. I installed the stable version, but if you're willing to be a beta tester, select the dev option.

    Note: The readme description, does not reflect the git commit comments (edge v1.22.0-3, stable v.1.22.2-1).

    Zigbee2MQTT needs also an MQTT-Broker (thus the name) that can be installed separately (as a service) or activated as a HA add-on. In the add-on list, you can find the Mosquitto broker. Click on the box and install it.

    You should activate the Advanced Mode, it gives you more information (logs) to debug, and more options to check if you are configuring something wrong (e.g. a check for the configuration.yaml file).
    Go to http://[HA-IP]:8123/profile and turn on the Advanced Mode switch.
    Activate also the Multi-factor Authentication Modules especially if you are going to open the HA to the Internet. To do that, turn on the Authenticator app switch.

    Add-on Configurations

    Before you start the installed add-ons, you should configure them first.

    MQTT Broker

    To configure the MQTT add-on, go to:

    • Configuration (menu on the left) →
    • Add-ons, Backups & Supervisor (option in the middle panel) →
    • Click on the Mosquitto broker(add-on in the middle panel) →
    • Click on the Configuration tab (top of the page)

    and add the following inside the Options field:

    logins:
      - username: [user]
        password: [password]
    customize:
      active: false
      folder: mosquitto
    certfile: fullchain.pem
    keyfile: privkey.pem
    require_certificate: false
    
    mqtt:
      base_topic: zigbee2mqtt
      user: user
      password: password

    Change the options [user] and [password] to a secure combination and keep it safe, you'll need them for the next step. Then, click on the Save button.

    After configuring the Mosquitto broker add-on, you can start it by going back to the Info tab and clicking on the Start button.

    Zigbee2MQTT

    To configure the MQTT addon, go to:

    • Configuration (menu on the left) →
    • Add-ons, Backups & Supervisor (option in the middle) →
    • Click on the Zigbee2mqtt(add-on in the middle panel) →
    • Click on the Configuration tag (top of the page)

    and change the following inside the Options field:

    [...]
    permit_join: true
    mqtt:
      base_topic: zigbee2mqtt
      user: [user]
      password: [password]
    serial:
      port: /dev/ttyUSB0
    [...]

    The user and password under mqtt: field you should have the same value that you've written for the Mosquitto broker add-on. The port: /dev/ttyUSB0 config works for the SONOFF Zigbee 3.0 USB Dongle Plus, because it has a CP2102N. Other dongles connect using the /dev/ttyAMA0 port. The permit_join: true config allows every Zigbee device to join the network if requested. After you add your devices (sensors and actors), you should set this option to false. Finally, change the network_key to increase your network security. Do that before starting the add-on and adding some devices to your network. If you've already done that, you will need to pair all devices again if you change the network key :(.

    [...]
      network_key:
        - 1
        - 1
        - 1
        - 1
        - 1
        - 1
    [...]

    Again, after configuring the Zigbee2MQTT add-on, you can start it by going back to the Info tab and clicking on the Start button.

    You can use dmesg command on Linux hosts to find the Zigbee dongle. If possible you should use the /dev/serial/by-id/ path of the stick, instead of /dev/tty*. The path /dev/tty* can change. The /dev/serial/by-id/ path doesn't change.

    The Zigbee2MQTT add-on has a nice and simple UI to add and configure Zigbee devices. To open the UI go to the Info tab and click on the OPEN WEB UI. There, you check and update the Zigbee devices, look at the network, and more. There is a button/option (permit join (all)) you don't need to press it to add new devices if you set the permit_join: true option. Then, after you added all devices and set this option to false, you can add devices by pressing that button. A countdown appears, and you'll be able to add a new device during the countdown.

    Note 1: Every time you modify the configuration of a specific add-on (`Options` field), you need to restart so that your changes take effect.

    Note 2: Devices that are connected to an AC supply (wall switches, curtain controllers - both with neutral) work as routers. Thus, these are good to have to extend and stabilize your network.

    Pair Devices to Zigbee2MQTT

    You can pair devices to Zigbee2MQTT by pressing the pair button. It is different from device to device: how many times or if it's a long or a short press. The README booklet that comes with the device usually has that information. Check that, or simply press the button a couple of times. But, as I said, read the README booklet. Maybe you need to reset the device first (e.g. by Aqara sensors you can do that by pressing the reset button for 5 seconds).

    Example of a Zigbee Network
    Fig. 2: Example of a Zigbee Network

    Tuya, Sonoff, and Xiaomi BLE devices

    Using this setup, you can add devices from Tuya, Sonoff and Xiaomi BLE Sensors. I use the following integrations to do that:

    • Tuya Integration - (devices controlled using Smartlife app on your smartphone): for WiFi devices, the integration uses the cloud and the update rate for the attributes (e.g. current, voltage, and power consumption) is not reliable. I used to used it, but not anymore.
    • LocalTuya - (devices controlled using Smartlife app on your smartphone): this integration works local (via LAN) but, it is not officially supported. Thus, you need to install it using HACS and you need the local key of the devices. You can get those keys using Tuya IoT Cloud/API Explorer. A tutorial is available here. I use this integration and recommend it.
    • Sonoff LAN - (devices controlled eWeLink app on your smartphone): this integration works local, cloud, or hybrid (configurable) but, it is not officially supported. Thus, you need to install it using HACS. The devices are synchronized from the Cloud and after that, you can switch to local. I use this integration and recommend it.
    • Passive BLE Monitor - (sensors that use BLE and are integrated using Mi Home app): this integration works local (via BLE). To read the data from some sensors, you needs an encrytion key. Read the FAQ to get that key. Both methods (the Xiaomi Cloud Tokens Extractor and TelinkFlasher (WebBluetooth) worked for me. I use this integration and recommend it.

    HAProxy for HTTPS

    As you may have noticed, HA runs on port 8123 as an HTTP-based application. If you want to access your application from outside you should use HTTPS. HTTPS makes that all traffic between you and the HA is encrypted with a key that only the client (e.g. your smartphone's browser) and server (Raspberry Pi running HA) know.

    To get an SSL certificate:

    • Your HA should be exposed to the Internet. That means you need to forward the ports of the Raspberry Pi in your router.
    • You need to point a domain name to your IP address, and for that, you'll need a dynamic DNS.

    To do that easyly, I build a Docker image that you can deploy to add an SSL layer (TLS) to HA.

    Pull the Docker Image

    You can build the Docker image (check the files inside docker-stack) or you can pull it from the Dockerhub register:

    docker pull lemariva/haproxy-certbot:1.1.1-SNAPSHOT

    To start the service, docker-compose must be installed. Otherwise, you need to configure all the variables and settings that you can find inside the docker-compose.yml file.

    sudo apt-get install python3-pip
    pip3 install docker-compose

    Configure the Docker Service

    But, before starting the service, edit [SERVICE_IP] and [PORT_IP] variables inside the file docker-compose.yml. Both should point to the service that you are forwarding. In my case, that is:

    SERVICE_IP: 192.168.178.161
    SERVICE_PORT: 8123

    The IP 192.168.178.161 and port 8123 points to my Raspberry Pi address and the standard port of HA, respectively.

    Then, you can start the service by typing the following:

    cd orchestration
    docker-compose up -d

    Configure HA

    If you are running HA with version greater than v2021.7.0, you will get a "400 Bad Request" error when I tried to access HA via the HTTP/HTTPS address. A breaking change was added to this version and if you are running a proxy you need to add to the configuration.yaml the following:

    # proxy
    http:
      use_x_forwarded_for: true
      trusted_proxies:
        - [PROXY_IP]

    If your proxy is running on another machine, you need to change [PROXY_IP] with the IP address of that machine. But, if you are running the proxy on the same machine that HA is running, you need to change [PROXY_IP] with the Docker internal IP of the container. You can get that by typing the following:

    docker container inspect `docker ps -aqf "name=haproxy-certbot"` | grep "\"IPAddress\": \"1"
    Note: this works if you didn't change the name to the container (LOAD_BALANCER_NAME) inside the .env file. Otherwise, you need to change that.

    You get something like this:

    pi@homeassistant:~ $ docker container inspect `docker ps -aqf "name=haproxy-certbot"` | grep "\"IPAddress\": \"1"
                        "IPAddress": "172.26.0.2",

    The 172.26.0.2 is the IP address that I needed.

    Get the first certificate

    To get a certificate from Let's Encrypt, you need to forward the ports 80 and 443 of your computer to the Internet and you need a URL pointing to your router (basically a domain name). Internet providers usually change the IP address that you get every 24 hours. Thus, you need to get a dynamic DNS service. I use NoIP, it's free but you need to confirm your host every 30 days (no big deal). If you are thinking of buying a subscription, you can get a 5 dollars discount using the promo code REFER5, After opening the ports on your router and getting a domain name that points to your router, you can get the first certificate from Let's Encrypt. To do that, you need to get inside the Docker container using:

    docker container exec -it `docker ps -aqf "name=haproxy-certbot"` /bin/bash

    and run the following commands:

    /usr/bin/certbot certonly -c /usr/local/etc/letsencrypt/cli.ini --agree-tos --email [YOUR_EMAIL] --domains [YOUR_DOMAIN]

    haproxy-refresh Replace [YOUR_EMAIL] and [YOUR_DOMAIN] with your valid email address and the domain that points to your router. If everything goes as planned, you will get a valid SSL certificate for your HA system.

    Then, you can integrate your HA to e.g. Google Assistant following the steps in this tutorial.

    Conclusions

    This article updates all my articles about Home Assistant. The setup described in this article can replace every Zigbee hub that you already have in your home and can be integrated with your Google Assistant. Additionally, sensors and actors that connect to the Wi-Fi such as Tuya or Sonoff or even Xiaomi BLE devices can be also integrated and combined with the Zigbee devices in automation rules, scenes, and scripts.

    The Google Assistant is possible due to a HAProxy configuration that adds an SSL layer to HA. In this article, I've described the steps to start a Docker container running HAProxy and acquire an SSL certificate from Let's Encrypt.

    I also included tips and some learnings that I've acquired while I was configuring my setup during the last couple of years. I hope this helps you to set up your home, but remember Good home automation never annoys but is missed when it is not working (source).


    Comments

    Steve 01.17.2022

    I like your pi set up instructions. And for assist++ do you also show system or application gui's? For instance I use Google home and open it all day long to adjust devices. If that ends up command line I would wish to avoid that solution.

    Fotios 01.18.2022

    Hi, great article. I've been meaning to give HA a try and this was really informative.

    FYI, the MAC address issue you have is called MAC address randomization and it's a security feature. It's described more in this fedora article along with some ways to configure it. Things might be slightly different in other OSes, but it's a starting point for you.
    https://fedoramagazine.org/randomize-mac-address-nm/

    Cheers!