Implementing Real-Time data pipelines on the Cloud. DIY Projects from data generation, acquisition, storage up to visualization. In this section, you walk through examples of a new transformative, more collaborative way of doing data science on the Cloud.
#M5Stack: Widgets on your Smartphone help you to stay safe from #COVID-19
Cloud Platforms, M5Stack | 2 min | 3953
This is the last follow-up of the tutorials:
- M5Stack: Fresh air checker helps you to stay safe from #COVID-19 and,
- Nginx: Docker, Flask and Let's Encrypt (SSL),
in which I explained how to build a CO2 measuring device using a M5Stack and an SG30 sensor, send the data to GCP, and offer it using HTTPS.
Fig. 1: CO2 reporting widgets on Android and iOS.
These sensors can help to prevent Covid-19 spread in a room. If you want to implement the complete pipeline, check out the links above. This tutorial connects the data to your Android or iOS phone to see that in a Widget (see Fig. ...
#Nginx: Docker, Flask and Let's Encrypt (SSL)
Cloud Platforms | 5 min | 8671
This tutorial is a follow-up of M5Stack: Fresh air checker helps you to stay safe from #COVID-19, in which I explained how to build a CO2 measuring device using M5Stacks and SG30 sensors. These sensors can help to prevent Covid-19 spread in a room. The collected measurements were sent to the Google Cloud Platform and a microservice offered them on a Website or a RestAPI using HTTP. This tutorial adds an
Nginxmicroservice to offer the Website and RestAPI over HTTPS.
Let's start with some explanation about how HTTPS works: The encryption and security functionalities for HTTP are implemented thr...
#M5Stack: Fresh air checker can help you to stay safe from #COVID-19
Cloud Platforms, M5Stack, MicroPython | 12 min | 215172
Winter is coming and the air-quality inside a closed room could be an index used to estimate the amount of aerosols in it, and hence the risk of Covid-19 infection. In other words, a system of CO2 traffic lights could help to spot potential corona-aerosols!
Where there are many people in a room, there is a lot of exhalation (approx. eight liters of air per minute per nose). This air contains CO2 (carbon dioxide). Long before Covid-19, in Germany, the German Environment Agenc...
#GCP: Data pipeline - from datastore to Google Data Studio
Analytics, Cloud Platforms | 8 min | 17059
This tutorial is about exporting data from Google Firestore to Google Data Studio to visualize it; or to import it on Google Colab to analyze it and train a machine learning model.
Fig. 1: Data diagrams on Google Data Studio
This is the third tutorial from the series "implementing real-time data pipelines - from generation to models" and the other tutorials are the following:
#GCP: Implementing Real-Time data pipelines - from ingest to datastore
Analytics, Cloud Platforms, M5Stack | 13 min | 41091
Two weeks ago, I published a tutorial that explains how to connect an M5Stack running MicroPython to the Google Cloud Platform using the IoT Core, and I did mention that upcoming tutorials will examine the following topics:
- Collecting and synchronizing external data (weather from OpenWeatherMap) and other sensors -window/door status, sneezing detector-.
- Saving the data to a NoSQL database
- Displaying the obtained data on Google Data Studio (chec...
#Portainer: Managing Docker Engines remotely over TCP socket (TLS)
Analytics, Cloud Platforms | 4 min | 45603
This tutorial is about managing a Docker Engine remotely using Portainer connected to the protected Docker daemon socket (
TCP port 2376). By default, you can manage Docker locally through a non-networked UNIX socket (option
-v /var/run/docker.sock:/var/run/docker.sockwhile running Portainer). But, if you want the Docker Engine to be reachable through the network in a safe manner, you need to enable TLS by specifying the
--tlsverifyflag and pointing Docker’s
--tlscacertflag to a CA certificate. Then, the daemon only accepts connections from clients that are authenticated by a certificate si...
#GCP: Running OpenProject almost FREE
Cloud Platforms | 7 min | 10932
The last days, I was looking for an online project management tool with bug tracking system. The best platform that I know is Jira from Atlassian. Jira offers a bug tracking and agile project management, however, it costs between $8.33 and $10 (annual/monthly payment) for the standard version with under 10 accounts. We use Jira in Finealyze and I am very happy with it, but I was looking for a free alternative for myself. Then, I found OpenProject, which describes itself as "the leading open source project management software".
OpenProject is offered in three different ways: Community, Cloud an...
#Zerynth: ESP32 & Google IoT Core - Part 2: Getting data from Smart Power Outlets
Cloud Platforms, Home Automation, MicroPython | 2 min | 3470
This part 2 of the tutorial will allow you to get data from the smart power outlets and send it to the Google Cloud Platform (GCP) throught the Google IoT Core and using an ESP32 programmed with Zerynth (Python). This data is not usually available to the user (directly, you need to use the power outlet application), and it is sent to the company servers.
If you want to check, if your power outlets are compatible for this tutorial, they should have the...
#Zerynth: ESP32 & Google IoT Core - Part 1: Sending data to the Cloud
Cloud Platforms, MicroPython | 7 min | 17678
There are lot of tutorials about connecting the ESP32 to a cloud service, but I planned to connect these boards using MicroPython directly to the cloud. That means, I didn't want to use a hardware or software bridge. I wanted to use the Google IoT Core, but MicroPython doesn't support JSON Web Token (JWT), which is needed for the device authentication. I tried to write the library myself, but it didn't work, because of the needed dependencies and the small mem...