ETL in MySQL – How to automatically load data into a table every day

I’m working on a dashboard to track COVID-19 cases per capita in Calgary, and while the government’s open data API provides daily case counts within the city it doesn’t have any history available. The easy solution to this is to download the data on a daily basis and archive it myself, but I want to automate the download and loading into the database so I don’t have to think about it. Luckily, MySQL and a bit of shell script goes a long way.

Installing MySQL 8 on Ubuntu 20.04

If you need a database for a project, MySQL is one of the most popular choices. It’s free, open-source and is a core part of of the popular LAMP (Linux, Apache, MySQL, PHP) web application stack. If you want to get started using MySQL for a project, here’s a guide of how to install it on a fresh installation of Ubuntu 20.04.

Secure your Raspberry Pi with a firewall

After you’ve changed the default username and password on our Raspberry Pi or BeagleBone Black, the next step to securing it is to install and configure a firewall. A firewall will help keep your device secure by limiting or block connections to the services running on your device, creating a far smaller attack surface that can be exploited.

Create Professional Reports in Excel using R and XLSX

While R is the real workhorse of data analysis and modeling, if you want to share the results of your work with other people your probably going to send it in a spreadsheet. Luckily the xlsx package for R makes it easy to export simple spreadsheets and also has advanced functionality to create workbooks with professional design and formatting.

CRISP-DM in depth: modeling

Once the data preparation phase is complete, its time to move on to the fun part of the CRISP-DM framework: modeling. Here you’ll chose which modeling technique to use, create some tests to assess the accuracy of your model, build the model, and then assess the model using the tests you created.

Scheduling R scripts to run automatically in Windows

A key component in data acquisition or reporting is the ability to trigger your script to run at a set time each day. Whether you are attempting to download the latest stock prices or update corporate earnings reports, once you’ve created the script to do the actual work, you need to find a way for it to be run on the correct schedule.

CRISP-DM in depth: data preparation

After developing business understanding and data understanding, the next big objective in the CRISP-DM methodology is to prepare the data for modelling and analysis. This involves selecting, cleaning and transforming the data which will be used for the project. While this isn’t flashy work, it typically accounts for 60% to 80% of the effort for a project.