Month: October 2020
-
CRISP-DM in depth: data understanding
Having developed business understanding and a deep knowledge of the problem you are trying to solve, the next step in the CRISP-DM framework is to develop that same level of understanding around the data itself. This step isn’t analysis, but rather looking at the structure and shape of the data in order to determine what…
-
Quickly create polished, professional reports with Markdown and R
One underappreciated feature in R is the ability to easily create beautiful reports using Markdown. Markdown files contain a combination of code and text, allowing you to write your analysis alongside your code and publish both the analysis document and code in a wide variety of formats with little effort.
-
Defining big data
Buzz words have the unfortunate tendency to be often used but seldom clearly defined. Today we are going to tackle the popular phrase “big data” and strip it down to a clear definition. Overall the term is fairly self explanatory, it refers to large data sets, but there are 5 defining characteristics specific to big…
-
Book review: Weapons of Math Destruction by Cathy O’Neil
As big data transforms our businesses, governments and society, it also presents us with new moral and ethical dilemmas that we need to consider. As is typical with new technology, we often tend to implement first, and consider the ethical issues later. Cathy O’Neil’s book Weapons of Math Destruction is an introduction to the ethical…
-
CRISP-DM in depth: business understanding
When using the CRISP-DM framework, the first step in the data mining process is to develop your business understanding. This stage of the process is about gaining knowledge of the business, the issues they face, opportunities for improvement, their objectives, their constraints and creating your project plan.
-
Error catching, logging, and reporting in R with tryCatchLog
One of the biggest benefits from creating an automatic reporting framework is that you no longer need to directly supervise the creation and distribution of reports. However, when things go wrong it can be difficult to understand what went wrong and why. Luckily, R’s tryCatchLog package makes it easy to trap and log errors as…
-
Book review: Whiplash by Joi Ito and Jeff Howe
Talking about the rate of change in our society has transcended being a statement of fact to being something of a cliché. Never the less, technical and societal changes are forcing us to regularly ask deep questions about how to move forward in the midst of rapid change. Joi Ito and Jeff Howe of the…
-
A brief overview of SQL’s SELECT statement
One of the first steps in any data science project is to acquire and analyze the raw data. Since this data will commonly be stored in databases, understanding Structured Query Language (SQL) will enable you to get the data you need and start working quickly. This post summarizes the basics of SQL’s SELECT statement, which…
-
The Cross-Industry Standard Process for Data Mining
While analysis tools and algorithms have evolved at a rapid pace, the overall business process for analytics has remained remarkably stable. One seminal work on the analytic process is IBM’s Cross-Industry Standard Process for Data Mining (CRISP-DM). At over 20 years old, it remains a relevant and useful tool for describing the overall data science…
-
Supporting skills for data science: relational databases & SQL
A successful data scientist needs to draw on skills from many disciplines, and one of the core skill sets is knowledge of relational databases and querying using structured query language (SQL). Relational databases are the most common way to store structured data, so a firm understanding of databases is key to obtaining performing simple analysis…