As big data transforms our businesses, governments and society, it also presents us with new moral and ethical dilemmas that we need to consider. As is typical with new technology, we often tend to implement first, and consider the ethical issues later. Cathy O’Neil’s book Weapons of Math Destruction is an introduction to the ethical issues raised by the widespread use of data to drive decisions in our lives.
When using the CRISP-DM framework, the first step in the data mining process is to develop your business understanding. This stage of the process is about gaining knowledge of the business, the issues they face, opportunities for improvement, their objectives, their constraints and creating your project plan.
One of the biggest benefits from creating an automatic reporting framework is that you no longer need to directly supervise the creation and distribution of reports. However, when things go wrong it can be difficult to understand what went wrong and why. Luckily, R’s tryCatchLog package makes it easy to trap and log errors as they occur.
Talking about the rate of change in our society has transcended being a statement of fact to being something of a cliché. Never the less, technical and societal changes are forcing us to regularly ask deep questions about how to move forward in the midst of rapid change. Joi Ito and Jeff Howe of the MIT Media Lab tackle these questions and propose new guiding principles in their book Whiplash.
One of the first steps in any data science project is to acquire and analyze the raw data. Since this data will commonly be stored in databases, understanding Structured Query Language (SQL) will enable you to get the data you need and start working quickly. This post summarizes the basics of SQL’s SELECT statement, which is how you retrieve information from the database.
While analysis tools and algorithms have evolved at a rapid pace, the overall business process for analytics has remained remarkably stable. One seminal work on the analytic process is IBM’s Cross-Industry Standard Process for Data Mining (CRISP-DM). At over 20 years old, it remains a relevant and useful tool for describing the overall data science workflow.
A successful data scientist needs to draw on skills from many disciplines, and one of the core skill sets is knowledge of relational databases and querying using structured query language (SQL). Relational databases are the most common way to store structured data, so a firm understanding of databases is key to obtaining performing simple analysis and reporting quickly.
Hiring is one of the most challenging tasks in a manager’s portfolio. Selecting individuals for a long-term commitment based on a few comparatively short interviews and conversations is not easy. Luckily, there are a number of personality traits to look for which can help you select promising individuals for your data science or analytics team.
When setting up automated reporting workflows, a key component is distributing the report to the various stakeholders so they can review it. One useful technique is to send a “push notification” by email to alert everyone that a report or analysis task has just been completed. Luckily, it isn’t hard to send out email from R by using Microsoft Outlook.