Know the Stages of Data Pre-Processing

 The role of a data worker, whether focused on analytics or machine learning tools, requires intense data pre-processing activity .

Although it is a “less glamorous” stage, most of the time spent is spent on this activity. It is estimated to consume around 70-80% of the total time and effort of a data analysis project.

Pre-processing is a set of activities that involve data preparation, organization and structuring. This is a fundamental step that precedes the performance of analyzes and predictions.

This step is of great importance, as it will be decisive for the final quality of the data that will be analyzed. It can even impact the forecast model generated from the data.

Among the main problems found within a dataset, also known as a dataset , we can list attributes with missing values, outliers and different scales for equal values.

In this article, I will introduce some data preprocessing techniques, which can be applied to any dataset.

The objective here is to transmit a more conceptual and general knowledge, so that it can be applied later by you in any programming language. 

Knowledge discovery process

The purpose of any data-related activity, such as data mining, is knowledge acquisition. It is essential knowledge for making more solid decisions.

Data mining is a business process for exploring large amounts of data with a focus on recognizing rules and patterns.

Within data mining, the three main models used for knowledge discovery/acquisition are: KDD, SEMMA and CRISP-DM, the latter being the most popular in the industry.

The CRISP-DM process involves:

  1. business understanding;
  2. understanding of the data;
  3. data preparation;
  4. data modeling;
  5. evaluation of results;
  6. deployment (production).

It is important to highlight that, before putting it into production, after the evaluation stage, it may be necessary to return to one of the previous steps, such as understanding the business, according to the following diagram:

Crisp-DM - ​​Knowledge Acquisition Process

Based on this methodology, for example, we would have in step number 3 the set of activities called data preparation or pre-processing . This is what we will focus on for the rest of the article.

Data structure

Before starting the pre-processing itself, it is important to know the main structures of the data. Generally speaking, they can be classified into three categories:

Structured data

It is the data that contains a rigid and pre-planned organization. They are usually “labeled” in lines and columns that identify their characteristics regarding certain subjects. They can be organized into semantic blocks (relations) and definition of descriptions for data from the same group (attribute).

Examples : Relational databases, excel spreadsheets, CSV files, among others.

Data Pre-Processing - Structured Data


Semi-structured data

Data that has a structure but does not conform to the formal structures of models associated with relational databases or other forms of data tables. They have placeholders, like tags, to separate semantic elements and create hierarchies for records and fields.

Examples : XML, JSON, HTML files, etc.

JSON - Semi-Structured Data


Unstructured data

It is the data in which we cannot identify a clear organization. In order to generate insights on this data, intensive pre-processing must be carried out to retrieve the information.

Examples : text documents, audio, images, etc.

unstructured data


Pre-Processing Techniques

Data preprocessing is a set of data mining techniques used to transform raw data into useful and efficient formats. It may be necessary in any of the 3 mentioned structures.

There are 3 main steps involved in this process: data cleansing, data transformation and data reduction. Each of them involves different activities.

1. Data cleaning

The original data in your dataset may contain many irrelevant or missing parts. To deal with this situation, data cleaning is essential. It involves handling and/or filling in missing data, reducing noise, identifying and removing outliers, and resolving inconsistencies.

A) Missing Data : This situation occurs when some data is missing. There are several practices to solve problems of this nature, including the following:

  • Remove records with null attributes;
  • Perform an average with the values ​​of the same attribute;
  • Perform a median with the values ​​of the same attribute;
  • Fill the missing attribute with the values ​​that occur the most in the dataset.

It is important to note that each data type may require a different strategy for dealing with missing data. None of the techniques is a panacea to solve all problems.

B) Noisy data : this is meaningless data that cannot be interpreted by machines. They can be generated due to failures in data collection, data entry errors, among other situations that are difficult to predict. They can be treated in the following ways:

  • Binning method : It is a data smoothing process, used to minimize the effects of small observation errors. The original data values ​​are divided into small ranges known as bins and then replaced with an overall value calculated for that bin. You can replace all data in a segment with their average or threshold values.
  • Regression : Here the data can be smoothed by fitting it to a regression function . It can be linear (with one independent variable) or multiple (with several independent variables).
  • Clustering : This approach groups similar data into a cluster. Outliers can be treated separately or left out of clusters.

2. Data Transformation

This step is performed to transform the original data into more appropriate and suitable formats for the mining process. It involves the following activities:

A) Normalization : This is done to scale the data values ​​in a specified range such as -1.0 to 1.0 or 0.0 to 1.0.

B) Attribute Selection : In this action new attributes are generated from the given attribute set to help in the mining process.

C) Discretization : It is the process of transferring continuous functions, models, variables and equations into discrete counterparts. This is important, because some algorithms only work with inputs of discrete values, not being able to predict continuous values. Discretization creates a limited number of possible states.

D) Concept hierarchy generation : Here, attributes are converted to a higher level in the hierarchy. For example: the attribute “city” can be converted to “country”.

3. Data reduction

Data mining is mainly used to deal with large volume of data. Therefore, computational processing becomes increasingly complex. To increase efficiency and reduce costs we use the data reduction process. Its main steps are:

A) Data Cube Aggregation : It is the activity of building a data cube , a multidimensional format that, despite generating a greater need for storage, allows faster processing as it does not need to scan the entire base in search of a certain value.

B) Selection of a subset of attributes : It is the option to use highly relevant attributes to the detriment of less relevant ones. To perform the selection of attributes, the significance level and the p -value of the attribute can be used. The attribute with p -value greater than the significance level can be discarded.

C) Numerosity reduction : Allows data to be replaced or estimated by smaller data representation alternatives , such as parametric models (which only store model parameters instead of the actual data) or non-parametric methods such as clustering, sampling and the use of histograms.

D) Dimensionality reduction : This reduces the size of the data by encoding mechanisms. It can be lossy or lossless. If after rebuilding from the compressed data, the original data can be recovered, this reduction is called lossless reduction. Otherwise, it is called lossy reduction. The two effective methods of dimensionality reduction are: Wavelet transformation and PCA ( Principal Component Analysis ).


Despite being considered a less interesting and quite laborious activity, data pre-processing is essential for any type of analysis. Otherwise, “garbage will go in and garbage will come out” in your model.

That is, not even the most powerful algorithm will be able to hit its predictions without quality datasets.

Did you like the post? Then share this article on social media, it was developed with a lot of effort and dedication.

Post a Comment

Previous Post Next Post