Data Exploration in Data Mining

• Computer Graphics OpenGL Mini Projects

• Download Final Year Projects

Introduction to Data Exploration – Feature Engineering and Feature Selection in Data Mining

In this article, I will discuss,

• How to read the dataset?
• How to know the data types of columns?
• general Data Description
• Univariate analysis and Bi-Variate Analysis

Video Tutorial – Data Exploration in Data Mining

Click here to download the titanic.csv file, the dataset used in this demonstration.

First, we will import the required libraries like pandas, numpy, seaborn, matplotlib, and explore from data_exploration.

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import os
plt.style.use('seaborn-colorblind')
%matplotlib inline
from data_exploration import explore

Next, we use the read_csv() function from the pandas library to read the dataset. We are interested in few columns hence a list with use_cols is created with required columns.

use_cols = ['Pclass', 'Sex', 'Age', 'Fare', 'SibSp', 'Survived']

data = pd.read_csv('./data/titanic.csv', usecols=use_cols)

Now we display the first five rows, to confirm whether the dataset is read successfully or not using the data.head(5) function.

Univariate Analysis

Below are some methods that can give us the basic stats on the variable:

• pandas.Dataframe.dtypes
• pandas.Dataframe.describe()
• Barplot
• Countplot
• Boxplot
• Distplot

pandas.Dataframe.dtypes

Now we use the get_dtypes() function to get the types of each column and display them.

str_var_list, num_var_list, all_var_list = explore.get_dtypes(data=data)
print(str_var_list) # string type
print(num_var_list) # numeric type
print(all_var_list) # all

Output:

[‘Sex’]

[‘Survived’, ‘Pclass’, ‘Age’, ‘SibSp’, ‘Fare’]

[‘Sex’, ‘Survived’, ‘Pclass’, ‘Age’, ‘SibSp’, ‘Fare’]

pandas.Dataframe.describe()

Next, we use the describe() function to get the general description of dataset. The describe() function displays different statistics like, count, unique values, frequency, mean, standard deviation, minimum, maximum, 25%, 50% and 75% percentile.

explore.describe(data=data,output_path=r'./output/')

Out of describe() function():

Discrete variable barplot

discrete_var_barplot() function is used to draw the barplot of a discrete variable x against y (that is target variable). By default, the bar shows the mean value of y.

explore.discrete_var_barplot(x='Pclass',y='Survived',data=data,output_path='./output/')

Discrete variable countplot

discrete_var_countplot() function is used to draw the countplot of a discrete variable x.

explore.discrete_var_countplot(x='Pclass',data=data,output_path='./output/')

Discrete variable boxplot

discrete_var_boxplot() function is used to draw the boxplot of a discrete variable x against y.

explore.discrete_var_boxplot(x='Pclass',y='Fare',data=data,output_path='./output/')

Bi-variate Analysis

Bi-variate Analysis is performed to understand the descriptive statistics between two or more variables.

• Scatter Plot
• Correlation Plot
• Heat Map

Continuous variable distplot

continuous_var_distplot() issued to draw the distplot of a continuous variable x.

explore.continuous_var_distplot(x=data['Fare'],output_path='./output/')

Correlation plot

correlation_plot() function I used to draw the correlation plot between variables.

explore.correlation_plot(data=data,output_path='./output/')

Summary

This article introduces the Data Exploration – FeatureEngineering and Feature Selection in Data Mining. If you like the material share it with your friends. Like the Facebook page for regular updates and YouTube channel for video tutorials.