Random forest is a type of machine learning algorithm in which the algorithm makes multiple decision trees that may use different features and subsample to making as many trees as you specify. The trees then vote to determine the class of an example. This approach helps to deal with the high variance that is a problem with making only one decision tree.

In this post, we will learn how to develop a random forest model in Python. We will use the cancer dataset from the pydataset module to classify whether a person status is censored or dead based on several independent variables. The steps we need to perform to complete this task are defined below

1. Data preparation
2. Model development and evaluation

Data Preparation

Below are some initial modules we need to complete all of the tasks for this project.

import pandas as pd
import numpy as np
from pydataset import data
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

We will now load our dataset “Cancer” and drop any rows that contain NA using the .dropna() function.

df = data('cancer')
df=df.dropna()

Next, we need to separate our independent variables from our dependent variable. We will do this by make two datasets. The X dataset will contain all of our independent variables and the y dataset will contain our dependent variable. You can check the documentation for the dataset using the code data(“Cancer”, show_doc=True)

Before we make the y dataset we need to change the numerical values in the status variable to text. Doing this will aid in the interpretation of the results. If you look at the documentation of the dataset you will see that a 1 in the status variable means censored while a 2 means dead. We will change the 1 to censored and the 2 to dead when we make the y dataset. This involves the use of the .replace() function. The code is below.

X=df[['time','age',"sex","ph.ecog",'ph.karno','pat.karno','meal.cal','wt.loss']]
df['status']=df.status.replace(1,'censored')
df['status']=df.status.replace(2,'dead')
y=df['status']

We can now proceed to model development.

Model Development and Evaluation

We will first make our train and test datasets. We will use a 70/30 split. Next, we initialize the actual random forest classifier. There are many options that can be set. For our purposes, we will set the number of trees to make to 100. Setting the random_state option is similar to setting the seed for the purpose of reproducibility. Below is the code.

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
h=RandomForestClassifier(n_estimators=100,random_state=1)

We can now run our modle with the .fit() function and test it with the .pred() function. The code is velow.

h.fit(x_train,y_train)
y_pred=h.predict(x_test)

We will now print two tables. The first will provide the raw results for the classification using the .crosstab() function. THe classification_reports function will provide the various metrics used for determining the value of a classification model.

print(pd.crosstab(y_test,y_pred))
print(classification_report(y_test,y_pred))

Our overall accuracy is about 75%. How good this is depends in context. We are really good at predicting people are dead but have much more trouble with predicting if people are censored.

Conclusion

This post provided an example of using random forest in python. Through the use of a forest of trees, it is possible to get much more accurate results when a comparison is made to a single decision tree. This is one of many reasons for the use of random forest in machine learning.

Hyperparameters are a numerical quantity you must set yourself when developing a model. This is often one of the last steps of model development. Choosing an algorithm and determining which variables to include often come before this step.

Algorithms cannot determine hyperparameters themselves which is why you have to do it. The problem is that the typical person has no idea what is an optimally choice for the hyperparameter. To deal with this confusion, often a range of values are inputted and then it is left to python to determine which combination of hyperparameters is most appropriate.

In this post, we will learn how to set hyperparameters by developing a grid in  Python. To do this, we will use the PSID dataset from the pydataset library. Our goal will be to classify who is married and not married based on several independent variables. The steps of this process is as follows

1.  Data preparation
2. Baseline model (for comparison)
3. Grid development
4. Revised model

Below is some initial code that includes all the libraries and classes that we need.

import pandas as pd
import numpy as np
from pydataset import data
pd.set_option('display.max_rows', 5000)
pd.set_option('display.max_columns', 5000)
pd.set_option('display.width', 10000)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV

Data Preparation

The dataset PSID has several problems that we need to address.

• We need to remove all NAs
• The married variable will be converted to a dummy variable. It will simply be changed to married or not rather than all of the other possible categories.
• The educatnn and kids variables have codes that are 98 and 99. These need to be removed because they do not  make sense.

Below is the code that deals with all of this

df=data('PSID').dropna()
df.loc[df.married!= 'married', 'married'] = 0
df.loc[df.married== 'married','married'] = 1
df['married'] = df['married'].astype(int)
df['marry']=df.married
df.drop(df.loc[df['kids']>90].index, inplace=True)
df.drop(df.loc[df['educatn']>90].index, inplace=True

1. Line 1 loads the dataset and drops the NAs
2. Line 2-4 create our dummy variable for marriage. We create a new variable called marry to hold the results
3. Lines 5-6 drop the values in  kids and educatn that are above 90.

Below we create our X and y datasets and then are ready to make our baseline model.

X=df[['age','educatn','hours','kids','earnings']]
y=df['marry']

Baseline Model

The purpose of  baseline model is to see how much better or worst the hyperparameter tuning works. We are using K Nearest Neighbors  for our classification. In our example, there are 4 hyperparameters we need to set. They are as follows.

1. number of neighbors
2. weight of neighbors
3. metric for measuring distance
4. power parameter for minkowski

Below is the baseline model with the set hyperparameters. The second line shows the accuracy of the model after a k-fold cross-validation that was set to 10.

classifier=KNeighborsClassifier(n_neighbors=5,weights=’uniform’, metric=’minkowski’,p=2)
np.mean(cross_val_score(classifier,X,y,cv=10,scoring=’accuracy’,n_jobs=1)) 0.6188104238047426

Our model has an accuracy of about 62%. We will now move to setting up our grid so we can see if tuning the hyperparameters improves the performance

Grid Development

The grid allows you to develop scores of models with all of the hyperparameters tuned slightly differently. In the code below, we create our grid object, and then we calculate how many models we will run

grid={'n_neighbors':range(1,13),'weights':['uniform','distance'],'metric':['manhattan','minkowski'],'p':[1,2]}
np.prod([len(grid[element]) for element in grid])
96 

You can see we made a simple list that has several values for each hyperparameter

1. Number if neighbors can be 1 to 13
2. weight of neighbors can be uniform or distance
3. metric can be manhatten or minkowski
4. p can be 1 or 2

We will develop 96 models all together. Below is the code to begin tuning the hyperparameters.

search=GridSearchCV(estimator=classifier,param_grid=grid,scoring='accuracy',n_jobs=1,refit=True,cv=10)
search.fit(X,y)

The estimator is the  code for the type of algorithm we are using. We set this earlier. The param_grid is our grid. Accuracy is our metric for determining the best model. n_jobs has to do with the amount of resources committed to the process. refit is for changing parameters and cv is for cross-validation folds.The search.fit command runs the model

The code below provides the output for the results.

print(search.best_params_)
print(search.best_score_)
{'metric': 'manhattan', 'n_neighbors': 11, 'p': 1, 'weights': 'uniform'}
0.6503975265017667

The best_params_ function tells us what the most appropriate parameters are. The best_score_ tells us what the accuracy of the model is with the best parameters. Are model accuracy improves from 61% to 65% from adjusting the hyperparameters. We can confirm this by running our revised model with the updated hyper parameters.

Model Revision

Below is the cod efor the erevised model

classifier2=KNeighborsClassifier(n_neighbors=11,weights='uniform', metric='manhattan',p=1)
np.mean(cross_val_score(classifier2,X,y,cv=10,scoring='accuracy',n_jobs=1)) #new res
Out[24]: 0.6503909993913031

Exactly as we thought. This is a small improvement but this can make a big difference in some situation such as in a data science competition.

Conclusion

Tuning hyperparameters is one of the final pieces to improving a model. With this tool, small gradually changes can be seen in a model. It is important to keep in mind this aspect of model development in order to have the best success final.

K Nearest Neighbor uses the idea of proximity to predict class. What this means is that with KNN Python will look at K neighbors to determine what the unknown examples class should be. It is your job to determine the K or number of neighbors that should be used to determine the unlabeled examples class.

KNN is great for a small dataset. However, it normally does not scale well when the dataset gets larger and larger. As such, unless you have an exceptionally powerful computer KNN is probably not going to do well in a Big Data context.

In this post, we will go through an example of the use of KNN with the turnout dataset from the pydataset module. We want to predict whether someone voted or not based on the independent variables. Below is some initial code.

from pydataset import data
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import classification_report

We now need to load the data and separate the independent variables from the dependent variable by making two datasets.

df=data("turnout")
X=df[['age','educate','income']]
y=df['vote']

Next, we will make our train and test sets with a 70/30 split. The random.state is set to 0. This argument allows us to reproduce our model if we want. After this, we will run our model. We will set the K to 7 for our model  and run the model. This means that Python will look at the 7 closes examples to predict the value of an unknown example. below is the code

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.3,random_state=0)
clf=KNeighborsClassifier(7)
clf.fit(X_train,y_train)

We can now predict with our model and see the results with the classification reports function.

y_pred=clf.predict(X_test)
print(classification_report(y_test,y_pred))

The results are shown above. To determine the quality of the model relies more on domain knowledge. What we can say for now is that the model is better at classifying people who vote rather than people who do not vote.

Conclusion

This post shows you how to work with Python when using KNN. This algorithm is useful in using neighboring examples tot predict the class of an unknown example.

Naive Bayes is a probabilistic classifier that is often employed when you have multiple or more than two classes in which you want to place your data. This algorithm is particularly used when you dealing with text classification with large datasets and many features.

If you are more familiar with statistics you know that Bayes developed a method of probability that is highly influential today. In short, his system takes into conditional probability. In the case of naive Bayes,  the classifier assumes that the presence of a certain feature in a class is not related to the presence of any other feature. This assumption is why Naive Bayes is Naive.

For our purposes, we will use Naive Bayes to predict the type of insurance a person has in the DoctorAUS dataset in the pydataset module. Below is some initial code.

from pydataset import data
import pandas as pd
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

Next, we will load our dataset DoctorAUS. Then we will separate the independent variables that we will use from the dependent variable of insurance in two different datasets. If you want to know more about the dataset and the variables you can type data(“DoctorAUS”, show_doc=True)

df=data("DoctorAUS")
X=df[['age','income','sex','illness','actdays','hscore','doctorco','nondocco','hospadmi','hospdays','medecine','prescrib']]
y=df['insurance']

Now, we will create our train and test datasets. We will do a 70/30 split. We will also use Gaussian Naive Bayes as our algorithm. This algorithm assumes the data is normally distributed. There are other algorithms available for Naive Bayes as well.  We will also create our model with the .fit function.

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.3,random_state=0)
clf=GaussianNB()
clf.fit(X_train,y_train)

Finally, we will predict with our model and run the classification report to determine the success of the model.

y_pred=clf.predict(X_test)
print(classification_report(y_test,y_pred))

You can see that our overall numbers are not that great. This means that the current algorithm is probably not the best choice for classification. Of course, there could other problems as well that need to be explored.

Conclusion

This post was simply a demonstration of how to conduct an analysis with Naive Bayes using Python. The process is not all that complicate and is similar to other algorithms that are used.

Linear discriminant analysis is a classification algorithm commonly used in data science. In this post, we will learn how to use LDA with Python. The steps we will for this are as follows.

1. Data preparation
2. Model training and evaluation

Data Preparation

We will be using the bioChemists dataset which comes from the pydataset module. We want to predict whether someone is married or single based on academic output and prestige. Below is some initial code.

import pandas as pd
from pydataset import data
import matplotlib.pyplot as plt
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

Now we will load our data and take a quick look at it using the .head() function.

There are two variables that contain text so we need to convert these two dummy variables for our analysis the code is below with the output.

Here is what we did.

1. We created the dummy variable by using the .get_dummies() function.
2. We saved the output in an object called dummy
3. We then combine the dummy and df dataset with the .concat() function
4. We repeat this process for the second variable

The output shows that we have our original variables and the dummy variables. However, we do not need all of this information. Therefore, we will create a dataset that has the X variables we will use and a separate dataset that will have our y values. Below is the code.

X=df[['Men','kid5','phd','ment','art']]
y=df['Married']

The X dataset has our five independent variables and the y dataset has our dependent variable which is married or not. We can not split our data into a train and test set.  The code is below.

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.3,random_state=0)

The data was split 70% for training and 30% for testing. We made a train and test set for the independent and dependent variables which meant we made 4 sets altogether. We can now proceed to model development and testing

Model Training and Testing

Below is the code to run our LDA model. We will use the .fit() function for this.

clf=LDA()
clf.fit(X_train,y_train)

We will now use this model to predict using the .predict function

y_pred=clf.predict(X_test)

Now for the results, we will use the classification_report function to get all of the metrics associated with a confusion matrix.

The interpretation of this information is described in another place. For our purposes, we have an accuracy of 71% for our prediction.  Below is a visual of our model using the ROC curve.

Here is what we did

1. We had to calculate the roc_curve for the model this is explained in detail here
2. Next, we plotted our own curve and compared to a baseline curve which is the dotted lines.

A ROC curve of 0.67 is considered fair by many. Our classification model is not that great but there are worst models out there.

Conclusion

This post went through an example of developing and evaluating a linear discriminant model. To do this you need to prepare the data, train the model, and evaluate.

Kmeans clustering is a technique in which the examples in a dataset our divided through segmentation. The segmentation has to do with complex statistical analysis in which examples within a group are more similar the examples outside of a group.

The application of this is that it provides the analysis with various groups that have similar characteristics which can be used to cater services to in various industries such as business or education. In this post, we will look at how to do this using Python. We will use the steps below to complete this process.

1. Data preparation
2. Determine the number of clusters
3. Conduct analysis

Data Preparation

Our data for this examples comes from the sat.act dataset available in the pydataset module. Below is some initial code.

import pandas as pd
from pydataset import data
from sklearn.cluster import KMeans
from scipy.spatial.distance import cdist
import numpy as np
import matplotlib.pyplot as plt

We will now load our dataset and drop any NAs they may be present

You can see there are six variables that will be used for the clustering. Next, we will turn to determining the number of clusters.

Determine the Number of Clusters

Before you can actually do a kmeans analysis you must specify the number of clusters. This can be tricky as there is no single way to determine this. For our purposes, we will use the elbow method.

The elbow method measures the within sum of error in each cluster. As the number of clusters increases this error decreases. However,  a certain point the return on increasing clustering becomes minimal and this is known as the elbow. Below is the code to calculate this.

distortions = []
K = range(1,10)
for k in K:
kmeanModel = KMeans(n_clusters=k).fit(df)
distortions.append(sum(np.min(cdist(df, kmeanModel.cluster_centers_, 'euclidean'), axis=1)) / df.shape[0])

Here is what we did

1. We made an empty list called ‘distortions’ we will save our results there.
2. In the second line, we told R the range of clusters we want to consider. Simply, we want to consider anywhere from 1 to 10 clusters.
3. Line 3 and 4, we use a for loop to calculate the number of clusters when fitting it to the df object.
4. In Line 5, we save the sum of the cluster distance in the distortions list.

Below is a visual to determine the number of clusters

plt.plot(K, distortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Distortion')
plt.title('The Elbow Method showing the optimal k')

The graph indicates that 3 clusters are sufficient for this dataset. We can now perform the actual kmeans clustering.

KMeans Analysis

The code for the kmeans analysis is as follows

km=KMeans(3,init='k-means++',random_state=3425)
km.fit(df.values)

1. We use the KMeans function and tell Python the number of clusters, the type of, initialization, and we set the seed with the random_state argument. All this is saved in the object called km
2. The km object has the .fit function used on it with df.values

Next, we will predict with the predict function and look at the first few lines of the modified df with the .head() function.

You can see we created a new variable called predict. This variable contains the kmeans algorithm prediction of which group each example belongs too. We then printed the first five values as an illustration. Below are the descriptive statistics for the three clusters that were produced for the variable in the dataset.

It is clear that the clusters are mainly divided based on the performance on the various test used. In the last piece of code, gender is used. 1 represents male and 2 represents female.

We will now make a visual of the clusters using two dimensions. First, w e need to make a map of the clusters that is saved as a dictionary. Then we will create a new variable in which we take the numerical value of each cluster and convert it to a string in our cluster map dictionary.

clust_map={0:'Weak',1:'Average',2:'Strong'}
df['perf']=df.predict.map(clust_map)

Next, we make a different dictionary to color the points in our graph.

d_color={'Weak':'y','Average':'r','Strong':'g'}

Here is what is happening in the code above.

1. We set the ax object to a value.
2. A for loop is used to go through every example in clust_map.values so that they are colored according the color
3. Lastly, a plot is called which lines up the perf and clust values for color.

The groups are clearly separated when looking at them in two dimensions.

Conclusion

Kmeans is a form of unsupervised learning in which there is no dependent variable which you can use to assess the accuracy of the classification or the reduction of error in regression. As such, it can be difficult to know how well the algorithm did with the data. Despite this, kmeans is commonly used in situations in which people are trying to understand the data rather than predict.

This post will provide a demonstration of the use of the random forest algorithm in python. Random forest is similar to decision trees except that instead of one tree a multitude of trees are grown to make predictions. The various trees all vote in terms of how to classify an example and majority vote is normally the winner. Through making many trees the accuracy of the model normally improves.

The steps are as follows for the use of random forest

1. Data preparation
2. Model development & evaluation
3. Model comparison
4. Determine variable importance

Data Preparation

We will use the cancer dataset from the pydataset module. We want to predict if someone is censored or dead in the status variable. The other variables will be used as predictors. Below is some code that contains all of the modules we will use.

import pandas as pd
import sklearn.ensemble as sk
from pydataset import data
from sklearn.model_selection import train_test_split
from sklearn import metrics
import matplotlib.pyplot as plt

We will now load our data cancer in an object called ‘df’. Then we will remove all NA’s use the .dropna() function. Below is the code.

df = data('cancer')
df=df.dropna()

We now need to make two datasets. One dataset, called X, will contain all of the predictor variables. Another dataset, called y, will contain the outcome variable. In the y dataset, we need to change the numerical values to a string. This will make interpretation easier as we will not need to lookup what the numbers represents. Below is the code.

X=df[['time','age',"sex","ph.ecog",'ph.karno','pat.karno','meal.cal','wt.loss']]
df['status']=df.status.replace(1,'censored')
df['status']=df.status.replace(2,'dead')
y=df['status']

Instead of 1 we now have the string “censored” and instead of 2 we now have the string “dead” in the status variable. The final step is to set up our train and test sets. We will do a 70/30 split. We will have a train set for the X and y dataset as well as a test set for the X and y datasets. This means we will have four datasets in all. Below is the code.

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

We are now ready to move to model development

Model Development and Evaluation

We now need to create our classifier and fit the data to it. This is done with the following code.

clf=sk.RandomForestClassifier(n_estimators=100)
clf=clf.fit(x_train,y_train)

The clf object has our random forest algorithm,. The number of estimators is set to 100. This is the number of trees that will be generated. In the second line of code, we use the .fit function and use the training datasets x and y.

We now will test our model and evaluate it. To do this we will use the .predict() with the test dataset Then we will make a confusion matrix followed by common metrics in classification. Below is the code and the output.

You can see that our model is good at predicting who is dead but struggles with predicting who is censored. The metrics are reasonable for dead but terrible for censored.

We will now make a second model for the purpose of comparison

Model Comparison

We will now make a different model for the purpose of comparison. In this model, we will use out of bag samples to determine accuracy, set the minimum split size at 5 examples, and that each leaf has at least 2 examples. Below is the code and the output.

There was some improvement in classify people who were censored as well as for those who were dead.

Variable Importance

We will now look at which variables were most important in classifying our examples. Below is the code

model_ranks=pd.Series(clf.feature_importances_,index=x_train.columns,name="Importance").sort_values(ascending=True,inplace=False)
ax=model_ranks.plot(kind='barh')

We create an object called model_ranks and we indicate the following.

1. Classify the features by importance
2. Set index to the columns in the training dataset of x
3. Sort the features from most to least importance
4. Make a barplot

Below is the output

You can see that time is the strongest classifier. How long someone has cancer is the strongest predictor of whether they are censored or dead. Next is the number of calories per meal followed by weight and lost and age.

Conclusion

Here we learned how to use random forest in Python. This is another tool commonly used in the context of machine learning.

Decision trees are used in machine learning. They are easy to understand and are able to deal with data that is less than ideal. In addition, because of the pictorial nature of the results decision trees are easy for people to interpret. We are going to use the ‘cancer’ dataset to predict mortality based on several independent variables.

We will follow the steps below for our decision tree analysis

1. Data preparation
2. Model development
3. Model evaluation

Data Preparation

We need to load the following modules in order to complete this analysis.

import pandas as pd
import statsmodels.api as sm
import sklearn
from pydataset import data
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn import tree
import matplotlib.pyplot as plt
from sklearn.externals.six import StringIO 
from IPython.display import Image 
from sklearn.tree import export_graphviz
import pydotplus

The ‘cancer’ dataset comes from the ‘pydataset’ module. You can learn more about the dataset by typing the following

data('cancer', show_doc=True)

This provides all you need to know about our dataset in terms of what each variable is measuring. We need to load our data as ‘df’. In addition, we need to remove rows with missing values and this is done below.

df = data('cancer')
len(df)
Out[58]: 228
df=df.dropna()
len(df)
Out[59]: 167

The initial number of rows in the data set was 228. After removing missing data it dropped to 167. We now need to setup up our lists with the independent variables and a second list with the dependent variable. While doing this, we need to recode our dependent variable “status” so that the numerical values are replaced with a string. This will help us to interpret our decision tree later. Below is the code

X=df[['time','age',"sex","ph.ecog",'ph.karno','pat.karno','meal.cal','wt.loss']]
df['status']=df.status.replace(1,'censored')
df['status']=df.status.replace(2,'dead')
y=df['status']

Next,  we need to make our train and test sets using the train_test_split function.  We want a 70/30 split. The code is below.

x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

We are now ready to develop our model.

Model Development

The code for the model is below

clf=tree.DecisionTreeClassifier(min_samples_split=10)
clf=clf.fit(x_train,y_train)

We first make an object called “clf” which calls the DecisionTreeClassifier. Inside the parentheses, we tell Python that we do not want any split in the tree to contain less than 10 examples. The second “clf” object uses the  .fit function and calls the training datasets.

We can also make a visual of our decision tree.

dot_data = StringIO()
export_graphviz(clf, out_file=dot_data, 
filled=True, rounded=True,feature_names=list(x_train.columns.values),
special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue()) 
Image(graph.create_png())

If we interpret the nodes furthest to the left we get the following

• If a person has had cancer less than 171 days and
• If the person is less than 74.5 years old then
• The person is dead

If you look closely every node is classified as ‘dead’ this may indicate a problem with our model. The evaluation metrics are below.

Model Evaluation

We will use the .crosstab function and the metrics classification functions

You can see that the metrics are not that great in general. This may be why everything was classified as ‘dead’. Another reason is that few people were classified as ‘censored’ in the dataset.

Conclusion

Decisions trees are another machine learning tool. Python allows you to develop trees rather quickly that can provide insights into how to take action.

In this post, we will explore a dataset using Python. The dataset we will use is the Ghouls, Goblins, and Ghost (GGG) dataset available at the kaggle website. The analysis will not be anything complex we will simply do the following.

• Data preparation
• Data  visualization
• Descriptive statistics
• Regression analysis

Data Preparation

The GGG dataset is fictitious data on the characteristics of spirits. Below are the modules we will use for our analysis.

import pandas as pd
import statsmodels.regression.linear_model as sm
import numpy as np

Once you download the dataset to your computer you need to load it into Python using the pd.read.csv function. Below is the code.

df=pd.read_csv('FILE LOCATION HERE')

We store the data as “df” in the example above. Next, we will take a peek at the first few rows of data to see what we are working with.

Using the print function and accessing the first five rows reveals. It appears the first five columns are continuous data and the last two columns are categorical. The ‘id’ variable is useless for our purposes so we will remove it with the code below.

df=df.drop(['id'],axis=1)

The code above uses the drop function to remove the variable ‘id’. This is all saved into the object ‘df’. In other words, we wrote over are original ‘df’.

Data Visualization

We will start with our categorical variables for the data visualization. Below is a table and a graph of the ‘color’ and ‘type’ variables.

First, we make an object called ‘spirits’ using the groupby function to organize the table by the ‘type’ variable.

Below we make a graph of the data above using the .plot function. A professional wouldn’t make this plot but we are just practicing how to code.

We now know how many ghosts, goblins and, ghouls there are in the dataset. We will now do a breakdown of ‘type’ by ‘color’ using the .crosstab function from pandas.

We will now make bar graphs of both of the categorical variables using the .plot function.

We will now turn our attention to the continuous variables. We will simply make histograms and calculate the correlation between them. First the histograms

The code is simply subset the variable you want in the brackets and then type .plot.hist() to access the histogram function. It appears that all of our data is normally distributed. Now for the correlation

Using the .corr() function has shown that there are now high correlations among the continuous variables. We will now do an analysis in which we combine the continuous and categorical variables through making boxplots

The code is redundant. We use the .boxplot() function and tell python the column which is continuous and the ‘by’ which is the categorical variable.

Descriptive Stats

We are simply going to calcualte the mean and standard deviation of the continuous variables.

df["bone_length"].mean()
Out[65]: 0.43415996604821117

np.std(df["bone_length"])
Out[66]: 0.13265391313941383

df["hair_length"].mean()
Out[67]: 0.5291143100058727

np.std(df["hair_length"])
Out[68]: 0.16967268504935665

df["has_soul"].mean()
Out[69]: 0.47139203219259107

np.std(df["has_soul"])
Out[70]: 0.17589180837106724

The mean is calcualted with the .mean(). Standard deviation is calculated using the .std() function from the numpy package.

Multiple Regression

Our final trick is we want to explain the variable “has_soul” using the other continuous variables that are available. Below is the code

X = df[["bone_length", "rotting_flesh","hair_length"]]

y = df["has_soul"]

model = sm.OLS(y, X).fit()

In the code above we crate to new list. X contains are independent variables and y contains the dependent variable. Then we create an object called model and use the  OLS() function. We place the y and X inside the parenthesis and we then use the .fit() function as well. Below is the summary of the analysis

There is obviously a lot of information in the output. The r-square is 0.91 which is surprisingly high given that there were not high correlations in the matrix. The coefficiencies for the three independent variables are listed and all are significant. The AIC and BIC are for model comparison and do not mean much in isolation. The JB stat indicates that are distribution is not normal. Durbin watson test indicates negative autocorrelation which is important in time-series analysis.

Conclusion

Data exploration can be an insightful experience. Using Python, we found mant different patterns and ways to describe the data.

In this post, we will look at some ways to calculate some inferential statistics in Python. We are mainly going to focus on z-scores and one/two-tailed test.

We will begin by import some needed packages and then we will make some data and plot it.  Below is the code and plot

import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats
quizscore=np.random.normal(69,9,60)
plt.hist(quizscore,density=True)

We create an object called “quizscore” which contains as you can tell quiz scores. In line four in the code above, we used the .random.normal function from the numpy package to create some random data. This function has 3 arguments in it, the first is the mean of the distribution, next is the standard deviation, and the last is the sample size. After this is the code for the histogram which uses the .hist function from the matplotlib.pyplot package.

Z-Scores

Z-Scores are a measure of how far a data point is from the mean. In the code below, we calculate the first five z-scores of our “quizscore” dataset.

stats.zscore(quizscore)[0:5,]
Out[127]: array([ 0.54362001, 1.56135871, -0.36924596, 0.53182556, -0.06014972])

In the output, you can see that the first number in the dataset is 0.54 standard deviations above the mean. Remember, our standard deviation was set to 9 so this means the score for the first quiz was about 73. We confirm this with the code below.

quizscore[0,]
Out[129]: 72.851341820695538

Fairly close, another question we can answer is what is the probability that someone got a score that was 1.5 standard deviations or higher on the quiz (about a score of 82). Below is the code followed by the answer.

1-stats.norm.cdf(1.5)
Out[132]: 0.066807201268858085

In the code above we subtract 1 from our code. The code uses the .cdf function. Inside the function, we put our z-score of 1.5. The answer is 0.066 or 6% of the scores have a Z-score of 1.5 or higher or a quiz score of around 82 or higher.

We can also determine what is the cutoff point for a certain high score. For us, we want to know what the cutoff for the top 15% of quiz scores. Below is the code.

stats.norm.ppf(0.85)
Out[136]: 1.0364333894937898

(1.03 * quizscore.std()) +quizscore.mean()
Out[137]: 77.748927759179054

In the code above, first we had to convert the percentage to a z-score and this is done with the .ppf function. The 85 percentile is equivalent to a Z-score of about 1. Next, we multiplied the z-score by the standard deviation of the quizscore and added the mean of the quizscore to this to get our final answer of 77.74. This means that a score of 77.74 and above is within the top 15%.

One & Two-Tailed Test

A one-tail test is used to compare the sample mean to a population mean. Another way to look at this is to convert a z-score to a p-value. What is happening here is you create a region in your distribution in which scores are considered unusual. For a one-tail test, this is the top 5%. For example, let’s say you want to know the probability that someone got a score of 92 or higher on the quiz. This can be solved with the following code.

quizZ=(92-quizscore.mean())/quizscore.std()
1-stats.norm.cdf(quizZ)
Out[138]: 0.0072370644085374414

Here is what we did

1. We create an object called “quizZ” which took our specific score of 92 subtracted the mean of “quizscore” and divided it by the standard deviation of “quizscore”. This becomes the z-score for the value 92.
2. We then subtract one from this while using the .cdf function
3. The output indicates that there is less than a 1% chance that someone got a score of 92% or higher.

In this example our sample mean was small (only 1) but the concept remains the same.

A two-tailed test is the same concept except that the rejections regions are on both sides of the distribution and divide in half. This means that the top 2.5% and bottom 2.5% are now considered unusual scores. As an example, if the average score on the quiz is 75 with a standard deviation of 5 and we want to see if our class mean of 67 is different. We can use the following code to answer this.

two_tail=(quizscore.mean()-75)/5
stats.norm.cdf(two_tail)
Out[160]: 0.063688920253590395

You can see that the probability of our class is not unusual as it is above the cutoff of 5% indicating no difference. This means that if 75 is the center of our distribution the quiz score of 67 would be within 2 standard deviations of the mean.

Conclusion

In this post, we explored some of the ways you can do inferential statistics with Python. Whatever you want to know can be calculated by knowing a few lines of code.

It is somewhat difficult to define a string. In some ways, a string is text such as what is found in a book or even in this blog.  Strings are made up of characters such as letters and even symbols such as %#&@**. However, the computer does not use these characters but converts them to numbers for processing.

In this post, we will learn some of the basics of working with strings in Python. This post could go on forever in terms of what you can do with text so we will only address the following concepts

• Finding individual characters
• Modifying text

Finding Individual Characters

Finding individual characters in a string is simple. You simply need to type the variable name and after the name use brackets and put the number of the location of the character in the string. Below is an example

example="Educational Research Techniques"

print(example[0])
E

As you can see, we created a variable called “example” the content of “example” is the string “Educational Research Techniques”. To access the first letter we use the “print” function, type the name of the variable and inside brackets put the letter 0 for the first position in the string. This gives us the letter E. Remeber that Python starts from the number 0 and not 1.

If you want to start from the end of the string and go backward you simply use negative numbers instead of positive as in the example below.

example="Educational Research Techniques"
print(example[-1])
s

You can also get a range of characters.

example="Educational Research Techniques"
print(example[0:5])
Educa

The code above is telling python to take from position 0 to 4 but not including 5.

You can do even more creative things such as pick characters from different spots to create new words

example="Educational Research Techniques"
print(example[0:2] + example[20:25])
Ed Tech

Modifying Strings

It is also possible to modify the text in the string such as making all letters upper or lower case as shown below.

example.upper()
'EDUCATIONAL RESEARCH TECHNIQUES'

example.lower()
'educational research techniques'

The upper() function capitalizes and the lower() function make small case.

It is also possible to count the length of a string and swap the case with the len() and swapcase() functions respectively.

len(example)
31

example.swapcase()
'eDUCATIONAL rESEARCH tECHNIQUES'

Python allows you to count how many times a character appears using the count() function. You can also use the replace() to find characters and change them.

example.count('e')
4

example.replace('e','q')
'Educational Rqsqarch Tqchniquqs'

As you can see, the lower case letter ‘e’ appears four times. In the second line, python replaces all the lower case ‘e’s with the letter ‘q’.

Conclusion

Of course, there is so much more than we could do with strings. However, this post provides an introduction to what is possible.

When you are unsure how much data your application may need to process it is probably appropriate to use a while statements. The while statement we keep processing until it runs out of items to process. If you remember with a traditional for loop the limit is preset by the data structure you are analyzing.

Nested for loops is another concept we will look at. They are useful when it is unclear what the conditions of execution should be.

Since they can go on forever it is possible with a while statement to create an endless loop. This is a loop that never stops processing. This will essentially crash most computers and should naturally be avoided. To avoid this you need set the environment, state the while statement, and update the condition of the environment. Below is a simple example of this.

1. In line 1, we have the environment for the condition which is a variable called “number” set to 0.
2. Line 2 is the while statement which states that as long as “number” is less than 10 do the following.
3. In line 3 the variable “number” is printed.
4. In line 4, after “number” is printed the number 2 is added to the current value
5. This takes place until the variable “number” is equal to ten.

Below is what the output looks like if you ran this

You can see that we start with 0. This is because we set the variable “number” originally to 0. Then the value increases by just as in line 4 of the code above.

Nested for Loops

Just as with functions you can also have nested for loops. This is a loop within a loop. Below is a simple example.

1. Lines 1-2 as for input and you can type whatever you want
2. Line 3-4 is the first for loop at it process whatever your input is from line 1. The final result is that the loop prints this
3. Line 5-6 are the second for loop and simply processes whatever you inputted in line 2. This loop simply prints the input from line 2.

Here is what the output would look like

You can see that the loops took turns. The first loop ran its first letter them then the second loop ran everything. Then the first loop ran its second letter and the second loop ran everything again. Therefore, nested for loops affects the timing of when the code is ran.

Conclusion

This post looked at the use of while statements and for loops. while statements are useful when you do not know how long you may need to process data. for loops allow you to run complex looping in which you are trying to do multiple tasks.

The use of for loops are valuable when you need your application to do a repetitive task. Once the task is completed there is some sort of output that is returned. Understanding how to create a for loop is a critical step in utilizing the Python language.

Making for loops

Here is the basic syntax for a for loop

for item in data:
  do something

The word “for” indicates a for loop. The word “item” is an iteration variable. An iteration variable is a variable that changes value each time the loop goes through the data. It takes on the current value that is being analyzed for whatever purpose the loop has. You can name the iteration variable anything you want but a general rule is to use names that make sense for the context. Otherwise, nobody else will be able to understand your code.

After the colon is where you find “do something” here you put the command for whatever the loop is supposed to do. Below is an actual example of the use of the for loop.

Here is what happened

1. At the top, we have our for loop. The iterator variable is “letter” and we are looping through the data of the string “education”.
2. The next line is the action the for loop will perform. Essentially, the loop will pull each later from the string “education” and insert them one at a time into the phrase “Give me an”,. Notice how the word “letter” is at the end of our print statement. This the iteration variable that changes each time our for loop goes through the string “education.
3. The output is several print statements each containing a different letter from the string “education”

for loops with Breaks

Breaks are used to provide conditions in which the loop will stop. In the example below, we add some code to our cheer that allows you to enter your own cheer. However, the church must be less than 10 letters otherwise you get a message that word is too long. Below is the code

Here is what it does.

1. In line 1, you provide a word as indicated by the instructions in the parentheses.
2. Line 2 is the for loop. letter is the iteration variable for our word in “Value”
3. Line 3 is the if statement. The strong on “Value” is checked to make sure it is 10 characters or less.
4. In line 4, if “Value” is greater than 10 characters you get the message that the cheer is too long.
5. Line 5 is the break which stops the loop from continuing.
6. In line 6, if the word is less than 10 characters you get the cheer with each letter.

Below is the output with less than 10 characters

Here is the output with more than 10 characters

Continue and for loop

The continue clause allows you to check the data and only process it based on certain conditions. In the code below, we are going to change our cheer code so that it removes spaces when making the cheer.

The code is mostly the same with a few exceptions

• The if statement looks for blank spaces and these are left out of the cheer.
• The continue clause tells python to keep going
• Finally, the cheer is given

Below is what the output looks like if you ran this code

You can see that I put many blank spaces in-between the letters but these do not appear in the code. This is because of the continue clause.

Conclusion

for loops are a basic yet powerful tool of programming. In Python, for loops are used for the same reason as other languages and that is for completing repetitive tasks. The examples, here provide some simple ways in which this can be done.

This post will provide a brief introduction into the use of the elif clause and nested statements in Python.

elif Clause

The elif clause is used to add an additional set of conditions to an if statement. If you have ever used some sort of menu on a computer in which you had to make several choices it is possible that an elif clause was involved in the code.

The syntax for the elif clause is the same as for the if statement. Below is a simple example that employs the elif clause.

Here is what this code does.

1. In lines 1-3, I print three lines of code at the beginning. These are the choices available to the user.
2. In line 4, the “pick” variable stores whatever number the user inputs through the “input” function. It must be an integer which is why I used the “int” function
3. In line 5 we begin the if statement. If the “pick” variable is set to 1 you can see the printout in line 6.
4. In lines 7 and 8 we use the elif clause. The settings are mostly the same as in the if statement in line 5 except the “pick” variable is set to different numbers with different outputs.
5. Lastly, in line 11 we have the else clause. If for any reason the person picks something besides 1,2 or 3 this message will appear.

What this code means in simple English is this

• If the pick is 1 then print “dogs are your favorite animal”
• or else if the pick is 2 then print “cats are your favorite animal”
• or else if the pick is 3 then print “rabbits are your favorite animal”
• else print “I do not understand your choice”

Here is what the code looks like when you run it

As a note, if you type in a letter or string you will get an error message. This is because our code is not sophisticated enough to deal with non-integers at this point.

Nested Statements

Just like with functions which can be nested so can decision statements by nesting inside each other. To put this simply the conditions set by the first if statement can potentially affect the second condition. This is explained better with an example.

Here is what is happening in the code above.

1. In lines 1 and 2 the user has to pick numbers that are saved in the variables “num1” and “num2.”
2. Line 3 and 4 are the if statements. Line 3 and line 9 are the outer if statement and line 4-8 are the inner if statement.
3. Line 3 shares that “num1” must be between 1 and 10.
4. Line 4 shares that “num2” must be between 1 and 10.
5. Line 5 is the results of the inner if statement. The results are printed using the “.format” method where the {0} and {1} are the variables “num1” and “num2”. After the comma is what is done to the variables, they are simply added together.
6. Line 8 is the else clause. If someone types something different form a number between 1-10 for the second number they will see this message
7. Line 9 is the else clause for the outer if statement. This is only seen if a value different from 1-10 is inputted.

If you run the code here is what it should look like

Conclusion

The elif clause and nested decision statements are additional tools that can be used to empower your applications. This is some of the most basic ideas in using a language such as Python.

Applications often have to make decisions and to do this they need a set of conditions to him them decide what to do. Python, like all programming languages, has tools that allow the application to execute various actions based on conditions. In this post, we will look at the use of if statements.

If Statement Define

An if statement is a statement used in Python that determines when an action should happen. At a minimum, an if statement should have two parts to it. The first part is the condition and the second part is the action that Python performs if the condition is true. Below is what this looks like

if SOMETHING IS TRUE:
    DO THIS

This is not the most beautiful code but this is the minimum for a if statement in Python.

If Statement 

Below is the actual application of the use of an if statement

 number=5
if number ==5:
   print("Correct")
 
Correct

In the code above we created a variable called number and set its value to 5. Then we created the if statement by setting the condition as “if number equals 5” the action for this being true was prin the string “correct”, which Python does at the bottom.

Notice the double equal sign used in the if statement. The double equal sign is used for relational equality while the single equal sign is used for assigning values to variables. Also, notice the colon at the end of the if statement. This must be there in order to complete the code.

There is no limit to the number of tasks an if statement can perform. In to code above the, if statement only printed “Correct” however, we can put many more print functions or other actions as shown below.

number=5
if number ==5:
   print("Correct")
   print("Done")
 
Correct
Done

All that is new above is that we have two print statements that were executed because the if statement was true.

Multiple Comparisons

In the code above python only had to worry about whether number equaled 5. However, we can have Python make multiple comparisons as well below is an example.

number=5
if (number>0) and (number<10):
 print("Correct")
 print("Finished")
 
Correct
Finished

Now for the print functions to execute the number has to be greater than 0 but less than 10. If you change the number to something less than 0 or greater than 10 nothing will happen when you run the code because the conditions were not met. In addition, if you type in a letter like ‘a’ you will get an error message because Python is anticipating a number and not a string.

If Else Statements

The use of an else clause in an if statement allows for an alternative task to be executed if the first conditions in the if statement are not met. This allows the application to do something when the conditions are not met rather than leaving a blank screen. Below we modify our code in the previous example slightly so that something happens because of the else clause in the if statement.

number=15
if (number>0) and (number<10):
 print("Correct")
 print("Finished")
else:
 print("Number is out of range")
 
Number is out of range

In this code above we did the following

1. we set our number to 15
2. We created an if statement that searches for a number greater than 0 and less than 10.
3. If the conditions in step 2 are true we print the two statements
4. If the conditions in step 2 are not met we print the statement after the else clause
5. Our number was 15 so Python printed the statement after the else clause

It can get much more complicated and powerful than this but I think this is clear and enough for now.

Conclusion

This post provided an introduction to the if statement in Python. THe if statement will execute a command based on one or more conditions. You must be careful with the syntax. Lastly, you can also include an alternative task with the else clause if the conditions are not met.