What is a Variable?
A variable is any characteristic, number, or quantity that can be measured or counted. The following are examples of variables:
- Age (21, 35, 62, ...)
- Gender (male, female)
- Income (GBP 20000, GBP 35000, GBP 45000, ...)
- House price (GBP 350000, GBP 570000, ...)
- Country of birth (China, Russia, Costa Rica, ...)
- Eye colour (brown, green, blue, ...)
- Vehicle make (Ford, Volkswagen, ...)
They are called 'variables' because the value they take may vary (and it usually does) in a population.
Most variables in a data set can be classified into one of two major types:
- Numerical variables
- Categorical variables
===================================================================================
Numerical variables
The values of a numerical variable are numbers. They can be further classified into discrete and continuous variables.
Discrete numerical variable
A variable which values are whole numbers (counts) is called discrete. For example, the number of items bought by a customer in a supermarket is discrete. The customer can buy 1, 25, or 50 items, but not 3.7 items. It is always a round number. The following are examples of discrete variables:
- Number of active bank accounts of a borrower (1, 4, 7, ...)
- Number of pets in the family
- Number of children in the family
Continuous numerical variable
A variable that may contain any value within some range is called continuous. For example, the total amount paid by a customer in a supermarket is continuous. The customer can pay, GBP 20.5, GBP 13.10, GBP 83.20 and so on. Other examples of continuous variables are:
- House price (in principle, it can take any value) (GBP 350000, 57000, 1000000, ...)
- Time spent surfing a website (3.4 seconds, 5.10 seconds, ...)
- Total debt as percentage of total income in the last month (0.2, 0.001, 0, 0.75, ...)
=============================================================================
Real Life example: Peer to peer lending (Finance)
Lending Club
Lending Club is a peer-to-peer Lending company based in the US. They match people looking to invest money with people looking to borrow money. When investors invest their money through Lending Club, this money is passed onto borrowers, and when borrowers pay their loans back, the capital plus the interest passes on back to the investors. It is a win for everybody as they can get typically lower loan rates and higher investor returns.
If you want to learn more about Lending Club follow this link.
The Lending Club dataset contains complete loan data for all loans issued through 2007-2015, including the current loan status (Current, Late, Fully Paid, etc.) and latest payment information. Features include credit scores, number of finance inquiries, address including zip codes and state, and collections among others. Collections indicates whether the customer has missed one or more payments and the team is trying to recover their money.
The file is a matrix of about 890 thousand observations and 75 variables. More detail on this dataset can be found in Kaggle's website
Let's go ahead and have a look at the variables!
====================================================================================================
To download the Lending Club loan book from Kaggle go to this website
Scroll down to the bottom of the page, and click on the link 'loan.csv', and then click the 'download' blue button towards the right of the screen, to download the dataset. Unzip it, and save it to a directory of your choice.
Note that you need to be logged in to Kaggle in order to download the datasets.
If you save it in the same directory from which you are running this notebook, then you can load it the same way I will load it below.
====================================================================================================
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
# let's load the dataset with just a few columns and a few rows
# to speed things up
use_cols = [
'loan_amnt', 'int_rate', 'annual_inc', 'open_acc', 'loan_status',
'open_il_12m'
]
data = pd.read_csv(
'loan.csv', usecols=use_cols).sample(
10000, random_state=44) # set a seed for reproducibility
data.head()
Continuous Variables
# let's look at the values of the variable loan_amnt
# this is the amount of money requested by the borrower
# in US dollars
data.loan_amnt.unique()
# let's make an histogram to get familiar with the
# distribution of the variable
fig = data.loan_amnt.hist(bins=50)
fig.set_title('Loan Amount Requested')
fig.set_xlabel('Loan Amount')
fig.set_ylabel('Number of Loans')
The values of the variable vary across the entire range of the variable. This is characteristic of continuous variables.
The taller bars correspond to loan sizes of 10000, 15000, 20000, and 35000. There are more loans disbursed for those loan amount values. This indicates that most people tend to ask for these loan amounts. Likely, these particular loan amounts are pre-determined and offered as such in the Lending Club website.
Less frequent loan values, like 23,000 or 33,000 could be requested by people who require a specific amount of money for a definite purpose.
# let's do the same exercise for the variable interest rate,
# which is charged by lending club to the borrowers
data.int_rate.unique()
# let's make an histogram to get familiar with the
# distribution of the variable
fig = data.int_rate.hist(bins=30)
fig.set_title('Interest Rate')
fig.set_xlabel('Interest Rate')
fig.set_ylabel('Number of Loans')
Again, we see that the values of the variable vary continuously across the variable range.
# and now,let's explore the income declared by the customers,
# that is, how much they earn yearly.
fig = data.annual_inc.hist(bins=100)
fig.set_xlim(0, 400000)
fig.set_title("Customer's Annual Income")
fig.set_xlabel('Annual Income')
fig.set_ylabel('Number of Customers')
The majority of salaries are concentrated towards values in the range 30-70 k, with only a few customers earning higher salaries. Again, the values of the variable, vary continuosly across the variable range.
Discrete Variables
Let's explore the variable "Number of open credit lines in the borrower's credit file" (open_acc in the dataset). This is, the total number of credit items (for example, credit cards, car loans, mortgages, etc) that is known for that borrower. By definition it is a discrete variable, because a borrower can have 1 credit card, but not 3.5 credit cards.
# let's inspect the values of the variable
data.open_acc.dropna().unique()
# let's make an histogram to get familiar with the
# distribution of the variable
fig = data.open_acc.hist(bins=100)
fig.set_xlim(0, 30)
fig.set_title('Number of open accounts')
fig.set_xlabel('Number of open accounts')
fig.set_ylabel('Number of Customers')
Histograms of discrete variables have this typical broken shape, as not all the values within the variable range are present in the variable. As I said, the customer can have 3 credit cards, but not 3,5 credit cards.
Let's look at another example of a discrete variable in this dataset: Number of installment accounts opened in past 12 months (open_il_12m in the dataset). Installment accounts are those that at the moment of acquiring them, there is a set period and amount of repayments agreed between the lender and borrower. An example of this is a car loan, or a student loan. The borrower knows that they are going to pay a certain, fixed amount over for example 36 months.
# let's inspect the variable values
data.open_il_12m.unique()
# let's make an histogram to get familiar with the
# distribution of the variable
fig = data.open_il_12m.hist(bins=50)
fig.set_title('Number of installment accounts opened in past 12 months')
fig.set_xlabel('Number of installment accounts opened in past 12 months')
fig.set_ylabel('Number of Borrowers')
The majority of the borrowers have none or 1 installment account, with only a few borrowers having more than 2.
A variation of discrete variables: the binary variable
Binary variables, are discrete variables, that can take only 2 values, therefore binary.
In the next cells I will create an additional variable, called defaulted, to capture the number of loans that have defaulted. A defaulted loan is a loan that a customer has failed to re-pay and the money is lost.
The variable takes the values 0 where the loans are ok and being re-paid regularly, or 1, when the borrower has confirmed that will not be able to re-pay the borrowed amount.
# let's inspect the values of the variable loan status
data.loan_status.unique()
# let's create one additional variable called defaulted.
# This variable indicates if the loan has defaulted, which means,
# if the borrower failed to re-pay the loan, and the money
# is deemed lost.
data['defaulted'] = np.where(data.loan_status.isin(['Default']), 1, 0)
data.defaulted.mean()
# the new variable takes the value of 0
# if the loan is not defaulted
data.head()
# the new variable takes the value 1 for loans that
# are defaulted
data[data.loan_status.isin(['Default'])].head()
# A binary variable, can take 2 values. For example,
# the variable defaulted that we just created:
# either the loan is defaulted (1) or not (0)
data.defaulted.unique()
# let's make a histogram, although histograms for
# binary variables do not make a lot of sense
fig = data.defaulted.hist()
fig.set_xlim(0, 2)
fig.set_title('Defaulted accounts')
fig.set_xlabel('Defaulted')
fig.set_ylabel('Number of Loans')
As we can see, the variable shows only 2 values, 0 and 1, and the majority of the loans are ok.
That is all for this demonstration. I hope you enjoyed the notebook, and see you in the next one.
