Logistic Regression

Logistic Regression in a Supervised Machine Learning algorithm, which is used where the response variable is categorical.

Table content:-

1. Definition
2. Types of Logistic Regression
3. Assumptions of Logistic Regression
4. Why Logistic Over Linear?
5. Odds Ratio and Logit
6. Logistic Regression Model
7. Cost function
8. Evaluation Metrics
9. References

Definition

Logistic Regression make the use of logit function, which helps to find the relationship between independent variable and the dependent variables by predicting the probability of their outcome.

Different types of Logistic Regression are :-

1. Binomial Logistic Regression :- In this target variables can have only two possible outcome. ex :- Yes/No or Pass/fail etc.
2. Multinomial Logistic Regression :- In this target variables can have three or more possible outcome ( outcomes don’t have any quantitative significance ). ex :- diseases A, diseases B and diseases C etc.
3. Ordinal Logistic Regression :- It deals with the target variables with ordered categories. ex :-“poor”, “good”, “very good” etc.

Assumptions of Logistic Regression :-

Logistic regression don’t make any key assumptions of linear Regression like based on linearity, normality, homoscedasticity etc.

But following are the assumption still apply :-

1. Binary logistic regression requires the dependent variables to be binary while the ordinal logistic regression requires the dependent variables to be ordinal.
2. Observations should not come from repeated measurements or matched data.
3. It requires no or little multicollinearity within independent variables which means that independent variables should not be too highly correlated with each other.
4. It assumes linearity of independent variables and log odds.
5. Logistic Regression typically requires a large sample size.

Reason why linear Regression is not suitable for Classification problem :-

1. In Linear Regression predicted value is continuous, they are not probabilistic like Logistic Regression.
2. Linear regression is sensitive to outliers or imbalance data.
3. Linear regression can predict the probability from negative to positive infinity, but the probability can only lies between 0 and 1. To tackle such problem we use logit function or log-odds function.

Odds Ratio and Logit

The logit function maps probabilities from the range (0,1) to the entire real number range (−∞,∞). It is written as

Logit function is just a log of odds

Logit function equation

Here we have L.H.S as logit function and the R.H.S as odds.

The inverse form of the logit function is also called the logistic function, and as we know that logistic function is also called sigmoid function due to its characteristic of S-shaped curve.

Logistic or sigmoid function

It always gives the probability between 0 to 1 as outcome.

The Logistic Model

Like all other regression analyses, the logistic regression is a predictive analysis.

Logistic regression uses the sigmoid function which will restrict the value of logistic regression between 0 and 1.

In logistic regression we have threshold value and value of logistic regression above that threshold value is considered as 1 and below that value below that threshold value will be considered as 0.

Cost function in Logistic Regression

Cost function quantifies the error between predicted value and the expected value. It also helps us to evaluate the performance of the machine learning model for a given dataset.

Cost function in Logistic Regression

The cost function used in Logistic Regression is Log Loss.

Evaluation of Logistic Regression Model

Following are few major metrics using which we can evaluate the performance of the logistic regression :-

1. Akaike Information Criteria (AIC) :- AIC is the measure of fit which penalizes model for the number of model coefficients. Therefore, a model having minimum AIC value should always be considered.
2. Confusion Matrix :- Confusion matrix shows us the tabular representation of the actual and predicted value. Using this we can find the accuracy and it also helps us in avoiding overfitting.
3. Receiver Operating Characteristic ( ROC Curve ) :- Higher the area under curve, better the prediction power of the model.

References :-

1. Wikipedia
2. Analytics Vidhya Blogs
3. Few other sources