What is Non-Linear Regression?

Non-Linear Regression aka Attributes Data Analysis is organized into discrete data types (groups or categories).

When do you use Non-Linear Regression aka Attributes Data Analysis?

Simple Linear regression is a statistical technique that is used to estimate the mathematical relationship between a dependent variable (usually denoted as Y) and an independent variable (usually denoted as X) and is denoted with a straight line y=mx+b. Although statistical linear models could describe the classic straight line, most statistically linear models are not represented by straight lines but by curvilinear graphs. While non-linear regression aka Attributes Data Analysis is used to explain the nonlinear relationship between a response variable and one or more than one predictor variable (mostly curve line). In other words, a regression model is called “non-linear” if the derivative of the model depends on one or more parameters.

Specifically use non-linear regression instead of ordinary least square regression when one cannot adequately model the relationship with linear parameters

DO NOT USE Linear Regression for attribute data analysis- because this assumes the response variable is continuous.

For instance Analyze attributes data using logit, probit, logistic regression, etc., to investigate sources of variation

Logistic Regression

Logistic regression is a method that is used to predict a dependent variable for a given set of independent variables, such that the dependent variable is categorical. This method is especially widely used in machine learning algorithms.

Logistic regression is similar to linear regression; however, logistic regression predicts categorical (like true or false) where as linear regression predicts continuous data. Hence the dependent variable in logistic regression is categorical, but in linear regression dependent variable is always continuous. Logistic regression uses a logit transformation on the dependent variable to fit a linear regression model.

The outcome of logistic regression is always categorical when the resultant outcome has always only two possible values of 0 or 1.  In logistic regression, the graph is not a linear line, but the line looks like a curve goes between 0 and 1. The curve is called the “S” curve and is also called as Sigmoid curve.

Although logistic regression tells the probability, it is most commonly used for classification. The team has to determine the threshold value if the probability of the event happening between 0 and 1, then based on the threshold value event is to be classified as 0 or 1.

For example, if the threshold value is 0.5, then any value between 0.5 and 1 then it should be classified as 1, similarly any value below 0.5 then it should classify as 0

Logistic Regression calculation

P ranges between 0 and 1, it represents the probability of an event to be happen

X₁ and X2 are the independent variables that determine the occurrence of an event

C is the constant

B₁, B₂ represents the respective coefficients of X₁, X₂

Additional Notes on Logistic Regression

• The regression coefficients for logistic regression models are based on MAXIMUM LIKELIHOOD ESTIMATION
• Sample sizes should be at least 50 per variable.
• Logistic Regression is best for:
  • Binary w/ 2 categories
  • Ordinal or Nominal w/ 3 or more categories

Logistic Regression Example

We can use Non-linear regression models in the Analyze phase of DMAIC. For example, American Cancer Institute wants to predict the chances of throat cancer based on the smoker’s average cigarette consumption per day. From the below random data, 0 indicates no cancer, and 1 indicates throat cancer.

From the above data logistic regression helps to assess what level of cigarette consumption leads to throat cancer. Moreover, from the above graph, one can say if the number of cigarette consumption is more than 8 it leads to throat cancer, and if cigarette consumption is below 8 no cancer. Hence logistic regression is the best method for complex problems.

Certainly, we have to use statistical tools (like R and Minitab) to model the logistic regression.

Using the formula

Suppose we get the following values (random values) from the above information

C= -0.100245

B1= 0.1814

If we want to predict 2.6

=-0.100245+ 0.1814*2.6= -0.53081

Use the sigmoid function to find the probability

Since the f(z ) value is 0.37 (towards zero), hence we can predict at 2.6 there will be no chance of throat cancer.

Logit Analysis

Logit regression is a model where the dependent variable is qualitative, In other words, dependent variable is categorical. If the dependent variable is categorical in two parts it is binary logistic regression, for more than two categories it is a multinomial logit regression.

Formula for P=f(z)

f is the cumulative distribution function for a standard logistic random variable.

z= β₀+ β₁x₁+ β₂x₂+………………+ β_nx_n

β₀ is the intercept and β_{1 ,}β_2,β₃ are the slopes against independent variable

f is a function : 0<f(z)<1, for all z

Logit analysis Example

A cardiology doctor wants to predict the chances of life threat with cholesterol. He collected data after specific periods having different cholesterol levels for both alive and dead patients. The response is a binary variable (i.e alive or dead)

Find the logarithm base 10 for cholesterol

Calculate the proportion of dead patients= dead/(alive+dead)

Calculate the logit (p) = log(p/1-p)

While calculating the logit values, calculate the slope and intercept using logit values and Log values of cholesterol.

We have to use statistical tools (like R and Minitab) to model the logistic regression.

Once the model was ready, Use the sigmoid function to find whether the chances of the patient is dead or alive for a particular cholesterol level.

Probit Analysis

The probit model was first introduced by Chester Bliss in 1934, but the maximum likelihood method was proposed by Ronald Fisher as an appendix to Bliss in 1935.

A probit model is a popular specification for an ordinal or a binary response model. This model, which employs a probit link function, was estimated using the maximum likelihood method, hence this estimation was named probit regression.

z= β₀+ β₁x₁+ β₂x₂+………………+ β_nx_n

Where f is the standard normal cumulative distribution function (cdf)

Probit Analysis Example

A coaching center interested in students’ GMAT (Graduate Management Admission Test ) scores and admission into the business school. The response is a binary variable (i.e admit or no admission into the business school)

Firstly, find the logarithm base 10 for GMAT Score

Then, calculate the proportion of no admission = No admission /(admission + no admission)

Lastly, calculate the Probit values using the chart

While calculating the probit values, calculate the slope and intercept using probit values and Log values of the GMAT score.

We have to use statistical tools (like R and Minitab) to model the logistic regression.

Once the model was established, the probit function was to find whether the chances of student admission or no admission into the business school for a particular GMAT score.

Difference between logit and probit

Both logit and probit models yield the almost same results. The logit and probit functions are increasing and both functions increase relatively quickly in the central portion and relatively slow at the extremities and both functions lie between 0 and 1. While the only key difference between the logit and probit models is in the use of the link function

In other words, the logit model will have slightly flatter tails when compared to the probit model.