What is Non-Linear Regression?

Non-Linear Regression aka Attributes Data Analysis is organized into discrete data types (groups or categories.)

When do you use Non-Linear Regression aka Attributes Data Analysis?

Simple Linear regression is a statistical technique to estimate the mathematical relationship between a dependent variable (usually denoted as Y) and an independent variable (usually denoted as X) and is denoted with a straight line y=mx+b. Although statistical linear models could describe the classic straight line, most statistically linear models are not represented by straight lines but by curvilinear graphs. While non-linear regression aka Attributes Data Analysis is used to explain the nonlinear relationship between a response variable and one or more than one predictor variable (mostly curve line). In other words, a regression model is called “non-linear” if the derivative of the models depends on one or more parameters.

Non- Linear Regression aka Attributes Data Analysis
Linear Vs Nonlinear Regression

Specifically use non-linear regression instead of ordinary least square regression when one cannot adequately model the relationship with linear parameters

DO NOT USE Linear Regression for attribute data analysis- because this assumes the response variable is continuous.

For instance Analyze attributes data using logit, probit, logistic regression, etc., to investigate sources of variation

Logistic Regression

Logistic regression is a method to predict a dependent variable for a given set of independent variables, such that the dependent variable is categorical. This method is especially widely using in machine learning algorithms.

Logistic regression similar to linear regression, however logistic regression predicts categorical (like true or false) where as linear regression predicts continuous data. Hence the dependent variable in logistic regression is categorical, but in linear regression dependent variable is always continuous. Logistic regression uses a logit transformation on the dependent variable to fit a linear regression model.

The outcome of logistic regression is always categorical, when the resultant outcome has always only two possible value of 0 or 1.  In logistic regression the graph is not a linear line, but the line looks like a curve goes between 0 and 1. The curve is called “S” curve and also called as Sigmoid curve.

Although the logistic regression tells the probability, it is most commonly used for classification. Team has to determine the threshold value if the probability of event happened between 0 and 1, then based on threshold value event to be classified as 0 or 1.

For example if the threshold value is 0.5, then any value between 0.5 and 1 then it should be classified as 1, similarly any value below 0.5 then it should classify as 0

Logistic Regression calculation

Non- Linear Regression aka Attributes Data Analysis

P ranges between 0 and 1, it represents the probability of event to be happen

X1, X2 are the independent variables which determines the occurrence of an event

C is the constant

B1, B2 represents the respective coefficients of X1, X2

Additional Notes on Logistic Regression

  • The regression coefficients for logistic regression models are based on MAXIMUM LIKELIHOOD ESTIMATION
  • Sample sizes should be at least 50 per variable.
  • Logistic Regression best for:

Logistic Regression Example

Non-linear regression models may be used in the Analyze phase of DMAIC. For example, American Cancer Institute wants to predict the chances of throat cancer based on the smoker’s average cigarette consumption per day. From the below random data, 0 indicates no cancer, and 1 indicates throat cancer.

From the above data logistic regression helps to assess what level of cigarette consumption leads to throat cancer. Moreover, from the above graph one can say if number of cigarette consumption is more than 8 it leads to throat cancer and if cigarette consumption is below 8 no cancer. Hence logistic regression is the best method for complex problems.

Certainly we have to use statistical tools (like R, Minitab) to model the logistic regression.

Using the formula

Suppose if we get following values (random values) from the above information

C= -0.100245

B1= 0.1814

If we want to predict for 2.6

=-0.100245+ 0.1814*2.6= -0.53081

Use sigmoid function to find the probability

Since f(z ) value is 0.37 (towards zero), hence we can predict at 2.6 there will no chance of throat cancer.

Logit Analysis

Logit regression is a model where dependent variable is qualitative, In other words dependent variable is categorical. If the dependent variable is categorical in two parts it is binary logistic regression, for more than two categories it is a multinomial logit regression.

Formula for P=f(z)

f is the cumulative distribution function for a standard logistic random variable.

z= β0+ β1x1+ β2x2+………………+ βnxn

β0 is the intercept and β1 ,β2,β3 are the slopes against independent variable

f is a function : 0<f(z)<1, for all z

Logit analysis Example

A cardiology doctor wants to predict the chances of life threat with the cholesterol.  Patient alive and also dead data collected after specific period having different cholesterol levels. The response is a binary variable (i.e alive or dead)

Find the logarithm base 10 for cholesterol

Calculate the proportion of dead patients= dead/(alive+dead)

Calculate the logit (p) = log(p/1-p)

While calculating the logit values, calculate the slope and intercept using logit values and Log values of cholesterol.

We have to use statistical tools (like R, Minitab) to model the logistic regression.

Once the model was ready, Use sigmoid function to find whether the chances of the patient dead or alive for a particular cholesterol level.

Probit Analysis

The probit model was first introduced by Chester Bliss in 1934, but the maximum likelihood method was proposed by Ronald Fisher as an appendix to Bliss in 1935.

A probit model is a popular specification for an ordinal or a binary response model. This model , which employs a probit link function, this was estimated using the maximum likelihood method, hence this estimation was named it as probit regression.

z= β0+ β1x1+ β2x2+………………+ βnxn

Where f is the standard normal cumulative distribution function (cdf)

Probit Analysis Example

A coaching center interested in students GMAT (Graduate Management Admission Test )score and admission into the business school. The response is binary variable (i.e admit or no admission in to the business school)

First of all find the logarithm base 10 for GMAT Score

Calculate the proportion of no admission = No admission /(admission + no admission)

Calculate the Probit values using the chart

While calculating the probit values, calculate the slope and intercept using probit values and Log values of GMAT score.

We have to use statistical tools (like R, Minitab) to model the logistic regression.

Once the model was establish, Use probit function to find whether the chances of student admission or no admission into the business school for a particular GMAT score.

Difference between logit and probit

Both logit and probit models yields the almost same results. The logit and probit function are increasing and both functions increase relatively quick in the central portion and relatively slow at the extremities and both function lie between 0 and 1. While the only key difference between logit and probit model is in the use of link function

In other words logit model have slightly flatter tails when compared to probit model. 

Comments (2)

Non-linear regression doesn’t refer to a model that isn’t a straight line. For example, we could have a squared term in our regression model, and it is still a linear regression. “Linear” refers not to the nature of the line, but how the coefficients (betas) are estimated. In linear regression, the beta-hats are linear combinations of the response, y.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.