#8 Sklearn - Python package - Model evaluate metrics for regression

    One of the most important steps in building prediction model is evaluating how accurate our model is by the way using metrics to calculate the amount of deviation between predicted value and actual value. In this journal, we will dive into 4 metrics, they are Mean Absolute Error, Mean Square Error, Root Mean Spare Error, and R-squared score. All the functions we use that come from sklearn Python package (from sklearn import metrics).  

    Mean Absolute Error (MAE) is the mean of the absolute value of the errors. In other words, we use the differences of each actual value with predicted value then divide by its number of observations. It measures the average of the residuals in the dataset. That means the weight of all the errors are the same. For example, the difference of first pair yi1 and y^i1 is 5, the difference of second pair yi2 and y^i2 is 1. The average of them will be 3. Then the error will be off by 3. So, "MAE is best used in scenarios where the magnitude of every error is not important". The range value of MAE is from 0 to infinity. The lower score, the better model. It is calculated by the formula below:

Code format: 

from sklearn.metrics import mean_absolute_error

mean_absolute_error(actual value, predicted value)

    Mean Squared Error (MSE) is the mean of the squared errors. MSE is more popular than MAE because it "punishes" the large error. MSE is more useful when the dataset contains a lot of noises, outliers, or unexpected values. It measures the variance of the residuals. For example, the errors are 1, 0, 2, 1, 8 (the result will be 2.4), the square errors in turn are 1, 0, 4, 1, 64 (13.1). Then we can see we have an outlier value here is 8, it makes the square error overall is bigger. The single high value leads the high mean score. So, we can say that the MAE is affected by large deviator or outliers. The range value of MAE is from 0 to infinity. The lower score, the better model.

from sklearn.metrics import mean_squared_error

mean_squared_error(actual value, predicted value)

    Root Mean Squared Error (RMSE) is the square root of the mean of the squared errors. This is a square root of the MSE's result. It measures the standard deviation of residuals. The range value of MAE is from 0 to infinity. The lower score, the better model.

from sklearn.metrics import root_mean_squared_error

root_mean_squared_error(actual value, predicted value)

yi: actual value

y^i: predict value

n: number of observations.

    "Mean Squared Error(MSE) and Root Mean Square Error penalizes the large prediction errors vi-a-vis Mean Absolute Error (MAE). However, RMSE is widely used than MSE to evaluate the performance of the regression model with other random models as it has the same units as the dependent variable (Y-axis)".

    R-Squared score is as known as a coefficient of determination or the testing score. The value of r-squared score indicates how close the linear regression (predicted value) is to the actual value. The range of this score is from 0 to 1. 0 means the predicted value doesn't fit the real value at all, we need to build another model or choose another way to approach the data. 1 means the predicted value fit perfectly with the actual data. The r-squared score can be negative, but it means that the model doesn't follow the trend of the data. That means it's worse than do nothing. 

from sklearn.metrics import r2_score

r2_score(actual value, predicted value) or

modelName.score(X_test, y_test)

Comparing table between MSE, RSME, MAE, and R2

    During the time I study for this journal, I also found the answer for my #7 journal. In #7 journal, I typed model3.score(X_test7, test_pred3) instead of model3.score(X_test7, test_y), that's the reason why I got the result is 1. And because I got the testing score = 1, so I thought model 3 must be the best model. But I was wrong. The actual testing score (also r-squared score) is -5973.93, when the training score is 0.32, that's a type of overfitting model. Thus, our best model in journal #7 is model 1 - linear regression. This answer leaves me a question that linear regression is better with or without polynomial feature? Why the curve is worse than a line in some cases or how?

References

MAE, MSE, RMSE, Coefficient of Determination, Adjusted R Squared — Which Metric is Better? | by Akshita Chugh | Analytics Vidhya | Medium

What is Mean Squared Error, Mean Absolute Error, Root Mean Squared Error and R Squared? - Studytonight