Logistic Regression

The sigmoid function

To turn a real-valued score into a probability we use

$$\sigma(z) = \frac{1}{1 + e^{-z}}, \qquad P(y = 1 \mid \mathbf{x}) = \sigma(\boldsymbol\theta^\top \mathbf{x}).$$

Cross-entropy loss

$$J(\boldsymbol\theta) = -\frac{1}{n}\sum_{i=1}^{n} \Big[ y^{(i)} \log \hat{y}^{(i)} + (1 - y^{(i)}) \log (1 - \hat{y}^{(i)}) \Big]$$

Its gradient has the same form as in linear regression: $\nabla J = \frac{1}{n} X^\top(\hat{\mathbf{y}} - \mathbf{y})$.

In scikit-learn

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1_000)
model.fit(X_train, y_train)
print(model.score(X_test, y_test))