Binary classification, a fundamental concept in machine learning, involves categorizing data into one of two distinct classes. Like predicting whether an email is spam or not, whether an image is of a cat or not, or whether a tumor is malignant or benign.

In this blog post, we will explore the concept of binary classification, focusing on logistic regression. We will cover the mathematical foundations, practical implementation, and provide interactive examples to illustrate the concepts.

Generated with xAI Grok

Binary Classification?

Let's say we have an $64 \times 64$ image of a cat and we want to classify it as either a cat or not a cat. This is a binary classification problem, where the two classes are "cat" and "not cat". The goal is to develop a model that can accurately predict the class of new images based on the features extracted from the images.

Every image can be represented as a vector of pixel values. For example, a $64 \times 64$ image can be represented as a vector of $64 \times 64 \times 3 = 12288$ values, where each value represents the intensity of a pixel in one of the three color channels (red, green, blue).

So, the feature vector for a $64 \times 64$ image can be represented as:

where $x_i$ is the intensity of the $i^{th}$ pixel in the image. Let $n = n_x = 12288$ be the number of features in our dataset. The feature vector $\mathbf{x}$ is a $n \times 1$ vector, where $n$ is the number of features (pixel values). In this case, $n = 12288$.

and the target variable can be represented as:

so if we have $m$ training examples in our dataset, we can represent the training set as:

where $\mathbf{X}\in \mathbb{R}^{n_x \times m}$ is the feature matrix with X.shape=(n_x, m), and $m$ is the number of training examples. The target variable can be represented as:

where $\mathbf{y} \in \mathbb{R}^{1 \times m}$ is the target variable vector with Y.shape=(1, m), representing the output labels for each training example.

Logistic Regression

Logistic regression is a statistical method used for binary classification. It models the probability that a given input belongs to a particular class. The logistic regression model uses the logistic function (also known as the sigmoid function) to map any real-valued number into the range of 0 and 1.

In simple words, Given $\mathbf{x}$(the input image), we want $\hat{y} = \mathbf{P}(y=1|\mathbf{x})$ or what is the probability that the image is a cat given the pixel values.

here, $\mathbf{x} \in \mathbb{R}^{n_x}$ and $0 \le \hat{y} \le 1$, and we have some extra parameters $w \in \mathbb{R}^{n_x}$ and $b \in \mathbb{R}$, where $w$ is the weight vector and $b$ is the bias term.

Output,

is the sigmoid function. The output $\hat{y}$ is interpreted as the probability that the input $\mathbf{x}$ belongs to class 1 (cat). If $\hat{y} \ge 0.5$, we classify the input as class 1 (cat), otherwise we classify it as class 0 (not cat).

This is the basic idea of logistic regression. In the next post, we will discuss the cost function, gradient descent optimization, and implement a complete logistic regression model from scratch.

Stay tuned for the next part of this series!