Linear Algebra for Data Scientists — Explained with NumPy

Linear Algebra for Data Scientists — Explained with NumPy

The core concepts and practices of linear algebra.

Image by author

Machine learning and deep learning models are data-hungry. The performance of them is highly dependent on the amount of data. Thus, we tend to collect as much data as possible in order to build a robust and accurate model. Data is collected in many different formats from numbers to images, from text to sound waves. However, we need to convert the data to numbers in order to analyze and model it.

It is not enough just to convert data to scalars (single numbers). As the amount of data increases, the operations done with scalars start to be inefficient. We need vectorized or matrix operations to make computations efficiently. That’s where linear algebra comes into play.

Linear algebra is one of the most important topics in data science domain. In this post, we will cover the basic concepts in linear algebra with examples using NumPy .

NumPy is a scientific computing library for Python and forms the basis of many libraries such as Pandas.

Types of Objects in Linear Algebra

Types of objects (or data structures) in linear algebra:

• Scalar: Single number
• Vector: Array of numbers
• Matrix: 2-dimensional array of numbers
• Tensor: N-dimensional array of numbers where n > 2

A scalar is just a number. It can be used in vectorized operations as we will see in the following examples.

A vector is an array of numbers. For instance, following is a vector with 5 elements:

We can use scalars in vectorized operations. The specified operation is done on each element of the vector and scalar.

A matrix is a 2-dimensional vector.

It seems like a pandas dataframe with rows and columns. Actually, pandas dataframes are converted to matrices and then fed into machine learning models.

A tensor is an N-dimensional array of numbers where N is greater than 2. Tensors are mostly used in deep learning models where the input data is 3-dimensional.

It is hard easy to represent with numbers but think of T as 3 matrices with a shape of 3x2.

The shape method can be used to check the shape of a numpy array.

The size of an array is calculated by multiplying the size in each dimension.

Common Matrix Terms

A matrix is called square if number of rows is equal to the number of columns. thus, the matrix A above is a square matrix.

Identity matrix,denoted as I, is a square matrix that have 1’s on the diagonal and 0’s at all other positions. Identity function of NumPy can be used create identity matrices of any size.

What makes an identity matrix special is that it does not change a matrix when multiplied. In this sense, it is similar to number 1 in real numbers. We will do examples with identity matrix in matrix multiplication part of this post.

The inverse of a matrix is the matrix that gives the identity matrix when multiplied with the original matrix.

Not every matrix has an inverse. If matrix A has an inverse, then it is called invertible or non-singular.

Dot Product and Matrix Multiplication

Dot product and matrix multiplication are the building blocks of complex machine learning and deep learning models so it is highly valuable to have a comprehensive understanding of them.

The dot product of two vectors is the sum of the products of elements with regards to their position. The first element of the first vector is multiplied by the first element of the second vector and so on. The sum of these products is the dot product. The function to compute dot product in NumPy is dot() .

Let’s first create two simple vectors in the form of numpy arrays and calculate the dot product.

The dot product is calculated as (1*2)+(2*4)+(3*6) which is 28.

Since we multiply elements at the same positions, the two vectors must have same length in order to have a dot product.

In the field of data science, we mostly deal with matrices. A matrix is a bunch of row and column vectors combined in a structured way. Thus, multiplication of two matrices involves many dot product operations of vectors. It will be more clear when we go over some examples. Let’s first create two 2x2 matrices with NumPy.

A 2x2 matrix has 2 rows and 2 columns. Index of rows and columns start with 0. For instance, the first row of A (row with index 0) is the array of [4,2]. The first column of A is the array of [4,0]. The element at first row and first column is 4.

We can access individual rows, columns, or elements as follows:

These are important concepts to comprehend matrix multiplication.

Multiplication of two matrices involves dot products between rows of first matrix and columns of the second matrix. The first step is the dot product between the first row of A and the first column of B. The result of this dot product is the element of resulting matrix at position [0,0] (i.e. first row, first column).

So the resulting matrix, C, will have a (4*4) + (2*1) at the first row and first column. C[0,0] = 18.

The next step is the dot product of the first row of A and the second column of B.

C will have a (4*0) + (2*4) at the first row and second column. C[0,1] = 8.

First row A is complete so we start on the second row of A and follow the same steps.

C will have a (0*4) + (3*1) at the second row and first column. C[1,0] = 3.

The final step is the dot product between the second row of A and the second column of B.

C will have a (0*0) + (3*4) at the second row and second column. C[1,1] = 12.

We have seen how it is done step-by-step. All of these operations are done with a np.dot operation:

As you may recall, we have mentioned that identity matrix does not change a matrix when multiplied. Let’s do an example.

We have also mentioned that when a matrix is multiplied by its inverse, the result is the identity matrix. Let’s first create a matrix and find its inverse. We can use linalg.inv() function of NumPy to find the inverse of a matrix.

Let’s multiply B with its inverse matrix, C :

Bingo! We have the identity matrix.

As we recall from vector dot products, two vectors must have the same length in order to have a dot product . Each dot product operation in matrix multiplication must follow this rule. Dot products are done between the rows of the first matrix and the columns of the second matrix. Thus, the rows of the first matrix and columns of the second matrix must have the same length.

The requirement for matrix multiplication is that the number of columns of the first matrix must be equal to the number of rows of the second matrix.

For instance, we can multiply a 3x2 matrix with a 2x3 matrix.

The shape of the resulting matrix will be 3x3 because we are doing 3 dot product operations for each row of A and A has 3 rows. An easy way to determine the shape of the resulting matrix is to take the number of rows from the first one and the number of columns from the second one:

• 3x2 and 2x3 multiplication returns 3x3
• 3x2 and 2x2 multiplication returns 3x2
• 2x4 and 4x3 multiplication returns 2x3