NumPy

In this lesson, we'll learn about ways to represent and manipulate images in Python. By the end of this lesson, students will be able to:

• Apply ndarray arithmetic and logical operators with numbers and other arrays.
• Analyze the shape of an ndarray and index into a multidimensional array.
• Apply arithmetic operators, indexing, and slicing to manipulate RGB images.

We'll need two new modules: imageio, which provides utilities to read and write images in Python, and numpy, which provides the data structures for representing images in Python.

import imageio.v3 as iio
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

Reading an image

Let's use imread to load a color picture of Dubs II in grayscale with mode="L", which means "luminance" or "lightness". To show an image, we can plot its pixels using the matplotlib function imshow.

dubs = iio.imread("dubs.jpg", mode="L")
plt.imshow(dubs, cmap="gray")

<matplotlib.image.AxesImage at 0x7993a54f9790>

Pandas uses NumPy to represent a Series of values, so many element-wise operations should seem familiar. In fact, we can load an image into a Pandas DataFrame and see that this grayscale image is really a 2-d array of color values ranging from [0, 255].

dubs

array([[238, 238, 239, ..., 232, 232, 232],
       [238, 239, 239, ..., 232, 232, 232],
       [239, 239, 239, ..., 232, 232, 232],
       ...,
       [240, 240, 240, ..., 236, 236, 236],
       [240, 240, 240, ..., 236, 236, 236],
       [240, 240, 240, ..., 236, 236, 236]], dtype=uint8)

# How does your phone camera process an image?
#   Computational Photography
pd.DataFrame(dubs)

	0	1	2	3	4	5	6	7	8	9	...	630	631	632	633	634	635	636	637	638	639
0	238	238	239	239	240	240	241	241	240	240	...	231	231	232	232	232	232	232	232	232	232
1	238	239	239	239	240	240	241	241	240	240	...	231	231	232	232	232	232	232	232	232	232
2	239	239	239	240	240	240	240	241	240	240	...	231	231	232	232	232	232	232	232	232	232
3	239	239	240	240	240	240	240	241	240	240	...	231	231	232	232	232	232	232	232	232	232
4	240	240	240	240	240	240	240	240	240	240	...	231	231	232	232	232	232	232	232	232	232
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
355	240	240	240	240	240	240	240	240	240	240	...	237	237	237	237	237	237	237	237	237	237
356	240	240	240	240	240	240	240	240	240	240	...	237	237	236	236	236	236	236	236	236	236
357	240	240	240	240	240	240	240	240	240	240	...	237	237	236	236	236	236	236	236	236	236
358	240	240	240	240	240	240	240	240	240	240	...	237	237	236	236	236	236	236	236	236	236
359	240	240	240	240	240	240	240	240	240	240	...	237	237	236	236	236	236	236	236	236	236

360 rows × 640 columns

What would a color image of Dubs II look like instead? Let's try loading the picture without mode="L" to maintain its color data.

dubs = iio.imread("dubs.jpg")
plt.imshow(dubs)

<matplotlib.image.AxesImage at 0x7993a540efd0>

What do you think the colorful DataFrame should look like?

pd.DataFrame(dubs)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], line 1
----> 1 pd.DataFrame(dubs)

File /opt/conda/lib/python3.11/site-packages/pandas/core/frame.py:827, in DataFrame.__init__(self, data, index, columns, dtype, copy)
    816         mgr = dict_to_mgr(
    817             # error: Item "ndarray" of "Union[ndarray, Series, Index]" has no
    818             # attribute "name"
   (...)
    824             copy=_copy,
    825         )
    826     else:
--> 827         mgr = ndarray_to_mgr(
    828             data,
    829             index,
    830             columns,
    831             dtype=dtype,
    832             copy=copy,
    833             typ=manager,
    834         )
    836 # For data is list-like, or Iterable (will consume into list)
    837 elif is_list_like(data):

File /opt/conda/lib/python3.11/site-packages/pandas/core/internals/construction.py:314, in ndarray_to_mgr(values, index, columns, dtype, copy, typ)
    308     _copy = (
    309         copy_on_sanitize
    310         if (dtype is None or astype_is_view(values.dtype, dtype))
    311         else False
    312     )
    313     values = np.array(values, copy=_copy)
--> 314     values = _ensure_2d(values)
    316 else:
    317     # by definition an array here
    318     # the dtypes will be coerced to a single dtype
    319     values = _prep_ndarraylike(values, copy=copy_on_sanitize)

File /opt/conda/lib/python3.11/site-packages/pandas/core/internals/construction.py:592, in _ensure_2d(values)
    590     values = values.reshape((values.shape[0], 1))
    591 elif values.ndim != 2:
--> 592     raise ValueError(f"Must pass 2-d input. shape={values.shape}")
    593 return values

ValueError: Must pass 2-d input. shape=(360, 640, 3)

# Is the grayscale image just the average of the color points?
# Human perception is not just a pure average of red-green-blue because
#   our eyes sense reds, greens, and blues with different intensity
dubs

array([[[235, 238, 245],
        [235, 238, 245],
        [236, 239, 246],
        ...,
        [229, 231, 243],
        [229, 231, 243],
        [229, 231, 243]],

       [[235, 238, 245],
        [236, 239, 246],
        [236, 239, 246],
        ...,
        [229, 231, 243],
        [229, 231, 243],
        [229, 231, 243]],

       [[236, 239, 246],
        [236, 239, 246],
        [236, 239, 246],
        ...,
        [229, 231, 243],
        [229, 231, 243],
        [229, 231, 243]],

       ...,

       [[237, 240, 247],
        [237, 240, 247],
        [237, 240, 247],
        ...,
        [233, 235, 247],
        [233, 235, 247],
        [233, 235, 247]],

       [[237, 240, 247],
        [237, 240, 247],
        [237, 240, 247],
        ...,
        [233, 235, 247],
        [233, 235, 247],
        [233, 235, 247]],

       [[237, 240, 247],
        [237, 240, 247],
        [237, 240, 247],
        ...,
        [233, 235, 247],
        [233, 235, 247],
        [233, 235, 247]]], dtype=uint8)

# All the red colors from the image
dubs[:, :, 0]
dubs[..., 0]

array([[235, 235, 236, ..., 229, 229, 229],
       [235, 236, 236, ..., 229, 229, 229],
       [236, 236, 236, ..., 229, 229, 229],
       ...,
       [237, 237, 237, ..., 233, 233, 233],
       [237, 237, 237, ..., 233, 233, 233],
       [237, 237, 237, ..., 233, 233, 233]], dtype=uint8)

...

Ellipsis

None

Array manipulation

Images are represented in Python with the type numpy.ndarray or "n-dimensional array." Grayscale images are 2-dimensional arrays with pixel luminance values indicated in each position. Color images are 3-dimensional arrays with pixel color values indicated for each channel (red, green, blue) in each position. How can we set the left and right sides of this picture to 0 so that Dubs II appears surrounded by black borders?

dubs.shape

(360, 640, 3)

dubs = iio.imread("dubs.jpg")
dubs[:, :50] = 0
dubs[:, -50:] = 0
plt.imshow(dubs)

<matplotlib.image.AxesImage at 0x7993965e13d0>

dubs = iio.imread("dubs.jpg")
dubs[:, -50:50] = 0
plt.imshow(dubs)

<matplotlib.image.AxesImage at 0x799396347910>

dubs = iio.imread("dubs.jpg")
dubs[:, 50:-50] = 0
plt.imshow(dubs)

<matplotlib.image.AxesImage at 0x7993964b4350>

dubs = iio.imread("dubs.jpg")
dubs[:, :50] = dubs[:, -50:] = [128] * 3
plt.imshow(dubs)

<matplotlib.image.AxesImage at 0x7993963855d0>

dubs = iio.imread("dubs.jpg")
dubs[:, :50] = 0, dubs[:, -50:] = 0
plt.imshow(dubs)

  Cell In[31], line 2
    dubs[:, :50] = 0, dubs[:, -50:] = 0
    ^
SyntaxError: cannot assign to subscript here. Maybe you meant '==' instead of '='?

dubs[:50, :25]
[1, 2, 1]

array([[[128, 128, 128],
        [128, 128, 128],
        [128, 128, 128],
        ...,
        [128, 128, 128],
        [128, 128, 128],
        [128, 128, 128]],

       [[128, 128, 128],
        [128, 128, 128],
        [128, 128, 128],
        ...,
        [128, 128, 128],
        [128, 128, 128],
        [128, 128, 128]],

       [[128, 128, 128],
        [128, 128, 128],
        [128, 128, 128],
        ...,
        [128, 128, 128],
        [128, 128, 128],
        [128, 128, 128]],

       ...,

       [[128, 128, 128],
        [128, 128, 128],
        [128, 128, 128],
        ...,
        [128, 128, 128],
        [128, 128, 128],
        [128, 128, 128]],

       [[128, 128, 128],
        [128, 128, 128],
        [128, 128, 128],
        ...,
        [128, 128, 128],
        [128, 128, 128],
        [128, 128, 128]],

       [[128, 128, 128],
        [128, 128, 128],
        [128, 128, 128],
        ...,
        [128, 128, 128],
        [128, 128, 128],
        [128, 128, 128]]], dtype=uint8)

When we're performing an assignment on 2-dimensions of a 3-dimensional image, NumPy follows broadcasting rules to evaluate the operation. The simplest version of broadcasting are just element-wise operations.

plt.imshow(dubs)

<matplotlib.image.AxesImage at 0x799396454c10>

Let's try a more complicated example. Using the floor division operator, fill in the imshow call to decrease only the green channel so that the overall picture is much more purple than before.

dubs = iio.imread("dubs.jpg")
plt.imshow(dubs // [1, 2, 1])

<matplotlib.image.AxesImage at 0x799396471cd0>

dubs = iio.imread("dubs.jpg")
dubs[:, :, 1] = dubs[:, :, 1] // 2
plt.imshow(dubs)

<matplotlib.image.AxesImage at 0x7993967f7b90>

dubs[:, :, 1] // 2

array([[119, 119, 119, ..., 115, 115, 115],
       [119, 119, 119, ..., 115, 115, 115],
       [119, 119, 119, ..., 115, 115, 115],
       ...,
       [120, 120, 120, ..., 117, 117, 117],
       [120, 120, 120, ..., 117, 117, 117],
       [120, 120, 120, ..., 117, 117, 117]], dtype=uint8)

Practice: Instafade

Write code to apply a fading filter to the image. The fading filter reduces all color values to 77% intensity and then adds 38.25 to each resulting color value. (These numbers are somewhat arbitrarily chosen to get the desired effect.)

The provided code converts the dog array from integer values to floating-point decimal values. To display the final image, the code converts the numbers in the dog array back to uint8 before passing the result to imshow.

dog = iio.imread("dog.jpg").astype("float32")
plt.imshow(dog.astype("uint8"))

<matplotlib.image.AxesImage at 0x79939635bbd0>

dog = iio.imread("dog.jpg").astype("float32")
# Reduces all color values to 77% intensity
#   then add 38.25 to each resulting color value.
dog = dog * .77 + 38.25
plt.imshow(dog.astype("uint8"))

<matplotlib.image.AxesImage at 0x799396600550>

Practice: Gotham

Write code to apply the following operations to an image.

1. Expand the red colors by 50% by subtracting 128 from each red channel value, multiply the result by 1.5, and then add 128 to restore the original value range.
2. Increase the blue colors by 13 by adding 13 to each blue channel value.
3. Add black letterboxing bars by setting the top 150 and bottom 150 pixels to black.
4. Clip color values outside the range [0, 255] by reassign all values above 255 to 255 and all values below 0 to 0.

dog = iio.imread("dog.jpg").astype("float32")
dog[..., 0] = (dog[..., 0] - 128) * 1.5 + 128
dog[..., 2] += 13
dog[:150] = dog[-150:] = 0
dog[dog > 255] = 255
dog[dog < 0] = 0
plt.imshow(dog.astype("uint8"))

<matplotlib.image.AxesImage at 0x7993964e68d0>

Optional: Advanced broadcasting

What is the result of adding the following two arrays together following the broadcasting rules?

x = np.array([[1], [2], [3]])
x

y = np.array([1, 2, 3])
y