Is this project useful? No. Is it inspiring? Also no. But is it fun? Yeah, I think so.

Two years ago I stumbled upon an article, which stated that the authors were able to identify anonymous participants of the MRI research project, using MRI brain scans and face-recognition software (Schwarz et. al). This paper attracted a lot of attention and was cited in an article by the Wall Street Journal, stating that:

The finding draws attention to a privacy threat that will increase with technology improvements and the growth of health-care data, experts in medical imaging and facial recognition said.

I was impressed, that reconstruction can be done so easily, with a few lines of code and some freely available software. Since it seemed quite manageable to achieve, I decided to try it out on my own data. I have received a couple of MRI brain scans in my life, acquired in a standard clinical setting.

My aim was to reconstruct my face from the MRI image of my brain that I possess, and perform facial recognition with my real-life photos.

Photo by National Cancer Institute on Unsplash

Let’s dive into the fascinating world of MRI and facial recognition and find out, why researchers, imaging companies, and hospitals need to be very attentive to the preservation of the personal information of their patients.

Why do MRI images of the brain preserve facial features?

Magnetic resonance imaging (MRI) is a well-developed, non-invasive imaging technique, which is used for the creation of high-resolution images of organs and tissues. In the clinical routine, if a patient shows symptoms of cancer, epilepsy, or any neurologic disorder, a brain MRI can be prescribed. Images in MRI are created by recording the signal, which is created based on nuclear magnetic resonance (NMR), and which geometrical location in space is encoded using gradient magnetic fields. In the clinical routine, MRI is based on the excitation of protons of the hydrogen nuclei in tissue, since it produces high-quality anatomical images due to the high gyromagnetic ratio of protons and their large amount in biological tissues, especially in water and fat.

Speaking extremely simplified, water (ca. 40%) and fat (ca. 60%) are the main components of brains (humans are cucumbers with anxiety ©). Of course, biology is more difficult than that. However, NMR and MRI do not care much about all the different components. If you put a human in the scanner, the strong magnetic field will see protons, and will not care much, if they are in the brain, in the nose, or in the cheeks. That’s why, when a radiologist is making a preliminary scan to determine the precise location of your brain, the full face is recorded as well. And if it is recorded, it can be also visualized.

My brain MRI

How can it be visualized? The idea is the following: since MRI records signals from every position of the face, and the signal is proportional to the number of hydrogen atoms in this location, we could try to reconstruct the surface, based on the signal from every point of the MRI image. The pipeline that I used to reconstruct my face from the T1 scan was as follows:

1. Get the MRI scan done (for me, it was done for medical reasons).
2. Retrieve the DICOM files of the MRI image on your PC. DICOM is the image format, usually used in the clinical routine.
3. Transfer DICOM into NIFTI (it is much easier to work with NIFTIs since Python and Matlab have packages to work with the .nii format). You can also visualize NIFTIs with the software ITK-Snap, among others.
4. Perform image denoising with Matlab (i.e. with the MainMRIDenoising function by Pierrick Coupe). Some images in MRI suffer from such artifacts, as aliasing. In this case, this error has to be corrected by placing missing information in the right spot.
5. Transfer .nii file into .surf.gii, which represents the surface reconstruction. This operation can be done with i.e. nii_nii2gii function, written for Matlab (by Chris Rorden’s Lab). Important parameters for the creation of the surface are the threshold for the isosurface, reduceFrac, which represents the mesh simplification factor, and smooth, which leads to the blurring of the image.
6. The newly created .surf.gii file is further converted to a Gifti. This allows us to see the properly reconstructed surface, which can be visualized with Matlab, or other software (i.e. SurfIce, created for neuroscience purposes).

This way, from the MRI image, after some transformations, the following image is acquired. Looks real, huh?

Reconstruction of my face from the MRI, shown above

Let’s CODE!

Now I will show you how well the reconstructed image overlaps with my real-life photo (see below). For that, I used face_recognitionanddlib packages from Python. The code below is adopted from here, here, and here. To perform face recognition and determine facial features, a pre-trained deep learning model from dlib is used.

My real-life picture to be compared to the MRI reconstructed image

Let’s look for faces in the pictures.

Face detection in the real and the reconstructed image

Let’s also draw detected facial features in the images.

Facial landmarks in the two pictures

While in the real photo the landmarks are positioned correctly, in the reconstructed image the face shape is not perfectly delineated. More work can be done in this aspect since the recognition appears to be very sensitive to the lighting of the reconstructed image.

Here you find the link to the GitHub repo with the full Jupiter Notebook.

If you never thought about it, should you be concerned about your medical data now?

If you are unsure about the privacy of your data, take care to understand, how the data are protected, and how they are being used. Do not sell your data, and consider carefully, what you are granting permission to do with your data! Data-sharing contracts exist exactly for this purpose. However, in case you are being treated in the hospital, keep in mind, that numerous instruments exist, that protect your privacy. These instruments are extremely well thought through. Also, the aim of data acquisition in the hospital is the improvement of your health.

The same Wall Street Journal article states the following:

Privacy threats from MRIs are likely limited, however, and the study’s test of facial-recognition software avoided some harder challenges found in everyday life, Mr. Schwarz and other researchers who weren’t involved in the study said.

“The risk to the typical patient is really small,” said Eliot Siegel, a radiology professor at the University of Maryland School of Medicine, who has studied the issue but wasn’t involved in the recent research.

Researchers typically pledge not to try to identify medical study participants under data-sharing contracts, he said. Facial-recognition software also likely wouldn’t work as well when searching for matches across thousands of random photos instead of dozens of images.

A number of instruments exist to perform face removal (defacing) and anonymization of medical images, which are routinely applied. For scientific studies, funding is given only if proper data protection is guaranteed. So do not be too alarmed! New instruments for face removal are also constantly being developed. In 2021 Schwarz et al. published another paper, suggesting new approaches to facial feature removal, and active research is performed both in the reduction of face recognition from medical scans, and preservation of image quality.

Sources

Schwarz, Christopher G., et al. “Identification of anonymous MRI research participants with face-recognition software.” New England Journal of Medicine 381.17 (2019): 1684–1686.