Visual Image Reconstruction using fMRI

Mind-reading is an exciting topic. Modern fMRI(functional magnetic resonance imaging) technique are making mind-reading coming true. Using fMRI combine with modern machine learning algrithoms(MVPA), we can now decode some infomation from people’s brain activities.

I repeated some previous prominence research about that topic in Badong Chen’s lab, such like fMRI signal decoding from primary visual cortex (V1) and reconstructing pattern image from fMRI signal (Miyawaki et al., 2008). Moreover, I improved the accuracy of the reconstruction by ~ 4% (from about 80% to about 84%), using the improved MVPA method with SVM classifier (Norman et al., 2006).

Introduction

Mind-reading is an exciting topic. Modern fMRI(functional magnetic resonance imaging) technique are making mind-reading coming true. Using fMRI combine with modern machine learning algrithoms(MVPA), we can now decode some infomation from people’s brain activities.

Visual_Img_Rec

I repeated some previous prominence research about that topic in Badong Chen’s lab, such like fMRI signal decoding from primary visual cortex (V1) and reconstructing pattern image from fMRI signal (Miyawaki et al., 2008). Moreover, I improved the accuracy of the reconstruction by ~ 4% (from about 80% to about 84%), using the improved MVPA method with SVM classifier (Norman et al., 2006).

This article is only a berif introduction of my work. You can contact me for the source code and more details.

Results

Subjects viewed 432 12x12 binary(Black or White) pixels image while recording their brain activities in Visual Cortex using fMRI. 352 of these image were in random. 80 were in regular (The pattern of them are resemble like numbers, cross, etc.), see figure below.

100

Figure 1. Left, binary image is in random. Right, binary image is in regular

I then used GLM procedures to model brain activity for every stimulus.

Firstly, I used MVPA(Multi-Voxels pattern analyze) to build a decoding model for every pixel (1x1 scale).

We then averaged the vaule of 2 neighbouring pixels vertically (2x1 scale), horizentally (1x2 scale) and 4 neighbouring pixels (2x2 scale). Figure shown below.(Miyawaki et al., 2008)

...

Figure 2. Multi-scale local image bases for buliding MVPA model

In MVPA feature selection for each pixel model, I firstly choosed 3037 voxels from V1, then caculated Pearson’s r between the activity of voxels and stimulus value (0 or 1). Then sort these voxels based on r vaule cauculated. We choosed 50 voxels from it(top 30 and last 20). Then we trained a SVM classifier using 352 random images as a trainning set. Note that each pixel has its own model(classifier and voxel selected), we overall have 12x12 models. The code example shown below.

1
function [ SVMStruct,corvox ] = singlevoxtrai(trainingsetnum,posi,beta,stim) %This model train a SVM model to a given posi of the stim % trainingsetnum=1:176; % posi=[6,6]; %% section a,trainnning one single voxal trainingset=beta(:,trainingsetnum); [~,selectvox,~] = voxcor(posi(1,1),posi(1,2),stim(:,:,... trainingsetnum),trainingset);%select the corresponding voxels of the stim Training=beta(selectvox,trainingsetnum)'; Group=reshape(stim(posi(1,1),posi(1,2),:),size(stim,3),1); Group=Group(trainingsetnum,:); SVMStruct = svmtrain(Training,Group,'kernel_function','polynomial'); corvox=selectvox; end

Then we used the remaining 80 example images to test our models using the MVPA models we trainned before, We computed the pixels’ vaule given brain activities by running the classifier.

Repeat this procedure to every pixels in 4 scale. After combining 4 scale results using liner regression for each pixels. Finally, we got a reconstructed image based on subject’s brain activities. We compute deviation between the reconstruced image and the actual image (square-root error). Example shown below.


Figure 3. The demo video of the reconstructed image

Main References

  • Miyawaki, Yoichi, et al. “Visual image reconstruction from human brain activity using a combination of multiscale local image decoders..” Neuron60.5(2008):915-29.

  • Norman, K. A et al.,(2006). Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn Sci, 10(9), 424-430. doi:10.1016/j.tics.2006.07.005