By Natalia Bilenko and Valkyrie Savage, UC Berkeley, Spring 2016
One human subject underwent fMRI brain scanning while viewing a complex natural movie. Voxel-wise modeling was used to estimate a forward encoding model for each location in visual cortex (see Nishimoto et al., 2011). For each one second segment of the movie, the encoding model was used in decoding mode to identify, from a library of 5000 hours of random video, 100 random clips most likely to be similar to the original clip that elicited the measured brain activity. The top 5 clips from these 100 were then selected using histograms of gradient features, image gradient similarity, and SIFT-flow features across time. These clips were averaged to produce the final reconstruction. This video includes [top left] the original movie shown that the subject saw, [top right] the final reconstructed movie, [bottom left] the image gradients in the original movie, and [bottom right] the image gradients in the reconstruction. The reconstruction is remarkably similar to the original movie.