Inferring the average face someone has seen

This post discusses how to infer the average face someone has seen (in this case, a macaque). It’s a follow-up to Translating visual experience between brains, where I review Chang and Tsao 2017, and probably won’t make sense without reading one of those.

To decode faces from face patch activity, Chang and Tsao 2017 averaged all the faces they presented to macaques and used this average as the origin of their face space. The origin may not have been the true average face macaques had encoded, but it was close enough that face decoding appeared pretty accurate.

This made me wonder if you could figure out a given macaque’s average “face” even if you didn’t know what faces it had been previously exposed to. I don’t think calling this its “prior” is technically accurate, but analogizing the origin of a face space contributing to my curiousity about this question.  

If you’re lacking an average face (or the training data from which you’d calculate the average), it could be roughly determined using the same tools Chang and Tsao employed to predict faces. This could be a case that may more readily apply to humans, or less controlled settings. If two humans participate in your study, you wouldn’t know the values of their respective “origins” in face space a priori. 

Let’s assume almost the same experimental setup as Chang and Tsao except that macaque B has participated in previous face patch studies using different datasets.

The experimenter doesn’t know what faces macaque B has been previously exposed to, and it’s possible they could be largely different from this study’s dataset. For example, perhaps macaque B had previously been exposed to faces that were 90% one gender. 

So while we’d expect neurons in both macaque’s face patches to have ramp-shaped tuning, the neurons’ firing rates may be relative to different averages. For example, if 90% of the faces macaque A had been exposed to were feminine, then masculine facial features may elicit higher frequency spiking than if macaque B had been exposed to an even split of masculine and feminine faces.

 This is because features are measured relative to the average, so macaque A viewing masculine facial features would be a large deviation from its average face.

A friend speculated that the face patch, and other regions with similar coding principles, likely adapts its principal components to maximize discriminatory power — in other words, the brain is trying to figure out what facial feature variations help it best discern the differences between faces.

To find the average face for a macaque, you can assume an average, calculate the error between the actual face and the predicted face, then iteratively test new averages that reduce this error.

Here’s how this would work in a scenario where you starting out knowing macaque A’s average face but lack macaque B’s:

Revisiting the equation from Chang and Tsao’s face decoding paper: