Neurotech Heuristics

Last updated 11/20/24

I started this list a while back after conversations with neurotech-interested friends made me realize some of the heuristics I use to think about the field aren’t always obvious to people outside of it. I ended up finishing it to present as a talk at Devcon’s d/acc day. This list may be fairly basic to most people who have experience in neurotech or neuroscience. I’m constraining “neurotech” to technology intended to interface with the central nervous system.

I expect many of the results I cite will become quickly outdated, and I don’t plan to keep this doc up to date with the state of the field.

Outline

Neurotech is multimodal
Most neurotech is medical
Neuroimaging is more tractable than neuromodulation
Neuroimaging is required for neuromodulation
More invasive doesn’t always = more effective
We don’t need single neuron resolution for most applications
Gene therapy will unlock new applications
We don’t fully understand how neuromodulation works
Unsolved neuroscience questions limit neurotech applications
AI is advancing neuroimaging more than neuromodulation
Opinions

Neurotech is multimodal
I almost didn’t include this one because it felt too basic, but it’s probably worth saying to someone totally new to neurotech. The most widely adopted, well-known neurotechnologies use electrodes, and I think it’s the default mental image most people have of neurotech. For example, Neuralink records neural activity from the motor cortex with electrodes and deep brain stimulation (DBS) devices used for treating a variety of neurological disorders act through stimulating neurons with electric current.

But, neurotech has all the fundamental physical principles and tools of biology at its disposal. In addition to electricity, we can interface with the brain using light, sound, magnetism, activity-dependent gene therapies, etc. I’d expect our concept of neurotech to continue expanding, especially as biological tools improve.

Relatedly, Milan Cvitkovic’s Every way to change your mind helps with this intuition.

The vast majority of neurotech is being developing to treat serious medical conditions and there has been very minimal progress on consumer neurotechnologies for the general public.
A couple notable medical results:

- An ALS patient who lost the ability to speak producing text from his thoughts with help from a neural implant.
- Progress in treatment resistant depression both invasively and non-invasively.
- A paralyzed man is able to walk again.

Reading is more tractable than writing:
Reading (recording neural activity) only requires knowledge of correlation e.g. associating visual cortex activity when you look at a photo of an apple, whereas writing (causing neural activity) requires knowledge of causation e.g. developing a reliable method to cause these neurons in this mm^3 fire at this frequency to create the feeling of x.
Writing generally requires higher resolution than reading. We can fairly accurately reconstruct visual imagery from fMRI data, like in this paper which used 1.8mm^3 voxels (there are about 100k neurons/mm^3). The top images subjects viewed and the bottom are reconstructions from the decoded neural data:

Also neuroscientists regularly decode brain activity without spike sorting, meaning they don’t attribute sets of spikes to specific neurons and don’t know the spatial relationships between spikes they’re recording.

But, recreating that mental imagery or visual percept through neural stimulation, aka writing, seems like a much more challenging task. How many neurons do you need to stimulate and in what ways to create and maintain a stable visual percept? How do you locate the causally relevant neurons? And how do you figure out this stimulation protocol safely? For example, we just need to record from 200 individual neurons to decode what face a macaque is looking at but the “face patch”, the region(s) where neurons selectively respond to variations in facial features, has tens of millions of neurons. Not to say it can’t be done, but I think it’s fair to claim this is harder.

In the case of a more sci-vision, like writing semantic information to the brain, there’s also continual representational drift (what features individual neurons code for and at what firing rates is unstable) that a neurotech device would have to track in order to maintain a functional stimulation protocol.

Another example, we can determine if someone is asleep or measure their anesthetic depth with EEG, a very low resolution recording method, but figuring out what regions to stimulate with what power levels and frequency to make someone fall asleep or enter a deeply anesthetized state is a much harder problem.

You have to read in order to write
To do neuromodulation, you need to identify a target which means you need a method for reading neural data. Either targets are identified prior to neuromodulation with a neuroimaging technology like magnetic resonance imaging (MRI) or a neuromodulation device also has components for reading neural activity live.

Some neurotech modalities can function without reading neural activity live — for example, tFUS patients get an MRI before receiving treatment for target identification. That said, some studies have patients in an fMRI during stimulation, to record the effect of the neuromodulation. Having a method to record activity live would generally improve neuromodulation and can enable closed-loop control.

Deep brain stimulation (DBS) was introduced without reading, but now DBS devices can record activity too, and closed-loop DBS has shown promise for treatment resistant depression.

In cases where neural activity is not being recorded during neuromodulation, behavior or self-report can be used as a readout. This is actually how a lot of brain regions’ functions were originally discovered — neuroscientists would stimulate a region and the patient would report what they felt or they’d have some involuntary behavior that suggested that region’s function. There’s also a rich history of lesion studies where patients with damage to some region of their brain experience impairment to specific functions1.

More invasive doesn’t always = more effective — it depends on your application.
I feel like people often intuit that an invasive brain implant that’s interfacing directly with neurons will be more “advanced” or effective. This is a reasonable and often correct intuition, but I think it depends on your application — if you want to read and write to/from multiple regions simultaneously, be able to flexibly change your target, or only need to write to larger regions, e.g. mm^3, cm^3, non-invasive approaches may work better. An invasive device can only read and directly write to the neurons adjacent to it.2

Some non-invasive neurotech, like tFUS, can interface with the entire brain, and it’s common that non-invasive neuroimaging modalities can image the entire brain or large regions simultaneously like the cortex.

With invasive neurotech, you have to commit to a region(s) that are related to your application. Because neurotech developed for medical use is tailored to that application then neuroscientists can try to get as close to the region of interest as possible.

For example, DBS can effectively symptoms of Parkinson’s, like tremors and impaired mobility, through stimulating the nucleus accumbens.

A DBS probe3:

With the working neurotech at our disposal, this form factor makes sense. A Parkinson’s patient needs their implant all the time and something like a helmet or other non-invasive device may be less practical— not to mention a non-invasive device may have more trouble maintaining an accurate target due to the device repositioning and head movements4.

We don’t need to read/write from individual neurons for most existing applications
Depending on your use case, you may not need to read/write to/from individual neurons or small groups of neurons (small here = sub mm^3 or <100k neurons ). For example, circuit level brain disorders, like the Parkinson’s and depression cases above, may be treatable by just stimulating a general region. Neurons expressing neurotransmitters associated with certain states or functions, like mood or wakefulness, are often clustered together and can be targeted in a relatively low resolution fashion to produce results5.

Gene therapy will unlock new applications
Several neurotech modalities require gene therapy to work in human brains, e.g. optogenetics, magnetogenetics, sonogenetics, and activity-dependent gene therapy. Genetically engineered rodents are an essential experimental model in academic neuroscience research. Gene therapy may begin to blur the definition of neurotechnology — does neurotechnology require electronics?

It seems like there’s some progress in using gene therapy for neurotech development, a company has started from an optical BCI developed at Yale, and some recent results for treating epilepsy with optogenetics in vitro show promise.

My understanding of the state of progress in gene therapy is pretty weak. Gene therapy researchers are mostly focused on enabling gene therapy for serious, often fatal, genetic conditions, typically single point (one base pair) mutations, and they’re still working through serious technical bottlenecks like delivery, off-target effects, and immune response. Clinical trials for these indications also last like ten years, and the long term effects aren’t yet studied since the therapies are so new. Interested in feedback on this one! That said delivery difficulty probably varies a lot by application and location/depth in the brain e.g. a cortical device vs a hippocampal one.

We still don’t fully understand the biological mechanisms of some neuromodulation modalities e.g. tFUS
Also, neuromodulation may have off-target effects that are difficult to account for such as heat generated from a neuromodulation device affecting neural activity.

There are unsolved paradigmatic questions in neuroscience that limit our applications of neurotech. But, neurotech could also help us answer them.
For example:

- Memory:
  Many sci-fi depictions of neurotech involve transferring knowledge into someone’s brain, e.g. instantly learning an entire language, or removing memories, like in Eternal Sunshine of the Spotless Mind. As far as I know, we have an incomplete understanding of memory and memory engrams.
  
  There’s an active field of memory engram research, often using optogenetics (genetically editing neurons to fire when hit with light), that suggests we can create false memories, erase memories, and activate memories. The field generally agrees that synaptic connections associated with different memories are distributed across the brain so activating or removing specific memories may be incredibly technically challenging — implying you’d need a BCI that can read and write to single neurons.6
- Neural code:
  “If we had an advanced BCI, we wouldn’t know how to use it” – Doris Tsao
  
  Solving “the neural code”, or understanding the brain’s coding principles, aka what does neural activity actually mean, is arguably the biggest open problem in neuroscience7. There are scientists studying this question from the level of dendritic computation to the topology of the activity of large brain regions 8.
- Consciousness:
  Are there physical principles that describe the conditions in which matter has subjective experience? Experimental physics has a rich history of contributing to theoretical developments, but we’ve barely scratched the surface in studying the physics corresponding to subjective experience. Better neurotechnology will likely contribute to this research program.

Next-generation neurotech may not be developed in neurotech/neuroscience labs
Neurotech is fundamentally about measuring and perturbing matter, and researchers develop tools to do this who aren’t focused on understanding the brain.

AI is advancing neuroimaging more than neuromodulation
AI is being widely applied to decode large neural data sets, which could mean neuroimaging improves more rapidly than neuromodulation which is comparatively more bottlenecked by biology and hardware progress than software.

Opinions

Neuroimaging will need really low latency to be a compelling consumer product
I think the dream of non-invasive neuroimaging is closed-loop, low latency, high-bandwidth computer interaction — thought to speech, dynamic visualization of mental imagery, cursor control, basically any computer command. To achieve the “dream”, you need lower latency than hemodynamic response which takes a couple seconds. fMRI, functional ultrasound, and FNIRS use hemodynamic response, leaving electric and magnetic readouts as candidates — neither of which have yet achieved the necessary resolution, coverage, and form factor. For context, a “motion-to-photon” latency of less than 20 ms is standard for VR/AR. Motion-to-photon is the delay between a user’s physical movement and the display’s updating to reflect that movement. You could probably get by with slower, and I’m noting this to consider what specs may be compelling enough replace existing tech.

BCIs should outperform existing tools.
For example, if you have functioning vision, why get a brain implant that functions as a visual prosthetic when you can just look at a screen or use VR? Why get a BCI that generates auditory experience when you can wear airpods?

There has yet to be a successful consumer neurotech product and the consumer applications haven’t been proven-out scientifically

Narratives
I don’t like the feeling of “machine goes into the brain”, but I really love “mind goes into the world” or “mind expands”, enabled by neurotech. Outside of the medical sphere, the prevailing sci-fi neurotech narrative in tech circles is “merging with AI” — popularized by Elon/Neuralink. The merge with AI story often feels motivated by fear of obsolescence, which may be a valid fear, but I guess I feel more inspired and curious about what makes human life so beautiful and fulfilling and whether neurotechnology could protect and enhance those aspects of our experience. I’d be happy to have technology that enriches interpersonal interactions like enhancing empathy, communication, and self-expression.

Thanks to Raffi Hotter, Quintin Frerichs, Laura Deming, and Milan Cvitkovic for feedback and suggestions and to David Wong-Campos, Jacques Carolan, and Janine Leger for conversations that influenced this.

Oliver Sacks’ books, such as The Man Who Mistook His Wife for a Hat and Hallucinations, are great examples of how brain disorders or injuries affect experience written for a general audience. ↩︎
This is a little more complicated since rhythmic stimulation in one region can entrain large groups of neurons, and there are brain-wide networks that you may be able to affect from stimulating one region.
↩︎
Epilepsy probes look very similar ↩︎
I don’t think we are near the ceiling on invasive methods, and maybe there are a few brain regions that are highly leveraged for driving changes in capabilities, perception, knowledge etc. and in theory a neurosurgeon could add more implants until the set of desired applications are covered. Health risks may limit this although in most medical scenarios there’s a cost-benefit negotiation a patient has to do.
↩︎
An exception with medical relevance may be vision where single neurons code for specific visual features. Existing cortical implants for treating blindness generally aim to help blind patients identify the presence of larger objects in order to navigate rather than recreate a visual scene in resolution of healthy vision (although perhaps this is a north star). There are also retinal implants in development for vision restoration — these require gene therapy to make retinal cells sensitive to light.
↩︎
There is some work on memory enhancement with neurotech
↩︎
Random but while looking around online for people defining this problem, I thought this Neural Codes course description did so in a way that was exciting/ambitious. (I don’t think I agree with every claim about the implications of understanding the neural code in there) ↩︎
From https://www.neurreps.org: “An emerging set of findings in sensory and motor neuroscience is beginning to illuminate a new paradigm for understanding the neural code. Across sensory and motor regions of the brain, neural circuits are found to mirror the geometric and topological structure of the systems they represent—either in their synaptic structure, or in the implicit manifold generated by their activity. This phenomenon can be observed in the circuit of neurons representing head direction in the fly (Kim et al. (2017); Wolff et al. (2015); Chaudhuri et al. (2019)), in the activities of grid cells (Gardner et al. (2022)), and in the low-dimensional manifold structure observed in motor cortex (Gallego et al. (2017)). This suggests a general computational strategy that is employed throughout the brain to preserve the geometric structure of data throughout stages of information processing.”
↩︎