Sebastian Michelmann

@s-michelmann.bsky.social

865 Followers 498 Following

Cognitive neuroscientist (Assistant Professor at NYU), human episodic memory, M/EEG, ECoG, and behavior. How do we reinstate temporally dynamic, information-rich memories?

(e)HPC CRM component plotted on a 2-D grid as a function of position in the maze. Warmer colors appear away from the home box of the animal (indicated by a black arrow); the home box is located at X position = 600 cm and Y position = 250 cm. (f) Same as e, but components computed with CCA. No relationship with the maze-position is apparent, consistent with CCA components being dominated by 60 Hz noise

Shown are spectrograms of LFPs recorded in the HPC (left, a) and mPFC (middle, b) from awake and behaving rats solving a spatial navigation task (compare Rosenblum 2025). The weights computed with CRM for the data shown in a/b are plotted in (blue/orange) as a function of frequency. Insets show the same weights computed with CCA. CCA weights peak at a shared noise frequency around 60 Hz, CRM weights have negative values in the lower theta band and positive values in the beta band. CRM weights have no peak around 60 Hz

Representational Similarity Matrix between all trials of watching videos in Reagh et al (2023). Correlations across components (indicated by warm colors) are high between trials of the same context (e.g., Café 1 or supermarket 2), but not between trials of different context within the same schema (e.g., Café 1 and Café 2)

Representational Similarity Matrix between all trials of watching videos in Reagh et al (2023). Correlations (indicated by warm colors) are high between trials of the same schema (Café or supermarket)

Panel a of the graphic shows illustrations of vector embeddings that are mapped onto high-frequency activity in iEEG during naturalistic story listening. Results from CRM and CCA are compared. Both CRM and CCA maximize the cross-covariance between vector embeddings and neural activity. Only CRM additionally constrains the cross-covariance between neural activity and the vector embedding of the previous words to zero. Panel b shows mean Pearson correlations with standard error (across 9 patients) in held-out data for CCA (purple) and CRM (orange), averaged across 10 folds. Correlations were computed at lags relative to word onset. Correlations for both methods increase towards word onset; CCA yields positive correlations up to 4 s before onset, potentially reflecting prediction of upcoming words and/or shared contextual information between embeddings within the same story. CRM yields positive correlations from ~1 s before up to ~ 2 s after onset, indicating that word-specific representations are present in neural activity during this window. Only the CRM-derived correlations can be interpreted as predictions of word-level representations. Panel c shows component loadings on iEEG channels for CCA (top), CRM (middle), and their difference (bottom). Warm colors in the difference map indicate stronger contribution to CCA; blue indicates stronger contribution to CRM. Only channels where the absolute difference exceeded 0.05 are plotted exceeded 0.05 are plotted. Component loadings on channels show a broadly similar distribution of the neural representation for CCA (top) and CRM (middle), with higher loadings for CCA in early auditory processing regions and higher loadings for CRM in higher-order auditory processing regions (e.g., posterior STS and right IFG, bottom)

Conceptual overview of CRM with illustrative pictograms of neuroimaging methods, neural network models, and different species: Researchers collecting two observations with repeated measures (e.g., any pair of continuous multi-channel recordings from different brain regions in human or animal recordings, different subjects, vector embeddings in neural network models, or multivariate behavioral data, etc.) can ask if the same information is represented in both time series. Three example multivariate series of observations are illustrated (red labels). If the relationship between pattern 1 and pattern 2 is of interest, the pairwise linear relationships can be captured in the cross-covariance matrix between the multivariate time series (CXY, orange). Canonical correlation analysis (CCA) can transform them, so that resulting components are maximally correlated. However, shared information can stem from confounding variables and common noise sources. A third series of observations (pattern 3) may share the confounding signal of no interest and can be used to estimate the pairwise linear relationships of confounding signal (DXY, yellow). Canonical Representational Mapping leverages this control-covariance to separate neural representations from noise and disentangle overlapping representational patterns. I.e., akin to CCA, CRM maximizes the correlation between two multivariate time series; however, it constrains the covariance with a matched control time series (DXY) to zero. Examples of control time series are lag-shifted data (e.g., a time-series of word-embeddings of the previous word, which shares the same context but not the same content), data filtered in a noise band (e.g., data that has been band-pass filtered around the line-noise), or pseudo-randomized data, where the correspondence of interest is shuffled but the correspondence of confounds is kept (e.g., trials are shuffled within a block to separate item representations from block-specific effects)

The left side of the image depicts the correlation between PSD in hippocampus and reinstatement of the upcoming state as correlation values in the format of a spectrogram. A significant negative correlation is shown in a cluster spanning 5-12 Hz between 760 and 619 ms, i.e., lower power before state transitions in the hippocampus predicted more evidence for reinstatement throughout the upcoming state. The right side of the image shows Gaussian Copula Mutual Information (GCMI) between hippocampal channels and CR-channels at different hippocampal leads (negative lag; x-axis) and at different times before the state-transition (y-axis) in the form of a 2-D image. The maximum cluster of increased GCMI is displayed, where a significant GCMI between hippocampal and CR-channels is present. It shows that at and before the state transition there is enhanced shared information between hippocampus and cortex with hippocampal spectrograms predicting cortical spectrograms

The image depicts neural underpinnings of state-transitions during memory-scanning. On the left, changes in Power Spectral Density (PSD) on CR-channels at state-transitions are shown as a cluster within a spectrogram. A significant decrease in PSD is visible in the cluster that spans 3-30Hz between 220 ms before and 180 ms after state-transitions. The middle part of the plot depicts the spectrogram that shows a significant decrease in PSD on hippocampal channels in a cluster spanning 3-30Hz between 760 and 510 ms before state-transitions. The right part of the plot displays the location of the cortical reinstatement (CR) channels in blue and the hippocampal channels in red

Figure depicting that neural states are reinstated during the naturalistic interview and unfurl in a forward direction during memory-scanning. On the left, cortical reinstatement channels are plotted on a transparent brain. The are located in regions that overlap substantially with the default mode network (DMN); specifically, 42 of CR-channels are located closest to the DMN defined in the 7 Network solution defined by Yeo et al., 16 are closest to the somatomotor, 11 to the frontoparietal, 6 to the dorsal attention, 4 to the visual network 3 to the ventral attention, and 2 to the limbic network. The right side displays the average state-pattern by memory-scanning correlation matrix that has been scaled to 10 state-patterns and 40 time-points. The inset represents the same correlation matrix scaled to 2 state-patterns and 2 time-points. The correlation matrix shows that early state-patterns from the scanned segment correlate highly with early memory-scanning times while later state-patterns correlate highly with late memory-scanning times

Figure depicting that neural state segmentation tracks shared event perception. The left side shows an example of the neural state segmentation on a single channel. White lines are overlaid on time by time correlation matrices of slow components; these lines delineate neural state boundaries. Overlaid colored lines delineate event boundaries defined by the behavioral norming sample on corresponding parts of the movie. The right side shows the grand average z-score of agreement from a behavioral segmentation locked to neural state boundaries (the average of averages across channels and patients with SEM across patients). Colored dots indicate points of significant increase controlling the false discovery rate in purple and derived from a cluster permutation in green. The agreement peaks 1261 ms after neural state-boundaries (which is zero on the x-axis in the plot)

Figure depicting the experimental paradigm. Patients watched the two movie parts on a laptop in the hospital while undergoing intracranial electroencephalography recording for clinical purposes. After each movie, the experimenter asked naturalistic interview questions that consisted of a setup and a target. The setup described a scene in the video, the target asked about a later scene. The wording of the questions was crafted to induce forward memory scanning. Patients then searched their memory and answered by describing the correct scene verbally. The time between the description of the target and the first word in the patient's answer marks the memory-scanning period. The figure shows a patient in a hospital bed next to an interviewer. Speech bubbles illustrate the described procedure

Distribution of distance to the consensus solution for human data and GPT-derived segmentation. Violin plots show the distribution of distances to the consensus for individual human button presses in the first (a) and second (b) behavioral run of the “Pieman” story, and for the story “Monkey in the Middle” (c). The thick gray bar represents the interquartile range, the red dot is the mean, the white line is the median, and the thin gray line displays the extremes of the distribution excluding outliers. Colored dots represent the distance of the GPT-3-derived solution to the consensus, with (orange) and without (green) the long qualifier

Figure: Vertical lines are GPT-3-derived boundaries (5 replications in different colors) overlaid on average (top) and individual (bottom) human button presses.