Skip to main content

A discussion of causal inference on fMRI data

Author

Russ Poldrack

Published

December 19, 2013

This was originally posted on Blogger here.

My facebook page was recently home to an interesting discussion about causal inference that was spurred by a recent article that I retweeted:

Gary Williams ‏@orestesmantra16 Dec
Effective connectivity or just plumbing? Granger Causality estimates highly reliable maps of venous drainage. http://feedly.com/e/EzJAf9px
Retweeted by Russ Poldrack

What follows is the discussion that ensued. I have moved it to my blog so that additional people can participate, and so that the discussion will be openly accessible (after removing a few less relevant comments). Please feel free to comment with additional thoughts about the discussion.

15 comments captured from the original post

daniele · 2013-12-19

Thanks for the fascinating discussion.

As shown by Lionel Barnett and Anil Seth in a recent paper, the problem is not HRV per se, but HRF together with downsampling.
Furthermore, the problem of generating a good ground truth is a though one.
We have developed a method for retrieving and deconvolving the HRF for each voxel even at rest

http://www.ncbi.nlm.nih.gov/pubmed/23422254

and this seems to improve the performance of GC.

We used it also to get whole brain GC maps from deconvolved data, which can be compared to the paper by Webb et al

http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0073670

I wrote Olivier David asking him his data from his PLOS Biology 2008 paper to see whether I can get GC to perform better.

tal · 2013-12-20

Lastly, I think it’s kind of unfair to characterize this as a debate between people on the machine learning side who favor exploratory data analysis but don’t really care about hypotheses, and people who think deeply about hypothesis teseting but are willing to take a couple of steps beyond the data. It’s a nice rhetorical move, because it allows you to write off half of the field as people with philosophical differences, but I don’t think this is about philosophical differences at all. One can think deeply about theoretical models and still recognize that just because a model makes sense in your head, and produces results you like when explicitly compared with several other models, does not mean that it approximates the truth in any meaningful sense. This is not a philosophical claim, it’s an empirical one that can be directly tested (and has been tested). A major problem with much of the DCM literature (and the causal modeling literature more generally) is that there is very little emphasis on validation beyond what makes theoretical sense to the authors–which isn’t really validation at all. *That* is the substantive criticism you see from many people in the above thread. I imagine that when you look at your model–which DCM reports has better evidence than at least a few other alternatives–you see an elegant theoretical story supported by strong evidence, and you probably view concerns about the quality of the underlying data as minor annoyances that aren’t your responsibility, and that future studies can address. The problem is that many other people who look at your results will immediately think of all the extremely strong assumptions you have to make in order to believe the results but that are almost certain to be false, and which you could have easily tested but didn’t.

The key point is that this really *isn’t* some deep philosophical divide between two incommensurable camps; it’s something that causal modeling proponents have the power to address. All it would take to bridge whatever divide exists is for people in the causal modeling camp to get more serious about external validation. This means two things, as far as I’m concerned: (a) testing the assumptions underlying DCM in cases where it’s straightforward to do so (e.g., by simply repeating your analyses several times with slightly different parameter choices); and (b) as Niko said, actually predicting novel observations in an unbiased way (i.e., showing that the DCM model with the highest evidence also consistently predicts new observations we care about better than models with lower evidence). If you can show me in your papers that your results aren’t entirely dependent on parameter choices that were clearly not under your control in any meaningful sense, and that your method is robust against very simple validity threats (e.g., random models having better evidence, as in Lohmann et al), then I will be happy to take them seriously. But if you can’t or aren’t willing to provide that kind of support, then I don’t see why I (or anyone else) should put any stock in the findings given all of the enormous assumptions involved. (4/4)

tal · 2013-12-20

Mind you, all this still depends on yet another very strong, and in my view largely indefensible, assumption, which is that a brain region that plays an important causal role in the network has to show a univariate difference between experimental conditions. Frankly this strikes me as wishful thinking. For one thing, many absolutely crucial causal effects may be driven by very transient changes in psychological state space that don’t translate into robust BOLD signals (i.e., they would be invisible to fMRI). We have no reason to think that fMRI is in a position to capture all, or even most, of the causal dynamics crucial to understanding what a network is doing, do we? Moreover, each voxel contains hundreds of thousands or millions of neurons, and a typical ROI may contain hundreds of millions. We know from electrophysiological studies that it is entirely typical to find interdigitated assemblies of neurons with quite different response properties within roughly the same chunk of tissue; it is not at all a stretch to think that a particular region might show no net change in BOLD signal even though vitally important causal interactions are taking place (e.g., two different populations of prefrontal neurons are involved in recurrent activation of different kinds of sensory representations in posterior cortex). Note that I’m not arguing that all of the signals must sum to a net of exactly zero when comparing two experimental conditions. I’m saying that as far as I’m aware, we have no good reason at all to expect anything like a strong linear relationship between the causal centrality of a region and its activation strength in a mass univariate analysis. And we have even less reason to expect that such a relationship would manifest in a readily interpretable way–which is what you require when you say that it is the researcher’s job to look at their experimental results and pick out the nodes that matter. I think this is giving researchers entirely too much credit–not to mention some very extraordinary powers of apprehension.

This of course all entirely leaves aside the Lohmann et al critique, which in my view carries exactly the force Lohmann et al suggested, and still hasn’t been rebutted in a convincing way. It’s no good to say that it’s the researcher’s job to select the right models to test (which was the gist of Friston’s response) if there’s no evidence that the method is capable of discriminating between models reliably to begin with. What Lohmann et al showed was that DCM cannot reliably distinguish families of models that are neuroscientifically plausible from those that are wildly implausible; in view of that result, why should anyone expect that DCM can discriminate between two models that *are* both neuroscientifically plausible–a discrimination that can only be more difficult? When you say that “With good experimental design I think DCM is largely robust to the small differences”, what are you basing that on? The assertion flies in the face of the Lohmann simulations, all of the considerations I raised above, and of many people’s experience tweaking DCM models for a very long time to try to get a stable result. It isn’t the kind of statement the purveyors and consumers of a sophisticated causal modeling technique that rests on numerous strong assumptions should just take for granted–especially when it’s so easy to empirically test. Or, to put it this way: how do you (and I mean you personally) actually *know* that your results are robust, if you haven’t done the work to test their resilience to small changes, or compared them to what you would get by generating models at random? Isn’t this a giant leap of faith? Isn’t it your responsibility to first verify for yourself that your results don’t disappear when you tweak key parameters before publishing them? (3/4)

tal · 2013-12-20

Note that this is exactly the same omitted variable issue that presents such trenchant problems for structural models in *any* domain; imaging is not special. The key point, which is almost invariably ignored, is that a given causal interpretation of a model is only sensible if one is willing to assume that all and only the relevant nodes are included in the network. In behavioral research this is already almost always an untenable assumption because it isn’t hard to think of confounding variables for almost any measured variable that is supposed to carry causal influence. But it gets much worse when you’re talking about neuroimaging, where the brain is a very dense causal system and we know for a fact that most of the nodes selected for DCM analysis correlate quite strongly with many other voxels or ROIs that were left out (typically for arbitrary reasons). To put it in perspective, this is kind of like constructing a simple causal model positing that the positive correlation between food intake on body mass is mediated by fat consumption, and then when someone pops up and politely interjects that fat consumption is heavily confounded with carb consumption, protein consumption, and any number of other things, simply saying “well sure, but we didn’t model those things, so oh well” and continuing along. When you build a model that includes, say, ACC and DLPFC, but not inferior parietal cortex, this is only an okay approximation if you have strong reason to think that IPC is not actually doing a lot of the work you’re attributing to ACC or DLPFC (with which it will be strongly correlated) AND you also know that including IPC wouldn’t fundamentally change the causal dynamics between the other regions (and frankly, I don’t know how you’d know that).

Even if you set the omitted variable issue aside–which, again, is not a small one, and for many people is a deal-breaker even in the behavioral literature–you still haven’t really addressed the issue of node definition: if you haven’t quantified the impact of, say, including or dropping an extra 10% of the voxels in each ROI based on a slight juggling of the threshold (which, again, you can’t argue is anything other than arbitrary!), what reason do I have to believe your results? Again, it’s not that this is some far-fetched hypothetical concern that could, under just the right circumstances, present some smalll problem. We have both theoretical reasons and empirical reasons to suspect that very small perturbations of the nodes will have very large effects on the model. It’s not an acceptable defense to say that, well, it’s the researcher’s job to get things right. Surely it’s also the researcher’s job to show the people evaluating the research that the results don’t completely fall apart when very minor, and essentially arbitrary, parameters are tweaked. And again, it’s not really up for debate that there *is* a large amount of arbitrariness, because as I understand it you’re defining the nodes using spheres centered around the mass univariate activation peaks, and I’m quite sure that if you look at the confidence intervals of all your voxels, you will find that the peak voxels would have been quite different with a different sample (plus the diameter of the sphere will make a big difference too). (2/4)

tal · 2013-12-20

Russ, you really need to raise the character limit on comments; 4,096 ain’t enough–this isn’t Twitter! ;)

Micah, I don’t really think it’s fair to summarize people’s concerns here by saying that “you can find examples where DCM for fMRI may perform poorly”. A better way to summarize them is that the space of possible models is so unimaginably huge, our understanding of what the brain is doing is so poor, and there are so many choices a researcher needs to make that are essentially arbitrary, that it’s almost inconceivable that there could be a method for reliably evaluating the (observational) evidence for causal models. This wouldn’t be such a terrible thing if there was a serious movement to try and validate the methods before putting them into use, but in practice there doesn’t be. The kind of simulations that Lohmann et al did, for instance, strike me as absolutely critical, and is the kind of thing one should do *before* a method becomes widely applied, not several years later.

To respond to your specific comments, I’m certainly not saying that the definition of nodes in the model is totally arbitrary; what I’m saying is that *much of it* is arbitrary, and that there is essentially no reason to think that just by blind luck we will ever get it right (not *perfectly* right; I mean even close to right). Let’s take again your data as an example. You say your model has 6 nodes because those 6 survived group-level correction. But surely you agree that if your sample size were twice as large, many more regions would survive correction. And if your sample size were halved, fewer would. This is not up for debate; it’s a necessary consequence of the null hypothesis significance testing framework. If we take your argument seriously that DCM should only be applied to models constrained by the results of a mass univariate analysis where regions activate in response to experimental manipulations, what are we now to make of this? Is it really reasonable to say that the 6-node model you selected happens to be the right one, and is a decent approximation to the truth, even thought we know full well that a doubling of sample size would have given you perhaps a 15-node model, and a halving might have perhaps given you a 4-node model? And that every time you re-run the experiment, you’re likely to get somewhat different nodes? (1/4; continued below.)

Anonymous · 2013-12-20

Just wanted to try and wrap up my thoughts on this debate, as I never really responded to Tals comments, which I found very useful. In general, I am sure you can find examples where DCM for fMRI may perform poorly; the analysis you suggest would inevitably point out places were it is robust and places were it is not. It certainly should not be surprising that iterating design and analysis alters the outcome. I think in cases where the HRF peforms poorly, the winning model is likely to have an extremely poor variance explained. I don’t buy into their skepticism that the assumptions of the HRF don’t scale with more complex models. We already have reasonably robust evidence that it does, and I thik it’s not enough just to be skeptical - I can see no reason in principle why the example with 2 nodes is any different than 6 nodes. Even the GLM can be a poor model in some cases; the point isn’t to produce a perfect method but rather a good compromise between model utility and accuracy. With good experimental design I think DCM is largely robust to the small differences. I do agree with Tal that a large scale data mining approach iterating all possible variables would be interesting, but I don’t agree with his totally arbitrary definition of biologically relevant activations. I get that if you add subjects infinitely the entire brain activates, but that doesn’t say much about biological interpretability as you’ve obviously overpowered the design. Solid designs with reasonable Ns typically report relatively constrained activation profiles that fit anatomy quite well. So i’d welcome the exhaustive approach he recommends, but I am also not sure what we would really learn beyond ‘the way you do things changes what you get’. There is a side of science that sees this as a drawback but I agree with viewpoint that this is where the responsibility of the scientist to understand their method comes into play.

In general it comes down to two (sadly) competing views of science. On one, the goal is to get endless amounts of data, very generally defined, and to apply high powered machine learning and data mining routines to let the data ‘reveal the truth’. The other more classical view advocates careful model development and hypothesis testing, on the notion that good theories will produce robust results without the need for endless data, and understanding that many of the findings produce by that approach are just rich descriptions that make few (if any) predictions about behavior. I think the best result comes from fusing these two views, letting exploratory data mining inform our hypothesis and using a tightly hypothesized approach to bang out principles. To the extent that our theories make strong, testable predictions, we should be able to tease out and replicate effects reasonably well. From my point of view and experience, predictive coding makes extremely powerful predictions about the brain and behavior that are increasingly being borne out by dynamic causal modelling. I appreciate the power of the data mining approach but in the end I doubt we’re ready to remove the ever fallible human from the process ;)

Anonymous · 2013-12-21

Last thought - it is very easy to download about 500 task related motor activation data sets from HCP and do DCM on them. Why don’t we collaborate to do so and make the exact analysis you are proposing? I have a colleague who did this in 300 participants viewing heider-simmle illusion and showed that modulation of forward vs backward by more complex stimuli was extremely consistent across the population. One draw back however is that with DCM we cannot compare models with different VOI data. Anyway, the data is out there for us to play with if you like…

Anonymous · 2013-12-21

Oh damn, my comments went in backwards. That is what I get for arguing on the internet first thing in the morning :(

Anonymous · 2013-12-21

I recall from Friston’s respons to Lohmann that he pointed out the simulation is largely based on a misunderstanding of the aims and mechanics of Bayesian model selection. By using this procedure you assume that all models have an a priori equivalent prior. Your argument is that the selection failed to discriminate plausible from obviously implausible models; if that model is so obviously implausible then you should specify a different prior for it, in which case it would be rejected. You also seem to be ignoring Friston’s argument (which I have directly observed to be true) that with increasing model space you have increasing model dilution, and a higher difficulty of selecting the best model. Lastly I hope I haven’t given you the idea that i’ve just put my data into a model and pushed some buttons. My convinctions come from working in a non-FIL group that does almost exclusively MMN/oddball work. I’ve seen in 3 different datasets from three different colleagues across EEG, MEG, and fMRI the exact same out come - namely that oddballs elicit activation in primary sensory, salience, and middle/inferior frontal regions; and secondary that the comparison of deviants to standards elicits a massive change in frontal vs all backwards connections. This is what I mean by having a theory that makes strong directional hypothesis.

The oddball task is extensively computationally validated, with a rich multimodal literature, and is thus very well suited for DCM. I just don’t want you to get the idea that I am blindingly advocating DCM because I happened to get a nice result in one analysis. I am aware that DCM can also be a fragile thing and is probably not ready for mass consumption or fluffy social neuroscience ‘models’. I also simultaneously agree with the underlying neural complexity you advocate, and believe that it is largely irrelevant to fMRI. I’ve run some 300 fMRI scans personally over 4 different experiments and see time and again that mostly what we can resolve is large-scale interactions between the major macroscopic networks (salience, dmn, control, sensory). I’m skeptical of anything beyond this level of inference, which is why I say fMRI is just a basic starting point for more advanced modelling. Anyway, I am not being dismissive, your points are well taken and I will get back to you more fully on this issue. If anything the debate itself has greatly enhanced my understanding of what is at stake here.

Happy holidays,
Micah (2/2)

Anonymous · 2013-12-21

Hey Tal-

Great response. I’m on vacation now in Italy so it will be a while before I can formulate a full response. To be clear, I don’t think anyone wants to argue against the validation approach you suggest (hard to argue for more data or analysis), but I just can’t help but feel like things are not quite as arbitrary as you suggest. I think there are some cases where we have much stronger priors - but you argue that these more basic examples are too ‘toy’ to be interesting. I guess we agree then that, fMRI isn’t much use for actual biophysical modelling or complex paradigms, but we don’t agree what can be done at the lower more conservative end of the scale. I do just want to touch on the issue of statistical significance. On the one hand i’m not a big fan of NHST, and like DCM largely because it introduces a Bayesian model selection procedure to users who might otherwise not encounter it.

But it seems like abuse of statistics to suggest that the fact that our retained activations changes drastically based on the statistical threshold suggests that the underlying phenomenon are just as arbitrary. Statistical thresholds are just conventions, and I don’t see why (if using NHST) we should be increasing or decreasing thresholds in the arbitrary way you suggest, given that the consensus seems to be approximately a pFWE < 0.05 (peak or cluster). I’m also not sure how much farther we can really continue this debate without pulling out some data because there is a lot of assertion of opinion on both sides. You say that most of my activations probably have CIs overlapping zero; I say that my activations provide almost localizer-specific regional activations of all a priori expected areas. We’re talking about a very simple high/low somatosensory stimulation with deviants and standards, not some theory of mind task. In pilots and at the subject level, the group level results are almost spatially identical with the subject level. There is very little variation in this response and it fits quite well with the literature. So I think we have quite strong priors to take a reasonable approximation of those areas into a causal model. (1/2)

Anonymous · 2013-12-23

Hi this is Anil Seth. What an excellent debate and I hope I can add few quick thoughts of my own since this is an issue close to my heart (no pub intended re vascular confounds).

First, back to the Webb et al paper. They indeed show that a vascular confound may affect GC-FMRI but only in the resting state and given suboptimal TR and averaging over diverse datasets. Indeed I suspect that their autoregressive models may be poorly fit so that the results rather reflect a sort-of mental chronometry a la Menon, rather than GC per se.

In any case the more successful applications of GC-fMRI are those that compare experimental conditions or correlate GC with some behavioural variable (see e.g. Wen et al. http://www.ncbi.nlm.nih.gov/pubmed/22279213). In these cases hemodynamic and vascular confounds may subtract out.

Interpreting findings like these means remembering that GC is a description of the data (i.e. DIRECTED FUNCTIONAL connectivity) and is not a direct claim about the underlying causal mechanism (e.g. like DCM, which is a measure of EFFECTIVE connectivity). Therefore (model light) GC and (model heavy) DCM are to a large extent asking and answering different questions, and to set them in direct opposition is to misunderstand this basic point. Karl, Ros Moran, and I make these points in a recent review (http://www.ncbi.nlm.nih.gov/pubmed/23265964).

Of course both methods are complex and ‘garbage in garbage out’ applies: naive application of either is likely to be misleading or worse. Indeed the indirect nature of fMRI BOLD means that causal inference will be very hard. But this doesn’t mean we shouldn’t try. We need to move to network descriptions in order to get beyond the neo-phrenology of functional localization. And so I am pleased to see recent developments in both DCM and GC for fMRI. For the latter, with Barnett and Chorley I have shown that GC-FMRI is INVARIANT to hemodynamic convolution given fast sampling and low noise (http://www.ncbi.nlm.nih.gov/pubmed/23036449). This counterintuitive finding defuses a major objection to GC-fMRI and has been established both in theory, and in a range of simulations of increasing biophysical detail. With the development of low-TR multiband sequences, this means there is renewed hope for GC-fMRI in practice, especially when executed in an appropriate experimental design. Barnett and I have also just released a major new GC software which avoids separate estimation of full and reduced AR models, avoiding a serious source of bias afflicting previous approaches (http://www.ncbi.nlm.nih.gov/pubmed/24200508).

Overall I am hopeful that we can move beyond premature rejection of promising methods on the grounds they fail when applied without appropriate data or sufficient care. This applies to both GC and fMRI. These are hard problems but we will get there.

daniele · 2013-12-25

I am a bit concerned about how GC was computed in that paper:
-they computed pairwise GC, that especially for datasets composed of many short time series will result in a high number of false positives.
-why did they consider only regions with a high asymmetry in GC? zero GC balance could result from no interaction or by high outflow plus high inflow, which are quite different scenarios.

Using our approach to retrieve and deconvolve HRF from resting state, I plotted the HRF shape on the brain:

http://figshare.com/articles/HRF_parameter/886139

looks like that regions with higher, slower and wider HRF are those that you identify as GC sinks (venous overlap), and this would explain that GC maps obtained after deconvolution (and with a conditioned approach) are different from the ones they find (Wu et al. in the references of the paper).

Anonymous · 2013-12-27

Excellent discussion, with some great arguments, a number of insights for me, but also a deja vu or two.

Concerning the initial post: I wonder about the argumentation… Showing that a method has high sensitivity for A (here: large vasculature) does not in any way establish it lacks the specificity to pick up B (here: neuronal interactions). You could also use almost any ICA variant on the same data to find the same (or even better?) vasculature maps. Does that establish that ICA cannot be used for rs-fMRI analysis to find neuronal resting state networks? Obviously (to me, at least): no, it doesn’t establish that.

From a historical perspective, when we first worked on Granger Causality Mapping (GCM) around 2004 (http://www.ncbi.nlm.nih.gov/pubmed/15734358; thanks for your appreciation for the work, Russ!) we wanted to answer pretty much the following questions

i) What could GCM add, why do it at all?
Our best answer: It adds structural model exploration that can (should?) precede confirmatory connectivity analysis, such as DCM. Confirmatory analysis selects between hypotheses or models containing a few areas (and for fMRI each hypothesis contains the same areas). It runs the risk of selecting one that compares poorly with many non-tested models that contain 1 or (many) more crucial missing regions (the missing region problem). Moreover: a causal mapping approach can correctly identify the boundaries between differently interacting nodes, which might have been lumped together based on a thresholded GLM, and can identify nodes that did not show up in a GLM at all. We have re-iterated this point in http://www.ncbi.nlm.nih.gov/pubmed/19786106, and after that (and Karl’s kind agreeing reply: http://www.ncbi.nlm.nih.gov/pubmed/19770049), methods to allow DCM to test models with more than 3-4 regions have started to appear. I am happy to now see FIL lab affiliates (read: Micah) say things like ‘a previous limitation has simply been that it’s incredibly frustrating and time consuming to estimate interesting models with more than 2 or 3 nodes’ and ‘DCM requires strong hypothesis, which is both it’s strength and weakness, and fits extremely well with exploratory data-driven methods’ agreeing with our original point almost 10 years ago. Likewise it is interesting to see in this discussion (e.g.: Tal’s posts) that the missing region problem is still one of the chief concerns scientists have in using confirmatory approaches.

ii) Can we pick up 10-100ms neuronal (conduction and processing) delays through 1000’s ms hemodynamics delays and in the presence of remote vasculature with 100’s ms delays?
Our best answer came from first testing this in simulation and was, perhaps surprisingly: yes. However, anything with more delay is picked up with more statistical power (more sensitivity, see Figure 3 in http://www.ncbi.nlm.nih.gov/pubmed/15734358). Shorter TRs (as in the HCP data) increase power altogether and separate long and short delays more effectively. From that fact and from seeing early ‘raw’ GCMs abundant in large vasculature, as expected, we devised a strategy to filter out vasculature: ‘instantaneously- filtered’ GCMs. In short: short delays components (such neuronal interactions) show up both in instantaneous correlations and G-causal (delayed) terms; large delay components (such as vasculature) show up almost exclusively in the G-causal terms. So, filter out delayed components that do not have an instantaneous correlation term. That proved very effective. Conversely: if you really want to look at vascular delays, unfiltered maps are a very sensitive instrument, which is what the cited PLoS ONE study seems to have done. No surprise here. And, if it is the study target, mapping out large vasculature can be very useful, e.g. as a danger-zone map for other methods.
(1/2)

Anonymous · 2014-01-06

Btw: Niko, to your comment on crossvalidation of connectivity model fits on independent test sets. This has been done by Jason Smith and Barry Hortwitz, see e.g.: http://www.ncbi.nlm.nih.gov/pubmed/19969092 . The switching linear dynamic system essentially learns a different AR connectivity model for each experimental condition from the training data. Then it predicts experimental conditions at each time point in the unseen test data. This has two more important side implications for the discussion. 1) experimental design and modulation have it again. 2) The prediction level on the test set indeed works as a global measure of fit immune to overfitting. We could accept, say, 80% correct as a minimum to accept any model if we were so inclined.

Anonymous · 2014-01-06

iii) How do we deal with the varying HRF confounds?
Our best answer: Always (at the very least) look for experimental modulation of G-causality. Period. We have made and re-made this point many times, but it is still often missed. This also lets experimental manipulations drive causal understanding, as they should, rather than just assumptions added to statistics that come from the data. Model based deconvolution can be an additional tool here but, as we discussed in http://www.ncbi.nlm.nih.gov/pubmed/19786106, this is yet another model-based endeavor and inaccurate HRF models can lead to new errors rather than remove old ones (see Daniel’s posts above). Incidentally, we wrote this work mainly as a reply to Olivier David’s study (cited by Marta above) to argue there are a few gross oversights in that study: first: they did not use experimental modulation in applying G-causality  and, second, when they added the deconvolution model of DCM to G-causality it performed perfectly against the gold-standard, unlike DCM which underperformed, even on 3 regions (Marta: Why cite this study to say GC does not perform well!?). Interestingly, even in deconvolution modeling, one can pit more or less parametric models against each other. As Pedro-valdes Sosa, Anil Seth and myself summarized in http://jmlr.org/proceedings/papers/v12/roebroeck11/roebroeck11.pdf, various AR and ARMA models (used for G-causality) have invariances that theoretically render them unaffected by things like varying HRF convolution kernels. Anil’s nice work subsequently showed this can indeed be the case for simple AR models if one samples fast enough (see Anil’s post above).
I second (third, fourth?) Niko’s call for a causal analysis competition. And, yes, the generating model must be realistic in the detail that we know and agree on, whether we a priori believe it will be relevant detail or not. Not being close to the estimating model, as Niko posed, is one important aspect. Actually containing the aspects we *know* to be present in the brain (such as 10ms conduction delays) rather than assuming these will not matter anyway, is another important aspect. To illustrate: when Steve Smith showed me the manuscript of the network modeling approaches paper, I told him I thought the conclusion that the sensitivity of G-causality methods is low is misleading. Steve used the DCM model for fMRI to generate the data, which does not have any neuronal delays. It can’t, it doesn’t use delays (in fMRI). The model necessarily puts any observed delays in the hemodynamics in the inversion. It has been designed to do that. So, if you generate data with it, you are generating data with only instantaneous dependencies. It is obvious connectivity models tuned to detect instantaneous dependencies or that use them in their causal modeling will do well on these simulations. And that those tuned to delayed interactions will (correctly) show nothing. I told Steve I expected the results to be different if, instead, he would use the neuronal DCM model for EEG/MEG with suitable neuronal delays (then chained with the DCM hemodynamic model, of course). Steve said he did not think it would make a difference. I still think it will, and I think Anil’s paper has shown as much with an even more complex neuronal simulation model with delays.
I, for one, am still simultaneously critical and enthusiastic about both GCM and DCM. The methods currently seem to be converging as each is taking up the strengths of the other. I agree external validation is important and should be a new norm: test the same (set of) model(s) with different modalities over several studies. But still both will be potentially very useful if used carefully and quite dangerous if used indiscriminately. Just like any other even moderately complex method. The thing is, I think that if we chose not to use them at all we would lose some of our best current instruments to help understand the wildly complex multiscale non-linearly coupled network that is the brain…
(2/2)