What are the best ways to improve a model

[renecotyfanboy]

Hello,

First, I would like to thank all of you for this awesome package, simulation-based inference is some sort of game changer in my research field and I would be glad to contribute to the diffusion of these methods.

What I don't grasp yet are the best ways to improve the modelisation of the likelihood to derive quantitive results. I have various ideas in mind but don't know what I should start with:

• At which point getting more simulations in the dataset could help?
• Should I focus on getting better summary statistics or training an embedding net?
• Should I change the density estimator internal architecture (adding more flows or anything?)

[michaeldeistler]

Hi there,

thanks for starting this discussion!

Overall, I think your questions are quite open-ended; I will post my opinions below, but these are not final answers and it would be great to hear the opinions of other other users here.

Anyways...

At which point getting more simulations in the dataset could help?

In general, more simulations yields better results. In addition, more parameters will often mean that you will need more simulations, but it is very difficult to give answers such as "you need X simulations for a simulator with N parameters". However, very very roughly, I would say:

• 1-5 parameters: 1k-10k simulations
• 6-10 parameters: 10k-100k simulations
• more than 10 parameters: >100k simulations

In order to identify if you need more simulations, I would recommend predictive checks and calibration checks

Should I focus on getting better summary statistics or training an embedding net?

General rule: if you can write down summary statistics (that are useful and you care about), I recommend using summary statistics (instead of using the raw data + embedding net). Using an embedding_net usually means that you will need more simulations. I only recommend using an embedding net if (1) you can not write down summary statistics and (2) your data is high-dimensional (e.g. an image or time series).

In addition, the choice of summary statistics is key to performance of sbi, see also here. As spelled out in this tutorial:

- Visualize the histogram of each summary statistic and plot the value of the observation. 
- If, for some summary statistics, the observation is not covered (or is at the very border, e.g. the MSE above), the trained neural network will struggle. 
- Do not use an “error” as summary statistic. This is common in optimization (e.g. genetic algorithms), but it often leads to trouble in sbi due to the reason above. 
- Only use summary statistics that are necessary. The less summary statistics you use, the less can go wrong with them. Of course, you have to ensure that the summary statistics describe the raw data sufficiently well.

Should I change the density estimator internal architecture (adding more flows or anything?)

In many cases, nsf gives better results than maf (but it trains longer and there definitely are simulators for which maf is better). Other density estimators (e.g. diffusion models or energy-based models are currently being investigated). They seems to improve performance on some problems, but perform worse on other problems. In my opinion, you might get a bigger performance boost by using more simulations in the dataset, so I would try this first.

[renecotyfanboy]

Thank you very much! This is very interesting, I will investigate your suggestions

[pscicluna]

This info is super useful! Could you elaborate a bit more on the trade-offs of using an embedding_net? For example, what is the typical point (e.g. number of features in the dataset) when it is worth using an embedding? Are there situations where an embedding can reduce the number of simulations you need compared to using the raw data (and if so, what sort of cases do these tend to be)? Do you have any insights on what sort of networks are effective, and which to avoid?

On the topic of number of simulations, I have often heard that the number of simulations should not depend on the nuisance parameters for the simulator, only the parameters of interest. The justification I was given is that the nuisance parameters are seen as sources of noise instead of useful information. Is this true of all SBI approaches or only things like TMNRE?

I was also wondering if you have a feel for how the required number of simulations changes in multi-round vs amortised inference? I'm sure there's huge variation again, but any guidance you can give would be great.

[michaeldeistler]

Hi @pscicluna

Embedding

I don't think that there is a typical point at which it is worth using an embedding. If you are using summary statistics, I would not recommend using an embedding net. I performed inference in a model with 35 summary statistics and using an embedding network did not improve performance. At the contrary, if you do not use summary statistics and your model produces time series or images, I could imagine that you get a major gain by using an embedding net. Unfortunately, I do not have much experience with this scenario myself, so I can not give information such as "us an embedding net only if your time series has more than X time points" etc.

Nuisance parameters

Indeed, for all sbi methods implemented here, the number of simulations does not depend on the number of nuisance parameters of the simulator.

Amortized vs multi-round

The answer strongly depends on how broad your posterior is in comparison to the prior. In cases with few parameters (<10), the gain of multi-round will likely be rather small (very roughly, ~3 three times fewer simulations). E.g., you can check out the results of the benchmark, Fig 3 and compare the sequential methods (e.g., SNPE) to amortized methods (e.g., NPE). Note that all benchmark tasks fall into the category of having fewer than 10 parameters.

If you have more than 10 parameters, the gains of multi-round can be much bigger: e.g., here, fig 4, we showed that the number of simulations can be reduced by a factor of 22 by using multi-round in a model with 31 parameters. And here, fig 6, we performed sequential inference in a model in which amortized inference took too long to run (too many simulations needed). I would say that you need at least 100 times fewer simulations in this particular model by using sequential inference.