Calculation of P/ R/ F1-Score from Model Export

[sushds]

Hi @StijnKas, directing this question to you as we have already discussed this topic before.

I am trying to calculate the metrics mentioned in the title using the model export. It was easier to process this data in BigQuery so I uploaded the tables in there. Now I see that there are a number of columns that could be of interest to calculate these metrics but are probably aggregated and named accordingly. What I am looking for: # of true positives, # of false positives, # of false negatives and # of true negatives.

I expect this information to be present in the model exports. I see column names such as: pyNegatives, pyPositives, pyBinPositives, pyBinNegatives, pyBinPositivesPercentage, pyBinNegativesPercentage, pyTotalBins, py Peformance and pyBinResponseCount.

I believe I need to use the numbers from these columns but I am not sure how to get there. Could you maybe help me in explaining these columns / getting to the numbers I am trying to calculate?

[operdeck]

Hi Sushant, It is pyBinPositives and pyBinNegatives that you'll need. Whether these positives and negatives are "true" or "false" obviously depends on the threshold - which is why an area under a curve is a better metric when you have a continuous outcome. FYI see the "ADM explained" articles on PDS tools documentation to see how we use these bin positives/negatives to calculate AUC.

[sushds]

Hi Otto, thanks for the response.

I referred to the function in PDS tools documentation. I understand that since we have varying thresholds it is nice to use AUC as it deals with the overall classification performance. It will be helpful to know more about the thresholds that determine these operating points. Is that information available somewhere?