Update RMSE/MAE average formula for BAC cross validation

[hwpang]

Motivation or Problem

When using cross validation, the average RMSE and average MAE are currently calculated as RMSE_avg = sum^{n_fold}_i(RMSE_i)/n_fold and MAE_avg = sum^{n_fold}_i(MAE_i)/n_fold.

However, during leave-one-out cross validation, this method returns the same number for RMSE_avg and MAE_avg. This is because when there is only one testing data, RMSE_i = MAE_i.

Instead, I think this average formula should be used, where
RMSE_avg = sqrt( sum^{n_fold}_i (RMSE_i^2 * Ndata_i) / sum^{n_fold}_i (Ndata_i) )
MAE_avg = sum^{n_fold}_i (MAE_i * Ndata_i) / sum^{n_fold}_i (Ndata_i)

Description of Changes

I calculate the RMSE_avg and MAE_avg for leave-one-out cross validation case.

[hwpang]

I had an offline discussion with @oscarwumit and he thinks it's better to keep it consistent, so I change the RMSE_avg and MAE_avg for not only leave-one-out cross validation but also other K-fold cross validation cases.

[oscarwumit]

Looks good. Thanks for identifying and fixing the bug.

[github-actions]

Regression Testing Results

⚠️ One or more regression tests failed.
Please download the failed results and run the tests locally or check the log to see why.

Detailed regression test results.

Regression test aromatics:

Reference: Execution time (DD:HH:MM:SS): 00:00:01:16
Current: Execution time (DD:HH:MM:SS): 00:00:01:24
Reference: Memory used: 2085.85 MB
Current: Memory used: 2089.46 MB

aromatics Passed Core Comparison ✅

Original model has 15 species.
Test model has 15 species. ✅
Original model has 11 reactions.
Test model has 11 reactions. ✅

aromatics Passed Edge Comparison ✅

Original model has 106 species.
Test model has 106 species. ✅
Original model has 358 reactions.
Test model has 358 reactions. ✅

Details

Observables Test Case: Aromatics Comparison

✅ All Observables varied by less than 0.500 on average between old model and new model in all conditions!

aromatics Passed Observable Testing ✅

Regression test liquid_oxidation:

Reference: Execution time (DD:HH:MM:SS): 00:00:02:37
Current: Execution time (DD:HH:MM:SS): 00:00:02:53
Reference: Memory used: 2216.51 MB
Current: Memory used: 2207.11 MB

liquid_oxidation Failed Core Comparison ❌

Original model has 37 species.
Test model has 37 species. ✅
Original model has 215 reactions.
Test model has 216 reactions. ❌
The tested model has 1 reactions that the original model does not have. ❌
rxn: CCO[O](30) <=> [OH](22) + CC=O(69) origin: intra_H_migration

liquid_oxidation Failed Edge Comparison ❌

Original model has 202 species.
Test model has 202 species. ✅
Original model has 1610 reactions.
Test model has 1610 reactions. ✅
The original model has 1 reactions that the tested model does not have. ❌
rxn: CCO[O](29) <=> C[CH]OO(70) origin: intra_H_migration
The tested model has 1 reactions that the original model does not have. ❌
rxn: CCO[O](30) <=> [OH](22) + CC=O(69) origin: intra_H_migration

Details

Observables Test Case: liquid_oxidation Comparison

✅ All Observables varied by less than 0.100 on average between old model and new model in all conditions!

liquid_oxidation Passed Observable Testing ✅

Regression test nitrogen:

Reference: Execution time (DD:HH:MM:SS): 00:00:01:38
Current: Execution time (DD:HH:MM:SS): 00:00:01:46
Reference: Memory used: 2203.95 MB
Current: Memory used: 2204.82 MB

nitrogen Failed Core Comparison ❌

Original model has 41 species.
Test model has 41 species. ✅
Original model has 359 reactions.
Test model has 360 reactions. ❌
The tested model has 1 reactions that the original model does not have. ❌
rxn: HNO(48) + HCO(13) <=> NO(38) + CH2O(18) origin: H_Abstraction

nitrogen Failed Edge Comparison ❌

Original model has 132 species.
Test model has 132 species. ✅
Original model has 995 reactions.
Test model has 997 reactions. ❌

Non-identical thermo! ❌
original: O1[C]=N1
tested: O1[C]=N1

Hf(300K)	S(300K)	Cp(300K)	Cp(400K)	Cp(500K)	Cp(600K)	Cp(800K)	Cp(1000K)	Cp(1500K)
116.46	53.90	11.62	12.71	13.49	13.96	14.14	13.85	13.58
141.64	58.66	12.26	12.27	12.09	11.96	12.26	12.72	12.15

thermo: Thermo group additivity estimation: group(O2s-CdN3d) + group(N3d-OCd) + group(Cd-HN3dO) + ring(Cyclopropene) + radical(CdJ-NdO)
thermo: Thermo group additivity estimation: group(O2s-CdN3d) + group(N3d-OCd) + group(Cd-HN3dO) + ring(oxirene) + radical(CdJ-NdO)
The tested model has 2 reactions that the original model does not have. ❌
rxn: HNO(48) + HCO(13) <=> NO(38) + CH2O(18) origin: H_Abstraction
rxn: HON(T)(83) + HCO(13) <=> NO(38) + CH2O(18) origin: Disproportionation

Non-identical kinetics! ❌
original:
rxn: NCO(66) <=> O1[C]=N1(126) origin: Intra_R_Add_Endocyclic
tested:
rxn: NCO(66) <=> O1[C]=N1(126) origin: Intra_R_Add_Endocyclic

k(1bar)	300K	400K	500K	600K	800K	1000K	1500K	2000K
k(T):	-49.54	-33.65	-24.16	-17.85	-10.01	-5.35	0.80	3.82
k(T):	-66.25	-46.19	-34.19	-26.21	-16.28	-10.36	-2.54	1.31

kinetics: Arrhenius(A=(6.95187e+18,'s^-1'), n=-1.628, Ea=(88.327,'kcal/mol'), T0=(1,'K'), comment="""Estimated from node Backbone0_N-2R!H-inRing_N-1R!H-inRing_Sp-2R!H-1R!H""")
kinetics: Arrhenius(A=(6.95187e+18,'s^-1'), n=-1.628, Ea=(111.271,'kcal/mol'), T0=(1,'K'), comment="""Estimated from node Backbone0_N-2R!H-inRing_N-1R!H-inRing_Sp-2R!H-1R!H""")
Identical kinetics comments:
kinetics: Estimated from node Backbone0_N-2R!H-inRing_N-1R!H-inRing_Sp-2R!H-1R!H

Details

Observables Test Case: NC Comparison

✅ All Observables varied by less than 0.200 on average between old model and new model in all conditions!

nitrogen Passed Observable Testing ✅

Regression test oxidation:

Reference: Execution time (DD:HH:MM:SS): 00:00:02:42
Current: Execution time (DD:HH:MM:SS): 00:00:02:52
Reference: Memory used: 2071.62 MB
Current: Memory used: 2074.57 MB

oxidation Passed Core Comparison ✅

Original model has 59 species.
Test model has 59 species. ✅
Original model has 694 reactions.
Test model has 694 reactions. ✅

oxidation Passed Edge Comparison ✅

Original model has 230 species.
Test model has 230 species. ✅
Original model has 1526 reactions.
Test model has 1526 reactions. ✅

Details

Observables Test Case: Oxidation Comparison

✅ All Observables varied by less than 0.500 on average between old model and new model in all conditions!

oxidation Passed Observable Testing ✅

Regression test sulfur:

Reference: Execution time (DD:HH:MM:SS): 00:00:01:04
Current: Execution time (DD:HH:MM:SS): 00:00:01:09
Reference: Memory used: 2167.14 MB
Current: Memory used: 2175.05 MB

sulfur Passed Core Comparison ✅

Original model has 27 species.
Test model has 27 species. ✅
Original model has 74 reactions.
Test model has 74 reactions. ✅

sulfur Failed Edge Comparison ❌

Original model has 89 species.
Test model has 89 species. ✅
Original model has 227 reactions.
Test model has 227 reactions. ✅
The original model has 1 reactions that the tested model does not have. ❌
rxn: O(4) + SO2(15) (+N2) <=> SO3(16) (+N2) origin: primarySulfurLibrary
The tested model has 1 reactions that the original model does not have. ❌
rxn: O(4) + SO2(15) (+N2) <=> SO3(16) (+N2) origin: primarySulfurLibrary

Details

Observables Test Case: SO2 Comparison

✅ All Observables varied by less than 0.100 on average between old model and new model in all conditions!

sulfur Passed Observable Testing ✅

Regression test superminimal:

Reference: Execution time (DD:HH:MM:SS): 00:00:00:40
Current: Execution time (DD:HH:MM:SS): 00:00:00:43
Reference: Memory used: 2337.24 MB
Current: Memory used: 2333.04 MB

superminimal Passed Core Comparison ✅

Original model has 13 species.
Test model has 13 species. ✅
Original model has 21 reactions.
Test model has 21 reactions. ✅

superminimal Passed Edge Comparison ✅

Original model has 18 species.
Test model has 18 species. ✅
Original model has 28 reactions.
Test model has 28 reactions. ✅

Regression test RMS_constantVIdealGasReactor_superminimal:

Reference: Execution time (DD:HH:MM:SS): 00:00:03:02
Current: Execution time (DD:HH:MM:SS): 00:00:03:22
Reference: Memory used: 2735.93 MB
Current: Memory used: 2722.04 MB

RMS_constantVIdealGasReactor_superminimal Passed Core Comparison ✅

Original model has 13 species.
Test model has 13 species. ✅
Original model has 19 reactions.
Test model has 19 reactions. ✅

RMS_constantVIdealGasReactor_superminimal Passed Edge Comparison ✅

Original model has 13 species.
Test model has 13 species. ✅
Original model has 19 reactions.
Test model has 19 reactions. ✅

Details

Observables Test Case: RMS_constantVIdealGasReactor_superminimal Comparison

✅ All Observables varied by less than 0.100 on average between old model and new model in all conditions!

RMS_constantVIdealGasReactor_superminimal Passed Observable Testing ✅

Regression test RMS_CSTR_liquid_oxidation:

Reference: Execution time (DD:HH:MM:SS): 00:00:07:40
Current: Execution time (DD:HH:MM:SS): 00:00:08:15
Reference: Memory used: 2735.63 MB
Current: Memory used: 2712.79 MB

RMS_CSTR_liquid_oxidation Passed Core Comparison ✅

Original model has 37 species.
Test model has 37 species. ✅
Original model has 233 reactions.
Test model has 233 reactions. ✅

RMS_CSTR_liquid_oxidation Failed Edge Comparison ❌

Original model has 206 species.
Test model has 206 species. ✅
Original model has 1508 reactions.
Test model has 1508 reactions. ✅
The original model has 1 reactions that the tested model does not have. ❌
rxn: CCCO[O](35) <=> [OH](21) + CCC=O(50) origin: intra_H_migration
The tested model has 1 reactions that the original model does not have. ❌
rxn: CCCO[O](34) <=> CC[CH]OO(51) origin: intra_H_migration

Details

Observables Test Case: RMS_CSTR_liquid_oxidation Comparison

✅ All Observables varied by less than 0.100 on average between old model and new model in all conditions!

RMS_CSTR_liquid_oxidation Passed Observable Testing ✅

beep boop this comment was written by a bot 🤖