Open Force Field 3.0.0 Alpha Force Field Benchmarks

@openforcefield.org www.openforcefield.org Fitting protein backbone torsions to GB3 and AAQAA3 NMR observables Chapin E. Cavender Open Force Field Initiative Mike Gilson Lab, University of California San Diego Protein Force Field Meeting October 16, 2025 Agenda ▶Update on NMR FF benchmarks including RDCs ▶I’m nominating Null-4-mer-AAQAA3 as an alpha release candidate ▶What’s the next fitting target beyond the alpha candidate? www.openforcefield.org 1 Glossary of parameter sets ▶Null-QM: Null model (Sage proper torsion types) fit to Sage-2.1 plus capped 1-mer QM data, Sage-2.1 priors, pairwise objective with no minimization for TorsionDrive targets ▶Null-4-mer-QM: Null model fit to Sage-2.1 plus capped 1-mer plus PDB 4-mer QM data ▶Specific-QM: Specific model (ff14SB proper torsion types for protein-specific backbones and sidechains) fit to Sage-2.1 plus capped 1-mer QM data, Sage-2.1 priors, pairwise objective with no minimization for TorsionDrive targets ▶Specific-4-mer-QM: Specific model fit to Sage-2.1 plus capped 1-mer plus PDB 4-mer QM data www.openforcefield.org 2 Glossary of parameter sets ▶Null-US-UB-4: Sage plus protein-specific backbone torsion types, protein-specific torsions fit to GB3 backbone J-couplings, Null-US-UB-3 priors ▶Null-AAQAA3: Sage plus protein-specific backbone torsion types, protein-specific torsions fit to GB3 backbone Jcouplings and AAQAA3 fraction helix, Null-US-UB-4 priors ▶(Null/Specific)-4-mer-AAQAA3: Sage plus protein-specific (backbones/backbones and sidechains), backbone torsions fit to GB3 backbone and AAQAA3 fraction helix, 4-mer-QM priors www.openforcefield.org 3 Conclusions ▶Null-4-mer-AAQAA3 is best performer out of SMIRNOFF FFs on all validation sets ▶Null-4-mer-AAQAA3 has performance equal to or better than ff14SB on training datasets (GB3 J-couplings and AAQAA3 fraction helix) ▶Null-4-mer-AAQAA3 is worse than ff14SB but similar to ff99SB on Tier 2 test set ▶Qfactor scores agreement with experiment across observable types ▶Trends in RDC performance mirror trends in J-coupling performance www.openforcefield.org 4 Trends in RDC Qfactors mirror trends in Jcoupling χ2 Force GB3 AAQAA3 Peptide Tier 2 GB3 Tier 2 Field J-coupling fhelix J-coupling J-coupling RDC RDC χ2χ2χ2ANE Q Q ff14SB 3.02(11) 27.7(61) 2.27(3) 0.1133(4) 0.21(1) 0.32(2) ff99SB 2.50(12) 62.1(36) 1.17(4) 0.1312(15) 0.20(1) 0.37(1) Sage-2.1.0-NAGL 9.71(115) 0.78(1) 0.65(2) Null-US-UB-4 3.02(8) 66.6(31) 1.10(1) 0.1379(23) 0.25 0.44(2) Null-AAQAA3 4.09(94) 9.2(15) 1.00(3) 0.1331(21) 0.30(7) 0.41(3) Null-AAQAA3-2 3.68(61) 5.7(17) 1.25(2) 0.1330(12) 0.27(6) 0.42(2) Null-AAQAA3-3 3.14(10) 5.4(15) 1.41(2) 0.1318(23) 0.23(1) 0.40(1) Null-4-mer-QM 6.31(72) 74.5(8) 1.65(5) 0.1297(11) 0.48(5) 0.46(2) Null-4-mer3.31(16) 8.4(37) 0.98(1) 0.1279(10) 0.23(1) 0.39(1) AAQAA3 Null-4-merAAQAA3-2 4.50(71) 10.8(14) 1.12(4) 0.1294(16) 0.32(5) 0.43(3) Specific-4-mer-QM 4.74(160) 66.0(25) 3.22(2) 0.1790(21) 0.37(10) 0.48(3) Specific-4-merAAQAA3 2.99(7) 3.5(8) 1.57(4) 0.1364(50) 0.24(1) 0.43(2) Specific-4-mer2.86(11) 2.3(6) 1.87(3) 0.1360(37) 0.24(1) 0.43(3) AAQAA3-2 www.openforcefield.org 5 Null-4-mer-AAQAA3 has best performance out of SMIRNOFF FFs on all validation sets Force GB3 AAQAA3 Peptide Tier 2 GB3 Tier 2 Field J-coupling fhelix J-coupling J-coupling RDC RDC χ2χ2χ2ANE Q Q ff14SB 3.02(11) 27.7(61) 2.27(3) 0.1133(4) 0.21(1) 0.32(2) ff99SB 2.50(12) 62.1(36) 1.17(4) 0.1312(15) 0.20(1) 0.37(1) Sage-2.1.0-NAGL 9.71(115) 0.78(1) 0.65(2) Null-AAQAA3-3 3.14(10) 5.4(15) 1.41(2) 0.1318(23) 0.23(1) 0.40(1) Null-4-mer3.31(16) 8.4(37) 0.98(1) 0.1279(10) 0.23(1) 0.39(1) AAQAA3 Specific-4-mer2.86(11) 2.3(6) 1.87(3) 0.1360(37) 0.24(1) 0.43(3) AAQAA3-2 www.openforcefield.org 6 Starting from 4-mer QM fit, AAQAA3/GB3 NMR fits stabilize GB3 helix 0 2 4 ff14SB OPC3 0 2 4 ff99SB OPC3 0 2 4 Null AAQAA3 3 OPC3 0 2 4 Null 4-mer AAQAA3 OPC3 0246810 0 2 4 Specific 4-mer AAQAA3 2 OPC3 02468100246810 Time ( s) RMSD (Å) www.openforcefield.org 7 ff99SB best performer on GB3 J-couplings, Specific-4mer-AAQAA3-2 best of alpha candidates ff14SB ff99SB Sage-2.1.0 NAGL Null AAQAA3-3 Null-4-mer AAQAA3 Specific-4-mer AAQAA3-2 0 2 4 6 8 10 GB3 J coupling 2 www.openforcefield.org 8 Null-4-mer-AAQAA3 maintains HEWL secondary structure but helices are mobile 0 2 4 ff14SB OPC3 0 2 4 ff99SB OPC3 0 2 4 Null AAQAA3 3 OPC3 0 2 4 Null 4-mer AAQAA3 OPC3 0246810 0 2 4 Specific 4-mer AAQAA3 2 OPC3 02468100246810 Time ( s) RMSD (Å) www.openforcefield.org 15 Null-4-mer-AAQAA3 maintains Ubq structure 0 2 4 ff14SB OPC3 0 2 4 ff99SB OPC3 0 2 4 Null AAQAA3 3 OPC3 0 2 4 Null 4-mer AAQAA3 OPC3 0246810 0 2 4 Specific 4-mer AAQAA3 2 OPC3 02468100246810 Time ( s) RMSD (Å) www.openforcefield.org 16 ff14SB best performer on Tier 2 J-couplings, Null-4-merAAQAA3 best of alpha candidates ff14SB ff99SB Null AAQAA3-3 Null-4-mer AAQAA3 Specific-4-mer AAQAA3-2 0.00 0.10 J coupling ANE www.openforcefield.org 17 Null-4-mer-AAQAA3 similar to ff14SB for BPTI and Ubq, worse for HEWL BPTI HEWL Ubq 0.00 0.10 0.20 0.30 ANE ff14SB ff99SB Null-AAQAA3-3 Null-4-mer-AAQAA3 Specific-4-mer-AAQAA3-2 www.openforcefield.org 18 Null-4-mer-AAQAA3 similar to ff14SB for backbones, worse for side chains BPTI Backbone BPTI Side Chain HEWL Side Chain Ubq Backbone Ubq Side Chain Ubq H Bond 0.00 0.10 0.20 0.30 ANE ff14SB ff99SB Null-AAQAA3-3 Null-4-mer-AAQAA3 Specific-4-mer-AAQAA3-2 www.openforcefield.org 19 Residual dipolar couplings measure the average orientations of internuclear vectors in the molecule frame ˆ zL ˆ yL ˆ xL  B  rPQ θPQ ˆ zM ˆ yM ˆ xM  B βx βy βz  rPQ αx αy αz Dipolar couplings can be expressed using angles between arbitrary molecular axes and the internuclear vector (α) or magnetic field (β). DPQ ≈⟨K R3 PQ ⟩(3⟨cos2θPQ⟩ − 1) DPQ = 3 ⟨K R3 PQ ⟩∑ i,j Aij ⟨cos αicos αj⟩ Aij =⟨cos βicos βj−1 3δij⟩ www.openforcefield.org 20 Qfactor compares agreement with experimental RDCs across alignment media and experimental conditions Given experimental RDCs  Dexp and structural matrix Mcomputed from an MD trajectory, the alignment tensor is given by  A=M+ Dexp Computed RDCs are given by  Dcalc =M A=MM+ Dexp Agreement with experiment is assessed using the Q factor Q=(∑i(wi(Dcalc,i−Dexp,i))2 ∑i(wiDexp,i)2)1/2 where wi=⟨Ki R3 i⟩−1 www.openforcefield.org 21 ff99SB best performer on GB3 RDCs, but AAQAA3/GB3 NMR fits are similar ff14SB ff99SB Sage-2.1.0 NAGL Null AAQAA3-3 Null-4-mer AAQAA3 Specific-4-mer AAQAA3-2 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 RDC Q factor www.openforcefield.org 22 Null-4-mer-AAQAA3 similar to ff14SB for backbones, worse for side chains GB3 Backbone GB3 Side Chain 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 1.10 RDC Q factor ff14SB ff99SB Sage-2.1.0-NAGL Null-AAQAA3-3 Null-4-mer-AAQAA3 Specific-4-mer-AAQAA3-2 www.openforcefield.org 23 ff14SB best performer on Tier 2 RDCs, Null-4-merAAQAA3 best of alpha candidates ff14SB ff99SB Null AAQAA3-3 Null-4-mer AAQAA3 Specific-4-mer AAQAA3-2 0.00 0.10 0.20 0.30 0.40 0.50 RDC Q factor www.openforcefield.org 24 Dipolar couplings depend on orientations of internuclear vectors with respect to a static magnetic field ˆ zL ˆ yL ˆ xL  B  rPQ θPQ The dipolar coupling D between nuclei Pand Qis DPQ =K⟨3cos2θPQ −1 R3 PQ ⟩ DPQ ≈⟨K R3 PQ ⟩(3⟨cos2θPQ⟩ − 1) where θPQ is the angle between the internuclear vector and the magnetic field and K=−µ0hγPγQ 16π3. www.openforcefield.org 31 Residual dipolar couplings (RDCs) are dipolar couplings measured in a weak alignment medium DPQ ≈⟨K R3 PQ ⟩(3⟨cos2θPQ⟩ − 1) ▶For a molecule that tumbles isotropically, ⟨cos2θPQ⟩=1 3and DPQ = 0 ▶In a medium that weakly aligns molecules to a particular orientation in the lab frame (e.g. bicelles), nonzero dipolar couplings can be measured ▶RDCs are the difference in dipolar couplings measured in a weak alignment medium and in dilute solution www.openforcefield.org 32 The alignment tensor describes the distribution of molecular orientations in the lab frame ˆ zL ˆ yL ˆ xL ˆ zM ˆ yM ˆ xM ϕ θ ψ  B  rPQ ˆ zM ˆ yM ˆ xM ˆ zLˆ yL ˆ xL  B  rPQ The alignment tensor Ais a symmetric and traceless 3×3matrix with 5 independent parameters. Acan be diagonalized to give 3 Euler angles (ϕ, θ, ψ) defining molecular axes and 3 eigenvalues (Az,Ay,Ax=−(Az+Ay)) related to the magnitude of alignment to each axis. A=RT ϕ,θ,ψ   Az0 0 0Ay0 0 0 Ax Rϕ,θ,ψ www.openforcefield.org 33 Arbitrary molecular axes define a common frame for both the magnetic field and the internuclear vector ˆ zM ˆ yM ˆ xM  B βx βy βz  rPQ αx αy αz Dipolar couplings can be expressed using angles between arbitrary molecular axes and the internuclear vector (α) or magnetic field (β). DPQ ≈⟨K R3 PQ ⟩(3⟨cos2θPQ⟩ − 1) DPQ = 3 ⟨K R3 PQ ⟩∑ i,j Aij ⟨cos αicos αj⟩ Aij =⟨cos βicos βj−1 3δij⟩ www.openforcefield.org 34 The alignment tensor can be estimated from experimental RDCs using a singular value decomposition (SVD) For angles αk,ibetween internuclear vector iand arbitrary molecular axis k, a set of Nmeasured RDCs  Dcan be expressed using the matrix equation  D=M Awhere Mis a N×5matrix whose rows are MT i= 3 ⟨Ki R3 i⟩      ⟨cos2αz,i⟩−⟨cos2αx,i⟩ ⟨cos2αy,i⟩−⟨cos2αx,i⟩ 2⟨cos αz,icos αy,i⟩ 2⟨cos αz,icos αx,i⟩ 2⟨cos αy,icos αx,i⟩       and  A=      Azz Ayy Azy Azx Ayx       Then  A=M+ Dwhere M+is the Moore-Penrose inverse determined by SVD www.openforcefield.org 35 Qfactor compares agreement with experiment across alignment media and experimental conditions Given experimental RDCs  Dexp and matrix M computed from an MD trajectory, the alignment tensor is given by  A=M+ Dexp Computed RDCs are given by  Dcalc =M A=MM+ Dexp Agreement with experiment is assessed using the Q factor Q=(∑i(wi(Dcalc,i−Dexp,i))2 ∑i(wiDexp,i)2)1/2 where wi=⟨Ki R3 i⟩−1 www.openforcefield.org 36 Tiers of benchmarks to assess SMIRNOFF protein force fields ▶Tier 1 (rapidly evaluate parameter sets) ▶15 short unstructured peptides ▶Tier 2 (select release candidate) ▶4 folded proteins (GB3, BPTI, HEWL, Ubq) ▶Tier 3 (assess release candidate) ▶11 folded proteins with diverse structures ▶2 disordered proteins ▶JACS protein-ligand binding free energies ▶Tier 4 (follow up publications) ▶29 additional folded proteins ▶8 additional disordered proteins ▶8 room-temperature protein crystals www.openforcefield.org 37 Folded proteins for Tier 3 were selected by smallest member of each CATH architecture ▶CATH characterizes protein folds hierarchically ▶Class is type of secondary structure (SS) elements (α,β,α/β, no SS) ▶Architecture is spatial arrangement of SS elements (e.g. βroll, βsandwich, βbarrel) ▶Topology is connectivity of SS elements in primary sequence ▶Homology is phylogenetic connectivity ▶40 folded proteins targets from NESG represent 13 CATH architectures ▶2 architectures represented in Tier 2 proteins ▶From remaining 11 architectures, I selected the example with the fewest residues www.openforcefield.org 38 QM fits with PDB 4-mer dataset achieve lower objective values on other datasets Model Iter Total Obj 4-mer TD BB TD SC TD SM Opt Geo Null-QM 0 7344.7 869.1 1082.5 1053.5 4339.6 Null-QM 9 5844.2 627.0 825.3 671.5 3700.2 Null-4-mer-DW 0 7371.8 27.0 869.1 1082.5 1053.5 4339.6 Null-4-mer-DW 11 5764.8 25.2 590.2 810.8 589.9 3709.0 Null-4-mer 0 8425.9 1081.2 869.1 1082.5 1053.5 4339.6 Null-4-mer 14 6391.4 597.2 589.1 802.7 662.7 3711.0 Specific-QM 0 7351.5 869.1 1082.5 1053.5 4346.4 Specific-QM 11 5234.6 388.7 595.5 608.6 3627.1 Specific-4-mer 0 8432.7 1081.2 869.1 1082.5 1053.5 4346.4 Specific-4-mer 7 5814.4 366.8 459.0 626.6 631.7 3707.7 www.openforcefield.org 39 QM fits with PDB 4-mers and NMR fits have similar RMSE on QM validation dataset ff14SB Sage 2.1.0 NAGL Null QM Null 4 mer Null US UB Null US UB 2 Null US UB 3 Null US UB 4 Specific QM Specific 4 mer 0 1 2 3 4 Capped 3-mer backbone RMSE (kcal mol 1) www.openforcefield.org 40 REUS produces sampling across many window centers 0.00 0.08 0.00 0.08 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.00 0.08 Fraction native contacts Fraction www.openforcefield.org 47 4-mer-QM produces high variance across replicas 0 4 8 (3.0) REUS ff14SB 0 4 8 (12.1) REUS Null QM 0 4 8 (3.8) REUS Null 4-mer QM 0.40.60.81.0 0 4 8 (2.9) REUS Specific 4-mer QM 0.40.60.81.0 0.40.60.81.0 Fraction native contacts Free energy (kcal mol 1) www.openforcefield.org 48 4-mer-QM free energy is a broad funnel 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Fraction native contacts 0 2 4 6 8 10 Free energy (kcal mol 1) (3.0) REUS-ff14SB (12.1) REUS-Null-QM (3.8) REUS-Null-4-mer-QM (2.9) REUS-Specific-4-mer-QM www.openforcefield.org 49 Metrics for NMR scalar coupling validation for other folded proteins ▶Karplus parameters were trained on: ▶Backbone HN-X: GB3 ▶Sidechain N-CG and CO-CG: GB3 and Ubq ▶Sidechain HA-HB: flavodoxin ▶χ2metric relies on good estimate of uncertainty σfor Karplus model χ2=1 Nobs ∑ i∈obs (Ji,calc −Ji,exp)2 σ2 i ▶Averaged normalized error (ANE) from ff14SB ANE =1 Nobs ∑ i∈obs |Ji,calc −Ji,exp| Ji,max −Ji,min www.openforcefield.org 50 Objective functions for pairwise energy differences Amber ff14SB O=1 Npair ∑ i,j |(EQM,i−EQM,j)−(EMM,i−EMM,j)| ForceBalance FB15 O=∑iwi(EMM,i−EQM,i)2 ∑iwi(EQM,i− ⟨EQM⟩)2 ForceBalance pairwise energy differences O=∑i,j(wi,wj)1/2 ((EMM,i−EMM,j)−(EQM,i−EQM,j))2 ∑i,j(wi,wj)1/2 (EQM,i−EQM,j)2 www.openforcefield.org 51 ForceBalance weights conformers based on QM and MM energies O=∑i,j(wi,wj)1/2 ((EMM,i−EMM,j)−(EQM,i−EQM,j))2 ∑i,j(wi,wj)1/2 (EQM,i−EQM,j)2 Conformer weights ware product of asymmetry term f(EQM,EMM)and attenuation term g(EQM) f(EQM,EMM) = {1EMM ≥EQM 100 EMM <EQM g(EQM) =      Elower −1EQM <Elower (Elower + (EQM −Elower)2)−1/2 Elower ≤EQM <Eupper 0Eupper ≤EQM www.openforcefield.org 52 Ace-Gly-Nme phi (Phi-General) profile 180 120 60 0 60 120 180 gly Phi (deg) 2 0 2 4 6 8 PMF (kcal mol 1) QM Energy ff14SB Null-QM Null-US-UB-4 Null-AAQAA3 Null-AAQAA3-2 Null-4-mer-QM Null-4-mer-AAQAA3 Specific-QM Specific-4-mer-QM Specific-4-mer-AAQAA3 www.openforcefield.org 53 Ace-Ala-Nme phi (Phi-Sidechain) profile 180 120 60 0 60 120 180 ala Phi (deg) 4 0 4 8 12 16 PMF (kcal mol 1) QM Energy ff14SB Null-QM Null-US-UB-4 Null-AAQAA3 Null-AAQAA3-2 Null-4-mer-QM Null-4-mer-AAQAA3 Specific-QM Specific-4-mer-QM Specific-4-mer-AAQAA3 www.openforcefield.org 54 Ace-Val-Nme phi (Phi-Beta-Branched) profile 180 120 60 0 60 120 180 val Phi (deg) 4 0 4 8 12 16 20 PMF (kcal mol 1) QM Energy ff14SB Null-QM Null-US-UB-4 Null-AAQAA3 Null-AAQAA3-2 Null-4-mer-QM Null-4-mer-AAQAA3 Specific-QM Specific-4-mer-QM Specific-4-mer-AAQAA3 www.openforcefield.org 55 Ace-Gly-Nme psi (Psi-General) profile 180 120 60 0 60 120 180 gly Psi (deg) 2 0 2 4 PMF (kcal mol 1) QM Energy ff14SB Null-QM Null-US-UB-4 Null-AAQAA3 Null-AAQAA3-2 Null-4-mer-QM Null-4-mer-AAQAA3 Specific-QM Specific-4-mer-QM Specific-4-mer-AAQAA3 www.openforcefield.org 56 MBAR can estimate averages of observables for unsampled states Sample weights for reweighting from the MBAR mixture distribution to unsampled state iare Wi(t) = exp (−β(Ui(t)−Ci)) ∑Nwindows k=1 Nframes,kexp (−β(Uk(xt)−Ck)) Wi(t) = exp (−βUi(t)) Z(xt)  Nsamples ∑ t=1 exp (−βUi(t)) Z(xt)  −1 Average of an observable O(t)in state iand number of effective samples after reweighting Neff ⟨O⟩i= Nsamples ∑ t=1 O(t)Wi(t)and Neff =  Nsamples ∑ t=1 Wi(t)2  −1 www.openforcefield.org 63 Scoring new FFs quickly by reweighting Reweighting potential from ref FF A to query FF B Urw(t) = UB(t)−UA(t) Sample weights for reweighting potential are Wrw(t) = exp (−βUrw(t)) Z(xt)  Nsamples ∑ t=1 exp (−βUrw(t)) Z(xt)  −1 Reweighted estimate of an observable ⟨O⟩and number of effective samples after reweighting Neff ⟨O⟩= Nsamples ∑ t=1 O(t)Wrw(t)and Neff =  Nsamples ∑ t=1 Wrw(t)2  −1 www.openforcefield.org 64 Fitting to NMR observables by reweighting Reweighting potential with same functional form as the dihedral term in the force field Urw( k,t) = Nparam ∑ p=1 kp(1 + cos (npϕp(t)−γp)) Wrw( k,t) = exp (−βUrw( k,t)) Z(xt)  Nsamples ∑ t=1 exp (−βUrw( k,t)) Z(xt)  −1 Reweighted estimate of an observable ⟨O⟩is ⟨O⟩( k) = Nsamples ∑ t=1 O(t)Wrw( k,t) www.openforcefield.org 65 Fitting to NMR observables by reweighting Find the parameters  kthat minimize the regularized loss function L( k) = Nobs ∑ ℓ=1 (⟨Oℓ⟩( k)−Oℓ,exp)2 σ2 ℓ,exp +α Nparam ∑ p=1 k2 p ▶147 observables, 3 types of backbone NMR scalar couplings (3JHN,CB,3JHN,CO, and 3JHN,HA) ▶Choose regularization strength αby leave-one-out cross validation over the 3 observable types ▶Line search to choose αthat minimizes cross validation error, then fit using all observables www.openforcefield.org 66 Fit SMIRNOFF force fields to GB3 NMR observables ▶24 fit parameters are 6 backbone torsion types with 4 cosine terms per type ▶Use umbrella samplings run and starting parameter values from Null-0.0.3-Pair and Specific-0.0.3-Pair ▶Bake in OPC3 water ▶Timeline: 4 weeks to fit parameters iteratively, 2 weeks to benchmark on non-GB3 systems www.openforcefield.org 67