TY - JOUR

T1 - Assessing spatial transferability of a random forest metamodel for predicting drainage fraction

AU - Bjerre, Elisa

AU - Fienen, Michael N.

AU - Schneider, Raphael

AU - Koch, Julian

AU - Højberg, Anker L.

N1 - Publisher Copyright: © 2022 The Author(s)

PY - 2022

Y1 - 2022

N2 - Fully distributed hydrological models are widely used in groundwater management, but model speed and data requirements impede their use for decision support purposes. Metamodels provide a simpler and faster model which emulates the underlying complex model using machine learning techniques. However, metamodel predictions beyond the ranges, in space and/or time, of training data are highly uncertain, and thus it is important to assess the predictive model performance to ranges outside the training data, i.e., model transferability. We present a novel methodology for evaluating model transferability to areas not contained in the training data set, based on various metrics that quantify the differences in covariate distributions between training and testing data. The transferability method can be employed as a screening tool to assess the suitability of a metamodel for spatial prediction beyond its training domain. We evaluated this transferability approach on a Random Forest metamodel of a 1000 km2 fully distributed coupled groundwater model for predicting drainage fraction, the partitioning of infiltrating water between drains and groundwater. We conducted spatial cross-validation on 9 holdout sub-basins to assess metamodel transferability beyond sampling locations and compared this estimate with a random split-sample validation test. Using mappable covariates only, the metamodel showed high performance (R2 = 0.79) tested on a 20% randomly sampled holdout. Conversely, metamodel performance significantly decreased for the 9 spatial holdouts (R2 ranging from 0.13 to 0.61). We document that the proposed transferability metric correlates with metamodel predictive performance, and demonstrate its use to assess model transferability to datasets outside the training data spatial domain.

AB - Fully distributed hydrological models are widely used in groundwater management, but model speed and data requirements impede their use for decision support purposes. Metamodels provide a simpler and faster model which emulates the underlying complex model using machine learning techniques. However, metamodel predictions beyond the ranges, in space and/or time, of training data are highly uncertain, and thus it is important to assess the predictive model performance to ranges outside the training data, i.e., model transferability. We present a novel methodology for evaluating model transferability to areas not contained in the training data set, based on various metrics that quantify the differences in covariate distributions between training and testing data. The transferability method can be employed as a screening tool to assess the suitability of a metamodel for spatial prediction beyond its training domain. We evaluated this transferability approach on a Random Forest metamodel of a 1000 km2 fully distributed coupled groundwater model for predicting drainage fraction, the partitioning of infiltrating water between drains and groundwater. We conducted spatial cross-validation on 9 holdout sub-basins to assess metamodel transferability beyond sampling locations and compared this estimate with a random split-sample validation test. Using mappable covariates only, the metamodel showed high performance (R2 = 0.79) tested on a 20% randomly sampled holdout. Conversely, metamodel performance significantly decreased for the 9 spatial holdouts (R2 ranging from 0.13 to 0.61). We document that the proposed transferability metric correlates with metamodel predictive performance, and demonstrate its use to assess model transferability to datasets outside the training data spatial domain.

KW - Drain partitioning

KW - Histogram distance

KW - Machine learning

KW - Model generalization

KW - Model portability

KW - Predictive modelling

U2 - 10.1016/j.jhydrol.2022.128177

DO - 10.1016/j.jhydrol.2022.128177

M3 - Journal article

AN - SCOPUS:85134575492

VL - 612

JO - Journal of Hydrology

JF - Journal of Hydrology

SN - 0022-1694

M1 - 128177

ER -