Hilary J. Oliver

The National Institute of Water and Atmospheric Research (NIWA) is mandated to research and develop tools to increase New Zealand’s resilience to environmental hazards, including floods. NIWA generates and delivers its bespoke past, present and future environmental information services via a platform called EcoConnect. Comprising forecast output from numerical models of meteorological, hydrological and hydrodynamical hazards and data from related observation platforms, EcoConnect specialises in the creation and delivery of information that increases awareness of a broad range of environmental conditions, and provides input for a variety of specialist decision-support tools, chief of which is a customisable user-interface called NIWA Forecast, that uses this information to mitigate environmental hazards and commercial risk.  EcoConnect operates 24 hours a day, 7 days a week and is fully supported by scientific and technical staff.The EcoConnect workflow, which operates autonomously via the Cylc workflow meta-scheduler, begins with the data-assimilating New Zealand Limited Area Model (NZLAM) and New Zealand Convective-Scale Model (NZCSM) numerical weather prediction models.  These are based on the Met Office Unified Model, running with horizontal resolutions of 4.5km and 1.5km respectively over the full New Zealand, Tasman Sea and eastern Australia region (NZLAM) and just New Zealand and its coastal waters (NZCSM). These models provide input data for a hydrological river flow model, TopNet, based on the TopModel framework, that forecasts streamflow for just under 50,000 river reaches around New Zealand and a hierarchy of sea state and wave forecast models, based on the Wavewatch III model and locally called NZWAVE and NZTIDE. A coastal inundation model called RiCOM is also driven using data from the weather forecast models. Observation datasets provided within EcoConnect include satellite imagery, surface weather station data, river gauges and wave buoys. All of these data are created, collected, processed and archived by bespoke tasks in the EcoConnect workflow, all managed by Cylc. Almost all users of forecast products have bespoke needs, such as operational decision-making, and hence it is important to be able to cater to specific client requirements. Through EcoConnect, fit-for-purpose warnings can be configured, based on a user’s operational requirements, for any of the data sources in EcoConnect. For example, if the forecasted wave, or streamflow discharge, at a specified location were to exceed a specific threshold, a client can be warned via customisable alerts within EcoConnect and thus react appropriately. A collection of standard products is generated within EcoConnect and tools within the primary user-interface are provided to interrogate the data and define custom “workspaces” that provide at-a-glance monitoring capabilities. In this presentation, we will describe capabilities of the EcoConnect platform as they relate to hazard forecasting and warning. By means of a case study, we will show how EcoConnect was used to provide heads-up forecasting and decision-making support for an event that comprised weather, hydrological and wave hazards at the same time.  We will also highlight lessons learned and future development plans.

Complex cycling workflows are fundamental to Numerical Weather Prediction (NWP) and related environmental forecasting systems. Large numbers of jobs are executed at regular intervals to process new data and generate new forecasts. Dependence between these forecast cycles creates a single never-ending workflow, but NWP workflow schedulers have traditionally ignored this-at the cost of efficiency when running "off the clock"-by enforcing a simpler non-overlapping sequence of single-cycle workflows. Cylc ("Silk") (1)(2)(3) is designed to manage infinite cycling workflows efficiently even after delays in real-time operation, or in historical runs, when cycles can typically interleave for much-increased throughput. Cylc is not actually specialized to environmental forecasting, however, and cycling workflows may also be useful in other contexts. In this article we describe the origins and major features of Cylc, future plans for the project, and our experience of Open Source development and community engagement.

Most geophysical models include many parameters that are not fully determined by theory, and can be “tuned” to improve the model's agreement with available data. We might attempt to automate this tuning process in an objective way by employing an optimisation algorithm to find the set of parameters that minimises a cost function derived from comparing model outputs with measurements. A number of algorithms are available for solving optimisation problems, in various programming languages, but interfacing such software to a complex geophysical model simulation presents certain challenges. To tackle this problem, we have developed an optimisation suite (“Cyclops”) based on the Cylc workflow engine that implements a wide selection of optimisation algorithms from the NLopt Python toolbox (Johnson, 2014). The Cyclops optimisation suite can be used to calibrate any modelling system that has itself been implemented as a (separate) Cylc model suite, provided it includes computation and output of the desired scalar cost function. A growing number of institutions are using Cylc to orchestrate complex distributed suites of interdependent cycling tasks within their operational forecast systems, and in such cases application of the optimisation suite is particularly straightforward. As a test case, we applied the Cyclops to calibrate a global implementation of the WAVEWATCH III (v4.18) third-generation spectral wave model, forced by ERA-Interim input fields. This was calibrated over a 1-year period (1997), before applying the calibrated model to a full (1979–2016) wave hindcast. The chosen error metric was the spatial average of the root mean square error of hindcast significant wave height compared with collocated altimeter records. We describe the results of a calibration in which up to 19 parameters were optimised.

Most geophysical models include a number of parameters that are not fully determined by theory, and can be ‘tuned’ to improve the model's agreement with available data. We might attempt to automate this tuning process in an objective way by employing an optimisation algorithm to find the set of parameters that minimises a cost function derived from comparing model outputs with measurements. A number of algorithms are available for solving optimisation problems, in various software programming languages, but interfacing such software to a complex geophysical model simulation, presents certain challenges. To tackle this problem, we have developed an optimisation suite ("Cyclops") based on the Cylc workflow engine (http://cylc.github.io/cylc/ and https://zenodo.org/badge/latestdoi/1836229) that implements a wide selection of optimisation algorithms from the NLopt python toolbox (Johnson, 2014). The Cyclops optimisation suite can be used to calibrate any modelling system that has itself been implemented as a (separate) Cylc model suite, provided it includes computation and output of the desired scalar cost function. A growing number of institutions are using Cylc to orchestrate complex distributed suites of interdependent cycling tasks within their operational forecast systems, and in such cases application of the optimisation suite is particularly straightforward. As a test case, we applied the Cyclops to calibrate a global implementation of the Wavewatch III™ (v4.18) third generation spectral wave model, forced by ERA-Interim input fields. This was calibrated over a one-year period (1997), before applying the calibrated model to a full (1979–2016) wave hindcast. The chosen error metric was the spatial average of the root-mean-square error of hindcast significant wave height compared with collocated altimeter records. We describe the results of a calibration in which up to 19 parameters were optimised.