MHMostafa Hadizadeh

Mostafa Hadizadeh

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Mostafa Hadizadeh has a total of 34 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    A Service-Oriented Distributed Zarr Solution for Climate Data Access Across Heterogeneous HPC and Storage Infrastructures

    by Mostafa Hadizadeh, Martin Bergemann, Etor Lucio Eceiza, Andrej Fast, Christopher Kadow

    Modern climate archives are increasingly distributed across heterogeneous storage systems, while analysis workflows are becoming more interactive, distributed, and cloud-native. Moreover, many high-performance computing centres (HPC) host large climate datasets on traditional file-based storage infrastructures, whereas computational resources are often located at different sites. This separation between data location and compute resources creates significant barriers to efficient, interactive, and scalable data access.This situation calls for climate data access services that are scalable, flexible, and independent of specific client-side environments, while supporting common climate data formats such as NetCDF, GeoTIFF, Zarr, HDF5, and GRIB. Nevertheless, efficient remote access to large and heterogeneous climate archives remains a major bottleneck for modern scientific workflows.We present aservice, the Freva Data Loader, which implements the logic required to open datasets from diverse storage backends and expose them as Zarr chunks through a lightweight, web-friendly REST interface with modern authentication mechanisms.The Freva Data Loader is implemented as a stateless, worker service exposing a REST interface for dataset access and Zarr endpoint generation. Upon receiving an authenticated request, the service resolves dataset metadata, opens the underlying data from the appropriate storage backend (e.g. POSIX file systems or object storage), and exposes the data as a Zarr-compatible, chunked stream. Authentication and authorisation are handled centrally using OAuth2, ensuring secure and controlled access across institutional boundaries. Requests are coordinated by a Loader component and distributed to worker instances via a message broker (Redis), enabling asynchronous execution and horizontal scalability.The service decouples data access from client-side tooling and enables users and applications to access data stored on traditional posix HPC file systems, tape archives, as well as cloud-based object storage through a unified Zarr interface. Instead of transferring complete files between data centres or downloading them in full, clients retrieve only the required data chunks on demand. Users and client applications can request chunked array access over the network and process data incrementally, supporting interactive exploration and scalable downstream computation using cloud-native, chunked storage semantics, while remaining compatible with existing analysis stacks based on Zarr, xarray, and Dask.

  • Open Access

    Accelerating Data Discovery: Automated, Scalable Harvesting and Indexing of Metadata Across Heterogeneous Storage Backends

    by Christopher Kadow, Martin Bergemann, Mostafa Hadizadeh, Manuel Reis, Etor Lucio Eceiza

    The discoverability and access of climate and Earth-system datasets is foundational for effective scientific analysis workflows, yet these datasets are often hosted across diverse storage systems and follow a variety of organisational conventions. Researchers and infrastructure engineers face challenges in ingesting distributed metadata into unified, searchable catalogues without sacrificing interoperability or scalability. Efficient metadata harvesting, normalisation, and ingestion at scale are therefore critical enablers for data discovery and FAIR (Findable, Accessible, Interoperable, and Reusable) data practices.To address this need, we present the Metadata Crawler, a metadata ingestion tool within the designed to automate the collection and indexing of climate dataset metadata across heterogeneous storage backends. The Metadata Crawler supports multi-backend discovery, including POSIX file systems, S3/MinIO object stores, and OpenStack Swift, enabling infrastructure administrators to aggregate metadata from local archives, cloud object storage, and institutional repositories. At its core, the Metadata Crawler implements a two-stage pipeline: harvested metadata are first collected into a temporary catalogue, and then indexed into downstream systems such as Apache Solr or MongoDB. Dataset definitions, directory structures, and extraction logic are governed by a flexible TOML configuration that encodes Data Reference Syntax (DRS) dialects for different standards. Users can make use of pre-defined standards or define their own, making the tool extremely flexible and versatile. This schema-driven approach, combined with path and data specifications, conditional rules, and computed fields, ensures consistent representation of key facets such as temporal coverage, geospatial bounds, and variables and many other metadata specifications.The tool provides both a command-line interface (CLI) and a Python API, supporting synchronous and asynchronous execution as well as multi-threaded crawling, facilitating integration into operational workflows. By normalising and indexing previously siloed metadata into searchable catalogues, the Metadata Crawler enhances data findability and empowers portals and analysis platforms to deliver efficient discovery services. Its modular design also allows deployment in diverse environments and easy extension to additional backends or indexing targets.

  • Open Access

    FAIR Environmental Data through a STAC-Driven Inter-Institutional Data Catalog Infrastructure – Status quo of the Cat4KIT-project

    by Mostafa Hadizadeh, Christof Lorenz, Sabine Barthlott, Romy Fösig, Katharina Loewe, Corinna Rebmann, Benjamin Ertl, Robert Ulrich, Felix Bach

    In the rapidly advancing domain of environmental research, the deployment of a comprehensive, state-of-the-art Research Data Management (RDM) framework is increasingly pivotal. Such a framework is key to ensure FAIR data, laying the groundwork for transparent and reproducible earth system sciences.Today, datasets associated with research articles are commonly published via prominent data repositories like Pangaea or Zenodo. Conversely, data used in actual day-to-day research and inter-institutional projects tends to be shared through basic cloud storage solutions or, even worse, via email. This practice, however, often conflicts with the FAIR principles, as much of this data ends up in private, restricted systems and local storage, limiting its broader accessibility and use.In response to this challenge, our research project Cat4KIT aims to establish a cross-institutional catalog and Research Data Management framework. The Cat4KIT framework is, hence, an important building block towards the FAIRification of environmental data. It not only streamlines the process of ensuring availability and accessibility of large-scale environmental datasets but also significantly enhances their value for interdisciplinary research and informed decision-making in environmental policy.The Cat4KIT system comprises four essential elements: data service provision, meta(data) harvesting, catalogue service, and user-friendly data presentation. The data service provision module is tailored to facilitate access to data within typical storage systems by using well-defined and standardized community interfaces via tools like the Thredds data server, Intake Catalogues, and the OGC SensorThings API. By this, we ensure seamless data retrieval and management for typical use-casers in environmental sciences.(Meta)data harvesting via our so-called DS2STAC-package entails collecting metadata from various data services, followed by creating STAC-metadata and integrating it into our STAC-API-based catalog service.This catalog service module synergizes diverse datasets into a cohesive, searchable spatial catalog, enhancing data discoverability and utility via our Cat4KIT UI.Finally, our framework's data portal is tailored to elevate data accessibility and comprehensibility for a wide audience, including researchers, enabling them to efficiently search, filter, and navigate through data from decentralized research data infrastructures.One notable characteristic of Cat4KIT is its dependence on open-source solutions and strict adherence to community standards. This guarantees not just the framework's ability to function well with current data systems but also its simple adaption and expansion to meet future needs. Our presentation demonstrates the technical structure of Cat4KIT, examining the development and integration of each module to adhere to the FAIR principles. Additionally, it showcases examples to illustrate the practical use of the framework in real-life situations, emphasizing its efficacy in enhancing data management practices within KIT and its potential relevance in other research organizations.

  • Open Access

    MDE-Thredds: A Django-based plugin for managing THREDDS data server

    by Mostafa Hadizadeh, Philipp S Sommer, Christof Lorenz, Linda Baldewein

    The Thematic Real-Time Environmental Distributed Data Services (THREDDS) Data Server is an open-source, Java-based web application that enables metadata and data access to scientific netCDF datasets. In recent years, more and more research institutes implemented THREDDS to give researchers and other end-users access to a wide range of real-time and archival data sets from earth system sciences. A number of features and interfaces are provided by THREDDS that facilitate the interactive and automated exploration, standardization and use of data like the automated generation of ISO-formatted metadata files or the provision of OGC-services (WMS and WCS). However, the configuration of THREDDS via XML-catalogs remains difficult and is usually restricted to system admins. And particularly the publication and consistent maintenance of a large number of datasets is prone to errors and hence proves to be difficult and time-consuming. Within the Model Data Explorer (MDE, https://model-data-explorer.readthedocs.io), a cross-institutional project to simplify a FAIR publication of model data on the web, we develop a module to overcome these configuration issues and enable scientists to make their environmental research data available on the web. This MDE-THREDDS module manages the catalogs and configurations of the THREDDS data server by providing a user-friendly web-interface for handling major components of THREDDS, including catalogs and web services. A flexible permission system enables scientists and other data producers to add and update their own datasets without the need for manually editing the underlying THREDDS catalogs. This permission system further allows server administrators to moderate and facilitate the publication of data on the web by scientists and other end-users which, hence, ensures a standardized and consistent THREDDS catalog infrastructure.Overall, with MDE-THREDDS, we want to give scientists and other data producers a simple and user-friendly framework for making their research data open and FAIR through a wide range of standardized and well-established web interfaces.

  • Open Access

    CAT4KIT: A cross-institutional data catalog framework for the FAIRification of environmental research data

    by Christof Lorenz, Mostafa Hadizadeh, Sabine Barthlott, Romy Fösig, Uğur Çayoğlu, Robert Ulrich, Felix Bach

    A contemporary and flexible Research Data Management (RDM) framework is required to make environmental research data Findable, Accessible, Interoperable, and Reusable (FAIR) and, hence, provide the foundation for open and reproducible earth system sciences. While data-sets that accompany scientific articles are typically published via large data repositories like Pangaea or Zenodo, intermediate, day-to-day, or actively-used data (e.g., data from research projects or prototypical data) is still exchanged via simple cloud storage services and email. And while the FAIR principles require data to be openly findable and accessible, it is often only available within closed and restricted infrastructures and local file systems.Our research project Cat4KIT hence aims to develop a cross-institutional catalog and RDM framework for the FAIRification of such day-to-day research data. This framework is comprised of four modules / services for providing access to data on storage systems through well-defined and standardized interfaces harvesting and transforming (meta)data into standardized formats making (meta)data accessible to the public using well-defined and standardized catalog services and interfaces enabling users to search, filter, and explore data from decentralized research data infrastructures. We develop, implement and evaluate each of these four modules within an inter-institutional consortium consisting of scientists, software developers and potential end-users. This allows us to include a wide-range of research data from multi-dimensional climate model outputs to high-frequency in-situ measurements. We emphasize the application of existing open-source solutions and community standards for data interfaces (THREDDS, STA, S3), (meta)data schemes, and catalog services (Spatio-Temporal Assets Catalog - STAC) in order to ensure an easy integration of research data into the Cat4KIT-framework and a straightforward extension to further research data infrastructures.In our presentation, we demonstrate the current status of our Cat4KIT-framework as an inter-institutional research data management and catalog platform for the FAIRification of day-to-day research data.

  • Open Access

    User-friendly data pipeline for FAIRification of environmental sensordata

    by Benjamin Louisot, Christof Lorenz, Mostafa Hadizadeh, Ingo Völksch, David Schäfer

    In recent years, the requirements for data from earth system sciences have increased massively. Data from observation systems needs to be transferred into larger research data infrastructures, evaluated and flagged via well-defined quality checks, enriched with standardized metadata and finally made available to the public via standard interfaces. And in order to fulfill the FAIR principles, we have to ensure transparency and reproducibility of all these steps. Moreover, the rising demand of near-real-time (NRT) data requires the whole data pipeline to run operationally with minimal manual effort.However, in many cases, there are still heterogenous data landscapes to be found without centralized control of data, data processing, version control and QA/QC. This is often aggravated by to inconvenient, outdated and isolated tools and software solutions.Therefore, we develop and implement an adaptable automated pipeline, which combines the assurance of data consistency, QA/QC (Quality Assurance / Quality Control), graphically supported validation and unified persistence and publication of data. User friendliness is achieved by making the system configurable and trackable through lightweight user interfaces over the complete data lifecycle. By only using open-source software solutions and applying community standards for data formats and interfaces, a high level of sustainability and independence can be ensured.In this presentation, we hence want to demonstrate such an end-to-end data pipeline that finally allows for the FAIRification of typical environmental sensor data.

  • Open Access

    ESM Data Exploration with the Model Data Explorer

    by Philipp S. Sommer, Linda Baldewein, Hatef Takyar, Rehan Chaudhary, Mostafa Hadizadeh, Housam Dibeh, Max Böcke, Christof Lorenz, Tilman Dinter, Stefan Pinkernell, et al.

    Making Earth-System-Model (ESM) Data accessible is challenging due to the large amount of data that we are facing in this realm. The upload is time-consuming, expensive, technically complex, and every institution has their own procedures.Non-ESM experts face a lot of problems and pure data portals are hardly usable for inter- and trans-disciplinary communication of ESM data and findings, as this level of accessibility often requires specialized web or computing services. With the Model Data Explorer, we want to simplify the generation of web services from ESM data, and we provide a framework that allows us to make the raw model data accessible to non-ESM experts.Our decentralized framework implements the possibility for an efficient remote processing of distributed ESM data. Users interface with an intuitive map-based front-end to compute spatial or temporal aggregations, or select regions to download the data. The data generators (i.e. the scientist with access to the raw data) use a light-weight and secure python library based on the Data Analytics Software Framework (DASF, https://digital-earth.pages.geomar.de/dasf/dasf-messaging-python) to create a back-end module. This back-end module runs close to the data, e.g. on the HPC-resource where the data is stored. Upon request, the module generates and provides the required data for the users in the web front-end.Our approach is intended for scientists and scientific usage! We aim for a framework where web-based communication of model-driven data science can be maintained by the scientific community. The Model Data Explorer ensures fair reward for the scientific work and adherence to the FAIR principles without too much overhead and loss in scientific accuracy. The Model Data Explorer is in the progress of development at the Helmholtz-Zentrum Hereon, together with multiple scientific and data management partners in other German research centers. The full list of contributors is constantly updated and can be accessed at https://model-data-explorer.readthedocs.io.

  • A new method to estimate cloud effective radius using Meteosat Second Generation SEVIRI over Middle East

    by Mostafa Hadizadeh, Mehdi Rahnama, Mehdi Kamali, Mona Kazemi, Ali Mohammadi


  • Verification of two machine learning approaches for cloud masking based on reflectance of channel IR3.9 using Meteosat Second Generation over Middle East maritime

    by Mostafa Hadizadeh, Mehdi Rahnama, Behnam Hesari

    Most cold channels of Meteosat Second Generation (MSG) satellites can distinguish between the sea and ice cloud tops, except for the IR3.9 channel because of the close reflectance and radiance values of the IR3.9 channel for maritime, low-level cloud and ice cloud tops. In this article, we introduce and evaluate two machine learning methods for cloud masking of Spinning Enhanced Visible and Infrared Imager (SEVIRI) images in the day and night that use the reflectance value of the IR3.9 channel. We reached a good correlation by comparing the results of the modelled cloud masking of Meteosat satellite images with MODIS (Moderate Resolution Imaging Spectroradiometer) and CLM (Cloud Mask product of EUMETSAT) images in a way that the coefficient of determination (R2) value was 92.34%, 89.91% and 83.69%, 78.23% in the cold season and 90.17%, 87.09% and 80.37%, 76.48% in the warm season, respectively, using the CHAID (chi-squared automatic interaction detection) decision tree and RBF (radial basis function) neural network approaches.