Fahamu

• Machine learning-based weather prediction.
• Ensemble-based model, aggregating multiple predictions to assess uncertainty and variability.

Initial conditions are sourced from ECMWF Open Data, including surface, soil moisture, and pressure level parameters. These are downloaded following the AIFS methodology https://huggingface.co/ecmwf/aifs-ens-1.0/blob/main/run_AIFS_ENS_v1.ipynb, preprocessed into standardized input states, and stored as pickle(pkl) files for each of the 50 ensemble members before GPU inference but uses reduced precision for the inference FP16 and low memory GPU. which differ from other model FahamuAIFSv1. The initial conditions are downloaded weekly on Thursday 00Z and previous day 18Z IFS 0h grib2 files. The code was modified into https://github.com/icpac-igad/ea-aifs/blob/main/ecmwf_opendata_pkl_input_aifsens.py to run on CPU-only computers to save costs, and methods were added to transfer the initial dataset for use within the GPU computer.

ECMWF Open Data provides real-time initial conditions—including surface parameters, soil moisture, and pressure levels—retrieved weekly at 00Z on Thursdays and 18Z on the previous day. The operational workflow downloads and preprocesses this data for all 50 ensemble members to initialize each forecast run. However, ECMWF Open Data only retains the most recent four days of data, which limits its use as initial conditions for AIFS inference. To overcome this constraint, an alternative approach is being explored using the AWS Open Data Registry, where IFS data are archived for over a year and remain available in near real time. A method is being developed using the Kerchunk and VirtualiZarr Python libraries to enable seamless access to GRIB2 files stored in the AWS Open Data Registry, thereby overcoming the bottleneck caused by the four-day data availability limitation in ECMWF Open Data.

The base AIFS ENS v1.0 model was trained by ECMWF on ERA5 reanalysis and ECMWF operational NWP analyses. We utilize this pre-trained model weight for inference (ecmwf/aifs-ens-1.0) from https://huggingface.co/ecmwf/aifs-ens-1.0) without any additional training.

ECMWF’s AIFS-ENS v1.0 is a machine-learning–based ensemble weather forecasting system. It uses a graph neural network (GNN) encoder-decoder architecture combined with a transformer-based temporal processor. The model is trained with the Continuous Ranked Probability Score (CRPS) as its loss function—specifically. The resulting model is stochastic, capable of producing an set of ensemble members to represent forecast uncertainty. In the current version of model architecture, after the initial condition files(pkl) are created, the script https://github.com/icpac-igad/ea-aifs/blob/main/fp16_automate_aifs_gpu_pipeline.py was used to run the GPU based inference on the pkl files. A 50-member ensemble inference is executed on an NVIDIA G2 (L4 24GB) —in this case via Google Cloud Platform using a Coiled https://coiled.io/ -managed Dask notebook environment. Post-processing includes ensemble statistical computation (e.g., mean, spread, and quantiles) to generate probabilistic forecast products using the AI-WQ-package Python library, which are then submitted as part of the weekly forecast product for AIquest. The python scripts following documentation at https://github.com/icpac-igad/ea-aifs/blob/main/README.md for running fp16 inference on https://github.com/icpac-igad/ea-aifs/blob/main/aifs_n320_grib_1p5defg_nc_cli.py and https://github.com/icpac-igad/ea-aifs/blob/main/ensemble_quintile_analysis_cli.py and https://github.com/icpac-igad/ea-aifs/blob/main/forecast_submission_cli.py was used for this purposes.

No publications yet. The foundational AIFS model is described in Lang et al. 2024. We use ECMWF's pre-trained model from https://huggingface.co/ecmwf/aifs-ens-1.0 and adapted it for operational ensemble forecasting. The complete workflow is published at https://github.com/icpac-igad/ea-aifs

At present validation occurs through AI Weather Quest competition library and submissions routine only. Model outputs are regridded to 1.5° resolution, ensemble statistics computed, and forecasts compared against climatology before submission to ensure quality and consistency.

Setting up the anemoi library and FlashAttention library requirements via Docker image linked with Coiled software to start a GPU cluster was challenging. GPU memory limitations were addressed using NVIDIA T4 24GB with optimized memory. There is scope for automating the routine using cloudrun GPU setup at GCP.

Sequential ensemble processing is time-intensive and costly in cloud computing; we plan parallel member execution for 2-3x speedup. High computational costs necessitate exploring preemptible GPU instances or GPU based compute functions.

Reduced precision model using FP16 option for AIFS model.

Nishadh Kalladath: DevOps cloud computing infrastructure, workflow and data pipeline programming, model deployment. Masilin Gudoshava, Climate science expertise, meteorological validation, and scientific methodology guidance. Anthony Mwanthi: Programming and scientific analysis support. Eunice Koech, Climate science expertise, meteorological validation Jason Kinyua: DevOps cloud computing infrastructure, Software development support and programming. Alex Ogalo: Software development support and programming. Hillary Koros: DevOps, workflow and data pipeline programming. Ahmed Amdihun: Program coordination, resource mobilization, and overall supervision.

• Machine learning-based weather prediction.
• Ensemble-based model, aggregating multiple predictions to assess uncertainty and variability.

Initial conditions come from ECMWF Open Data (0.25°) for the two timesteps [t−6h, t] at 00Z Thursday, following the AIFS-ENS-2.0 notebook: 112 fields per member- surface, constants, soil temperature and soil moisture (vsw→swvl1/2), 14 pressure levels (1000…10 hPa), and a new wave group from the waef stream (mwd decomposed to cos_mwd/sin_mwd) - interpolated to the N320 grid (542,080 points) and stored as one pickle per ensemble member. Preparation is CPU-only (fp16FahamuAIFSv2/ecmwf_opendata_pkl_input_aifsens_v2.py) so it can run on cheap hardware separately from the GPU step; this is the main v1→v2 input change (92 → 112 fields, adding snow depth, soil moisture, the wave group and the 10 hPa level, dropping q_10, and NaN-masking sd/swvl1/swvl2 over sea).

Real-time initial conditions are ECMWF Open Data (IFS ensemble enfo + wave ensemble waef, 0h step from Thursday 00z, Wednesday 18z) retrieved weekly for all 50 members. Because the ECMWF portal retains only ~4 days, v2 now reads the same open data from its cloud mirrors (AWS S3 / Google Cloud), which archive over a year, using each GRIB file's small JSON-lines .index to byte-range fetch only the ~112 messages a member needs.

None by us AIFS-ENS-2.0 was trained by ECMWF on ERA5 reanalysis and ECMWF operational analyses. We run the pre-trained ecmwf/aifs-ens-2.0 checkpoint for inference only, with no additional training or fine-tuning.

AIFS-ENS-2.0 is ECMWF's graph-based encoder–processor–decoder ensemble model trained with a CRPS objective to produce stochastic members; we execute it unchanged via anemoi-inference in FP16, 50 members × 792 h at 6-hourly steps. Inference now runs locally on a single NVIDIA RTX 5000 Ada (30 GB, sm_89) - ~283 s/member, ~3.9 h for the full 50-member ensemble at ~11.8 GB peak VRAM - instead of v1's Coiled/Cloud-Run L4 cluster, and writes native-N320 output straight into a local Icechunk (Zarr) store rather than GRIB; post-processing then regrids to 1.5°, computes ensemble quintile probabilities and submits via the AI-WQ package.

No publications yet; the foundational AIFS model is described in Lang et al. (2024). We use ECMWF's pre-trained checkpoint (https://huggingface.co/ecmwf/aifs-ens-2.0) and the full workflow is open at https://github.com/icpac-igad/ea-aifs.

Validation is through the AI Weather Quest submission routine only: outputs are regridded to 1.5°, ensemble quintile probabilities are computed against the competition's 20-year climatology, and files are checked by the AI-WQ package before upload. We additionally verify each input state is field-complete (112/112, correct shapes, land–sea mask applied) before inference.

The dominant constraints were storage and retrieval, not model compute: a full 50-member weekly run would write over 1.5 TB to the Icechunk store, so we persist only the forecast window actually used downstream (hours 432–792, days 18–33), cutting each cycle to ~583 GB. Retrieval was the other bottleneck: heavy AWS mirror throttling (frequent HTTP 503 SlowDown) stretched the 50-member input step to ~14.5 h, which we are addressing with parallel single-range fetches and the faster Google Cloud mirror.

The pipeline is network-bound at the input stage and the local GPU box is serial, so the ensemble runs one member at a time (~3.9 h) with no horizontal scaling; we intend to parallelise the input fetch and keep a cloud path for burst capacity. Storage is also pruned to the forecast window actually used downstream (hours 432-792, days 18-33) to keep the run within disk limits.

Two engineering pieces: writing model output natively at N320 into a versioned Icechunk/Zarr store with a commit per 6-hourly step (no GRIB round-trip, snapshot-tagged per cycle for reproducible downstream reads), .index byte-range reader that pulls only the required GRIB messages from the cloud mirrors and decodes them without scanning whole files, making multi year of archived open data usable as initial conditions.

fp16FahamuAIFSv2 was developed by the Fahamu Team at ICPAC.