Evaluation System
Forecast evaluation methods
Weekly-updated leaderboards display Ranked Probability Scores (RPSSs) for all variables except for the MJO. For the MJO, leaderboards will be populated with Brier Skill Scores (BSSs).
- Mean sea level pressure forecasts are assessed across all grid points.
- Temperature forecasts are evaluated over land-dominated regions (>=50% land coverage).
- Precipitation forecasts are evaluated over land-dominated regions (>=50% land coverage) while excluding persistently arid areas, i.e., regions where even the wettest climatological quintile has 0 mm of precipitation.
- MJO forecasts are evaluated only for the Thursday within the forecasted week.
- Tropical storm days forecasts are tercile-based and evaluated only across basins which are active during the forecasted 13-week period:
- JJA and SON: Atlantic Ocean and North-West Pacific Ocean.
- DJF: South-West Indian Ocean and South-East Indian Ocean.
All variables are benchmarked against climatology. Additionally, skill scores associated with global variables are area weighted.
What is a Ranked Probability Skill Score (RPSS)?
Unlike deterministic predictions, probabilistic forecasts cannot be strictly classified as “right” or “wrong”, except in cases where probabilities of 0% or 100% are given. To evaluate probabilistic forecasts, a variety of verification methods have been developed.
In the AI Weather Quest, probabilistic forecasts are evaluated using Ranked Probability Skill Scores (RPSSs), as they allow for a comprehensive assessment across multiple categories within a probabilistic system, such as individual quintiles. The RPSS quantifies the deviation of forecast probabilities assigned to specific categories, in our case 20% intervals, to corresponding observations (Weigel et al., 2008).
Mathematically, the RPSS is defined by:
Here:
- Yk is the kth component of the cumulative categorical forecast vector Y.
- Ok is the kth component of the corresponding cumulative observation vector 0.
- Pk is the kth component of the cumulative climatological probability, which equals 0.2 for quintiles.
The following RPSSs under this framework can be interpreted as:
- RPSS = 1: the forecast has perfect skill compared to climatology, indicating the forecast is highly beneficial.
- RPSS = 0: the forecast has no additional skill compared to the climatology.
- RPSS < 0: the forecast is less accurate than climatology, indicating that the forecast lacks skill.
In addition to calculating weekly RPSSs, the AI Weather Quest also computes aggregated RPSS values for each competitive period and across the full competition forecasting year. The period-aggregated RPSS compares the ranked probability scores (RPSs) when using climatology and the forecast at every temporal and spatial point within the competitive period.
The aggregated RPSS is defined as:
where t represents temporal points and l represents spatial points. Please note that both weekly and period-aggregated RPSSs are area-weighted, meaning each spatial point contributes to the overall score proportionally to its geographic area.
Instead of averaging weekly RPSSs, which may not provide the most meaningful comparison between observations and forecasts, this more comprehensive aggregation approach is applied to ensure a more accurate evaluation.
Further details regarding different methods for evaluating probabilistic forecasts can be found in ECMWF’s forecast user guide.
What is a Brier Skill Score (BSS)?
Unlike probabilistic forecasts where the corresponding observation lies within a set of ranked categories, MJO forecasts do not naturally form a ranked set of categories. Although neighbouring phases may represent similar MJO states, each phase is treated as a distinct category, particularly when including the inactive MJO state. As a result, a verification metric based on ranked categories, such as the Ranked Probability Skill Score (RPSS), is less appropriate.
In the AI Weather Quest, probabilistic MJO forecasts are therefore evaluated using Brier Skill Scores (BSSs). The BSS measures the forecast improvement relative to climatology by comparing the forecasted probabilities of each MJO phase with the phase that was subsequently observed. A perfect forecast would assign a probability of 100% to the observed phase and 0% to all others.
Mathematically, the BSS is defined by:
Here:
- Y is the vector of forecast probabilities for each MJO phase.
- O is the observed MJO state represented as a binary vector, where the observed phase has a value of 1 and all other phases have a value of 0.
- P is the vector of climatological probabilities for each MJO phase.
The climatological probabilities are determined using 20 years of historical observations. To increase the sample size and improve the robustness of the climatology, observations from ±4 days around the target date are included at two-day intervals, providing a total sample of 100 observations.
The following BSS values can be interpreted as:
- BSS = 1: the forecast has perfect skill compared to climatology.
- BSS = 0: the forecast has no additional skill compared to climatology.
- BSS < 0: the forecast is less accurate than climatology, indicating that the forecast lacks skill.
As with the RPSS, BSSs are calculated for each forecast week and aggregated across a competitive period to assess overall forecast performance. Like period-aggregated RPSSs, we will compute period-aggregated BSSs using the following formula:
where t represents each forecasted day. This comprehensive aggregation of Brier scores provides a more accurate evaluation.
Further details regarding different methods for evaluating probabilistic forecasts can be found in ECMWF forecast user guide.
Forecast evaluation datasets
With the exception of tropical storms, all forecasts are evaluated against initial ECMWF Reanalysis version 5 (ERA5) release data (ERA5T). For tropical storm activity, forecasts are evaluated against International Best Track Archive for Climate Stewardship (IBTrACS) data.
For temperature and pressure
Forecasts will be evaluated against weekly averages computed from six-hourly data (0, 6, 12, and 18 UTC).
For precipitation
Forecasts will be evaluated against weekly accumulations derived from hourly data.
For Madden-Julian Oscillation indices
Forecasts will be evaluated against daily-mean data derived by hourly data.
For tropical storms
Forecasts will be evaluated against three-hourly tropical storm observations from IBTrACS.
The evaluation system relies on a climatological reference, which is derived by calculating quantiles and probabilities of historical data. For all variables, the climatological boundaries are calculated from 20 years of historical weekly data aggregated into weekly statistics. To enhance statistical spread, the sample size is expanded to 100 observations by supplementing with data from +/- 4 days at two-day intervals around the requested target date.
With the exception of tropical storms, ERA5 provides climatological quantile boundaries, while ERA5T is used for the most recent observations. For tropical storms, IBTrACS dataset is primarily used.
In addition to ERA5T-based evaluations, forecasts are compared against other global observational and reanalysis datasets such as IMERG and MSWEP. While these additional comparisons do not affect leaderboard rankings, they offer a more comprehensive assessment of AI/ML sub-seasonal forecasts.
Forecast evaluation tools
To ensure transparency and replicability of the evaluation techniques used, participants can download the evaluation code and evaluate their forecasts using the AI-WQ-package Python package:
- retrieve_evaluation_data.py retrieves the required datasets including observations and climatological quintile boundaries.
- forecast_evaluation.py computes evaluation metrics.
The evaluation code enables participants to self-assess their forecasts.
Understanding the leaderboards
Benchmarking ML models
Each week, leaderboards publicly display the latest rankings. Teams are ranked according to their best performing model.
For each metric, scores are computed for each forecast window (days 19-25 or days 26-32) and each variable. Additionally, a mean RPSS (variable-averaged RPSS) is calculated by averaging scores across all spatially-variable diagnostics (tas, mslp and pr).
A score of 0.0 indicates a climatology-based forecast (i.e. a uniform 20% probability assigned to each quintile bin).
Benchmarking against dynamical models
Leaderboards also include dynamical model skill based on forecast data from the S2S Database. These entries are not associated with a registered team and thus appear without a rank (“N/A”). Details regarding forecast and reforecast configuration can be found on the following confluence page.
Weekly and period-aggregated skill scores
Teams are ranked on two evaluation metrics:
- Weekly scores – tracking weekly performance.
- Period-aggregated scores – calculated from mean skill scores across all evaluated weeks within a competition period. They evolve as additional weeks are evaluated, providing a view of teams’ average performance over the competitive period. To appear on the period-aggregated leaderboards and be eligible for end-of-period recognition, participants must submit forecasts every week using the same model throughout a full 13-week competition period.
Historical leaderboards
Beyond real-time rankings, historical leaderboards display skill scores by competitive period and forecast initialisation date for reference. Models appear in the period-aggregated leaderboards only if they have submitted forecasts for the selected variable and forecast window for every week up to the chosen week.
Transparency, learning and recognition
Transparency and open outputs
The competition is designed not only to identify the best-performing AI-driven sub-seasonal forecasts but also to serve as a valuable resource for the broader scientific and AI communities.
To promote transparency and innovation:
- A public ECMWF-hosted sub-seasonal AI forecasting portal displays submitted probabilistic forecasts on day 5 of the forecast submission schedule, following the closure of the submission window. This portal enables spatial comparisons, potentially revealing patterns of model agreement and skill.
- Submitted forecast data are made publicly accessible via an AI Weather Quest Open Data portal at the conclusion of each competitive period.
- Team pages provide information on participating teams, while allowing participants to anonymise their participation if they wish. At the conclusion of each competitive period, submitted model summaries are made publicly accessible on these pages.
Community webinars and learning exchange
Periodic community webinars focus on sharing what has been learned from recent forecasting periods, highlighting promising approaches, and supporting knowledge exchange across the participant community.
These webinars showcase high-performing models, innovative methodologies, approaches developed by participants from diverse organisation types or with limited computational resources, and broader insights emerging from the competition. They provide an opportunity for participants to present their work, discuss what has contributed to model performance, and exchange lessons on the use of AI/ML for sub-seasonal forecasting.
