Assumptions for System Requirements preparation

The project team is putting together a system requirements document for our vision of the implementation of our current prototyping work. This cannot be done independently of a scenario for what should be within scope for the system, from among all the things we could do. I was asked to prepare notes with the team on the scenario to specify. The outcome is copied below. Note that this is subject to change, although we expect the scope below to be close. The System Requirements Document in the end will be the document of reference, once written and accepted.

Scope and context of scenario for system requirements document

Introduction

The System Requirements for SST CCI follow from a number of considerations.

First, there are User Requirements, where “User” here refers to those looking to use outputs of the SST CCI system. The prime sources of information about User Requirements are (1) the SST CCI User Requirements Document (URD), (2) the SST CCI Technical Note synthesizing URD survey results into quantitative targets for aspects of products, and (3) the GCOS ECV satellite supplement and its update.

Second, there are ESA requirements for the CCI programme as a whole, expressed as five Cardinal Requirements in the Statement of Work that initiated the present Phase 1 activity within CCI.

Third, there are Science Team requirements. A Science Team will need to work within the SST CCI system to deliver adaptation to changes in the satellite constellation over time and the cycle of improvement of the full Climate Data Record (CDR) as understanding and techniques improve. In effect, we must consider the requirements of these “internal users”.

Fourth, there are judgements, with justifications, by the SST CCI Science Team of scope for the SST CCI system that is scientifically credible within a conceivable cost, while taking account of all the considerations above. As will be clear below, “a system that does everything for SST” in CCI Phase 2 would be extremely costly, in terms of both the infrastructure and volume of research and development that would be required.

Overview of how the system is required to operate

See http://sst-cci.blogspot.com/2011/10/what-should-routine-climate-system-look.html

The potential satellite constellation for SST

It is useful in this first section of this System Requirements Document (SRD) to review what a full constellation of observations for an SST system could encompass, and within that to prioritise and comment the various components. We then define the components included in the scenario used as the basis for this SRD.

The present SST CCI Phase 1 is funded until mid 2013. The system specified in this document is to be implemented and used for routine production in the period 2013 to 2016. It is necessary to look beyond 2016, but the constellation up to that time is the focus here.

Scientifically credible baseline

Climate users require global SST coverage [UR-REF-2; UR-QUF-42]# and products that cover at least 30 years [UR-REF-3; UR-QUF-43]. The only sensor series that addresses these requirements is that of the Advanced Very High Resolution Radiometer (AVHRR), via archives of level 1b Global Area Coverage (GAC) data.

Climate users also require bias to be less than 0.1 K, demonstrated on spatial scales of 100 km [UR-QUF-49]. The 0.1 K bias target has been demonstrated for Advanced Along Track Scanning Radiometer (AATSR) SST-20cm globally at scales of order 1000 km, for SSTs from the ATSR Reprocessing for Climate (ARC) project [RD.184] using radiative transfer based algorithms that are independent of in situ observations. It is likely to have been met for all the ATSRs in the ARC time series, at least back to 1993, except at the end-of-life of ATSR-1 (second quarter of 1996).

On the other hand, for AVHRR this bias requirement is known to be violated, despite empirical tuning to drifting buoys. The limitations of the non-linear SST retrieval formalism [RD.221, RD.248] lead to biases of up to 0.5 K in the tropics. Within Phase 1 of the CCI programme, the SST CCI team has therefore chosen to base its long-term climate data record on ATSR and AVHRR together, with the intention that AVHRR be brought into line with ATSR to address this bias requirement.

Part of the reason ATSRs can obtain lower biases, particularly in the tropics, is the greater information content delivered by having dual view observations. The future dual view sensor will be the Sea and Land Surface Temperature Radiometer (SLSTR). The present launch schedule for SLSTR and end-of-life schedule for AATSR promise an adequate 6 month overlap to give continuity of an independent accurate. SLSTR will take the role of the ATSRs in the SST CCI system.

Nonetheless, there is some risk that AATSR and SLSTR overlap may not be achieved. The Metop AVHRRs become particularly important in this connection. It is probable that at least one Metop AVHRR (likely Metop-B) will bridge any AATSR to SLSTR gap. Metop will be in a stable orbit (most AVHRRs are on platforms whose local times drift) with an equator crossing time within an hour of AATSR and SLSTR.

Unlike most AVHRRs, Metop AVHRR is downlinked globally at full spatial resolution. This means that it can, in principle, also bridge AATSR to SLSTR with regards to global 1 km resolution. Cloud detection undertaken at 1 km rather than GAC resolution is also likely to be more effective.

Thus, if it is necessary to carry the independent calibration of AATSR forward to SLSTR indirectly because they do not overlap, the bridging sensor will be Metop AVHRR.

There are three options for introducing Metop into the SST CCI system, with different costs.

The lowest cost option, used within CCI Phase 1, is via an interface to the system of the Centre de Meteorologie Spatiale (CMS) (the SST team there is also part of the EUMETSAT Ocean and Sea Ice Satellite Application Facility, OSI-SAF). After full resolution cloud detection, CMS create 0.05 deg resolution clear-sky brightness temperature products with a large range of useful additional information. An interface to CMS brings these intermediate files into the SST CCI system. The breakthrough requirement for spatial resolution of L3 SST is 0.1 deg [UR-QUF-36; UR-QUE-32], and two thirds of climate users would be satisfied at objective level by 0.05 degree, so interfacing to spatially averaged data at 0.05 deg resolution is reasonable. This approach would allow SST CCI to (re)process 0.05 deg data for the SST retrieval step. However, it would mean there is no SST CCI ability to reprocess for cloud and ice detection improvements. Archive reprocessing for Metop is in the plans of CMS, and under this option the future SST CCI system would depend on this for cloud and ice detection improvement. So, with this option, Metop would not bridge between AATSR and SLSTR at full spatial resolution, and the in-house reprocessing capability would be limited to SST retrieval methods.

The middle cost option is to interface with the CMS system at full spatial resolution, to access a 1 km level 1b, with CMS cloud detection and other auxiliary information that would be determined. This would incur additional costs for both the CMS system and the SST CCI system and the data flows between the systems. Although CMS cloud detection could be used for short-delay mode, the data collected in the SST CCI system would permit in principle reprocessing for cloud and ice detection at full resolution, as well as SST retrieval.

The high cost option is to obtain level 1 full resolution Metop brightness temperatures at the SST CCI system and do all processing (short delay mode and reprocessing). This would increase the consistency between the short-delay Metop SSTs and any reprocessed Metop SSTs, at the cost of not taking advantage of an existing operational system in Europe for short-delay mode production.

The Metop series will be the final AVHRRs flown, and should continue until 2021. The Visible/Infrared Imager Radiometer Suite (VIIRS) has been launched in October 2011 on the NPP mission (planned end of life August 2016), and is the functional replacement for AVHRRs in the Joint Polar Satellite System (JPSS) (JPSS-1 launch planned in July 2017; note there is no continuity therefore with the NPP VIIRS). Relevant VIIRS channels are similar in conception to those of the earlier Moderate-resolution Imaging Spectroradiometer (MODIS). Nonetheless, MODIS and VIIRS are likely to have quite distinct instrumental characteristics that would need to be carefully related to the ATSR and AVHRR series. Moreover, there is a cardinal requirement on the programme [CR-3] to optimize the impact of European sensors in CDR production. This is addressed by the intention to use ATSR/SLSTR as reference sensors in the baseline constellation to which other observations are tied, but may also be taken to imply that MODIS and VIIRS (at least prior to JPSS-1) should be given a lower priority.

Users have requirements for stability of observation [UR-QUF-50] with ⅔ of climate users surveyed satisfied at “breakthrough” level by drift of less than 0.05 K per decade for L3 SSTs. On the available evidence (limited to tropical areas by lack of stable in situ observations elsewhere) this objective stability is met for 1993 to 2010 by ARC SSTs, while preserving independence. Assuming adequate AATSR-SLSTR or AATSR-Metop-SLSTR overlap, the SST CCI system must support continuation of this level of stability (after cross-calibration using overlaps), using the ATSR-series as an anchor for AVHRRs. The system needs also to support improvement of stability post-Pinatubo (1991 to 1993). Prior to ATSR, only the AVHRR series are available to meet the temporal coverage requirement, and the system will need to support exploitation of overlaps through the AVHRR record to optimise stability.

To achieve exploitation of overlaps, cross-referencing of sensors and the stability objectives, the system must support an extension of the Multisensor Matchup System for all these sensors.

Stability is arguably the quality that should most distinguish a ‘Climate Data Record’ from a mere ‘Data Record’. It is also a quality that is easily destroyed by using input data streams with markedly different sampling or instrumental characteristics. All the sensors discussed to this point are on polar orbiting platforms and operate in the infra-red with broadly similar channels. They provide a relatively coherent set of input data streams for development of a consistent CDR for SST. Together they should support a baseline SST CDR addressing the central user requirements for coverage, resolution, bias and stability. Table 1 summarises some characteristics and priority within the SST CCI system.

Sensor series	Temporal coverage	Input data description	Approximate input data volume by 2016	Priority / comments
AVHRR (exc. Metop)	1981# - 2016	GAC (sub-sampled to 4 km at nadir)	25 TB	Essential for baseline [UR-REF-3]
ATSR	1991 - 2014	L1b 1 km	26 TB	Essential for baseline [UR-QUF-50; UR-QUF-49; CR-3]
AVHRR Metop	2006 - 2016+	0.05 deg clear-sky BTs + auxiliary data	10 TB	Essential for baseline (unless full resolution is used) [UR-QUF-50]
		Full resolution L1b + aux.	50 TB	Highly desirable for baseline (would be essential if 0.05 deg option did not exist) [UR-QUF-50]
SLSTR	2013 - 2016+	L1b 1 km	180 TB	Essential for baseline [UR-QUF-49; CR-3]
VIIRS	2012 - 2016, 2017 onwards	L1b 1 km	40 TB	Highly desirable for baseline. Will become essential, from JPSS-1 in 2017.
MODIS	1999 - 2014(?)	L1b 1 km	200 TB	Desirable for baseline. Priority for Terra mission over Aqua mission if only one were feasible.

Table 1. Baseline constellation of sensors -- characteristics and relative priority.

The central SST CCI product is and will remain a daily product (day and night separate), including optional adjustments to a fixed local time of observation to eliminate aliasing of variable sampling of the diurnal cycle into the long term record. The essential components in Table 1 will support this, with desirable and highly desirable elements providing increase spatial coverage and decreasing sampling uncertainty.

A significant proportion of climate users require sub-daily temporal resolution of SST, i.e., to have a daily diurnal cycle estimate for each place (e.g. with 3 hourly resolution) [UR-QUF-40] or to obtain SST estimates at fixed synoptic times (in universal time, i.e., a snapshot of different phases of the diurnal cycle with longitude, either daily [UR-QUF-37] or 6-hourly [UR-QUF-41]). Such estimates are supported with the baseline constellation for those periods where a sufficient number of polar orbiting sensors are flying with a range of local equator crossing times. The research has not yet been done to establish how best to do this and for what sample of the full record this is feasible with a given uncertainty. Adding the desirable and highly desirable elements in Table 1 would increase temporal resolution as well as coverage, likely improving the ability to address sub-daily SST.

Addition of passive microwave sensors

Passive microwave (PMW) sensors have lower intrinsic accuracy than many of the baseline IR constellation, and coarser spatial resolution. However, they add considerable increase in spatio-temporal coverage in open-ocean areas (roughly, more than 50 km from coasts, islands and sea ice) because their SST observations are prevented or degraded only by actively precipitating clouds and radio frequency interference. This means each PMW obtains SST for a significantly greater proportion of the ocean in a day than does an IR.

PMW sensors therefore could play a useful role in the SST CCI system by (i) reducing sampling uncertainty, and (ii) increasing the observational resolution of the diurnal cycle, thereby improving sub-daily SST information.

However, caution is required, in that :

(i) We don’t know the degree to which superior PMW sampling (but with inferior single pixel retrieval uncertainty) will reduce total uncertainty, since propagation of realistic context-sensitive uncertainty from the L1 to L2 retrieval process to L3 and L4 products has never been systematically investigated.

(ii) Having very different instrumental and sampling characteristics, naive inclusion of PMW is most likely to degrade the stability of the CDR. This is because they are available comparatively late in the satellite era, there are relatively more significant differences between sensors, and the overlap between missions is not likely to be ideal. In principle, research can be conducted to analyse and learn to remove relative biases compared to the IR constellation. But when dealing with stability targets at the level of 0.005 K/yr, practical success in this is not to be lightly assumed. It will be necessary to maintain the SST CCI Phase 1 approach of having both polar-IR only and polar-IR+PMW versions of L4 products.

(iii) With respect to CR-3 (maximising the impact of European sensors), this is done by research to cross-reference PMW observations to the ATSR/SLSTR series (to obtain both relative bias adjustments and consistent uncertainty estimates). In the current phase, this will be attempted at L2 level. It is likely that a better approach would be to work from L1 in combination with L2, so this is assumed hereafter.

On the positive side, PMW data streams are relatively modest in data volume, so any benefits they bring may be cost-effective. It will also be an opportunity to increase practical expertise with PMW SST in Europe.

Table 2 summarises some characteristics and priority within the SST CCI system.

Sensor	Temporal coverage	Input data description	Approximate input data volume by 2016	Priority / comments
TMI	1997 - 2013	5 chans (10.7 to 85 GHz), ~25 km resolution	...	Highly desirable
AMSR-E	2002 - 2011	...	...	Highly desirable
AMSR-2	2012 - 2016			Highly desirable
SSMR	...	...	...	Optional: experimental
WindSat				Highly desirable once performance is validated to be as expected

Table 2. Passive and active microwave extension to constellation of sensors -- characteristics and relative priority.

Addition of geostationary sensors

To address better the requirements of some climate users for sub-daily information on SST, a possible extension to the baseline constellation is to include SST-capable sensors on geostationary sensors.

Since global coverage is a requirement, this would require use of geostationary data for the full range of longitudes. This introduces a large diversity of sensors and data formats. While the dual view capability of ATSR/SLSTR makes cross-calibration of geostationary sensors at level 1 feasible using a multi-sensor match-up system (MMS), it is nonetheless a major research effort to do this for a comprehensive suite of geo sensors. To achieve climate-quality stability across time and longitudes would demand detailed sensor-by-sensor work. Bayesian approaches to cloud detection are sufficiently general to apply in a consistent manner across all the sensors, but would required development of forward-model adjustments. The design of optimal estimators for geo sensors has been demonstrated on SEVIRI, but there are complexities related to trading between SST sensitivity and retrieval noise levels. A very large increase in input data volume would be entailed

Assuming success, the main benefits would be systematic resolution of diurnal cycle for cloud free areas, increased coverage and reduced sampling uncertainty.

On balance, inclusion of geostationary sensors would be scientifically “Desirable” (not accounting for considerations of cost effectiveness).

However, there is an additional mode of use for geostationaries, as components of the Multi-sensor Match-up System (without being part of the main product stream). This is because diurnally resolved products are specified in the SRD scenario (below), and SEVIRI in the MMS provides a means of assessing these more effectively. A full constellation of geostationary SST-capable sensors would be “desirable”, whereas implementing at least one is “essential”. The team is most familiar with SEVIRI and is satisfied with its use for this application.

Conclusion on satellite constellation

For the purposes of this SRD, the assumption is that all the “Essential” and “Highly Desirable” data streams will be required in the system. That is, in summary:

• ATSR series
• AVHRR GAC series
• AVHRR Metop series at full global resolution
• SLSTR
• VIIRS
• AMSRE-E
• TMI
• AMSR-2
• WindSat SST

In addition, it will be assumed that matched data from other sensors can be accommodated in the Multi-sensor Match-up System, and that in particular this will include:

• SEVIRI

Reprocessing capability

The SST CCI system will need to be able to undertake reprocessing of historical data in parallel to routine short-delay processing. The absolute threshold functional requirement is that all input L1b can be reprocessed to L2/3/4 in six months of continuous processing, simultaneously to routine processing, use of the multi-sensor match-up system (see below) and and computational services to users. This permits an updated cycle on the CDR of order 1 to 2 years.

However, this threshold configuration would be scientifically sub-optimal, in that it does not permit the science team to undertake internal reprocessing (as envisaged above in section ??).

To maximise the scientific quality of the CDR (i.e., to maximise the investment in having a science team working on the SST CDR improvement cycle), the internal reprocessing capacity needs to support relatively short internal reprocessing cycles and verification that changes made have had the required positive effects on accuracy, stability, etc, without introducing obvious new artefacts.

The breakthrough requirement scientifically for reprocessing capacity is:

• System is able to reprocess all data streams from L1 to L2/3/4 in two months, including updating data in the MMS. Individual components of total data stream can then likely be reprocessed on time scales of weeks, which allows a reasonably interactive and efficient improvement cycle, with possibility of fixing “unintended consequences”.
• System is able to store outputs of at least the last three internal reprocessing runs for all input streams, for inter-comparison purposes.
• System is able simultaneously to continue routine processing, use of the multi-sensor match-up system (see below) and and computational services to users.

The target requirement scientifically for reprocessing capacity is:

• System is able to reprocess all data streams from L1 to L2/3/4 in one month, including updating data in the MMS. Individual components of total data stream can then likely be reprocessed on time scales of days, which allows a fully interactive and efficient improvement cycle, with rapid progress on “unintended consequences”.
• System is able to store outputs of at least the last three internal reprocessing runs for all input streams, for inter-comparison purposes.
• System is able simultaneously to continue routine processing, use of the multi-sensor match-up system (see below) and and computational services to users.

The assumption in this SRD is that the target scientific requirement be met, since maximising the scientific quality of satellite CDRs for climate is the principal reason to have the CCI programme.

Scope of Products

The range of SST CCI products to be generated and distributed affects the system requirements in various ways, most obvious in the data volumes associated with these.

The output product assumptions made are as follows:

• L2P SST-skin (or subskin) with time-adjusted SST-20cm for all sensors in constellation
• L3U 0.05 deg clear-sky SST-skin, time-adjusted SST-20cm and brightness temperatures for ATSR and SLSTR
• Daily (day and night separate) 0.05 deg L3C SST-skin (or subskin) with time-adjusted SST-20cm for all sensors in constellation
• Daily L4 satellite-only analyses at 0.05 deg of
• SST-skin, SST-20cm and SST-foundation

for the following sensor sets:

• full-archive IR analysis (AVHRR GAC, AVHRR Metop, ATSRs, SLSTR, VIIRS)
• best-coverage analysis (all sensors)

with enhanced error covariance information (assume this doubles the output data volume)

• Diurnally resolved (3 hourly) L4 analyses of time-adjusted SST-skin and SST-20cm (assume this is possible since 2002 without geostationary sensors)

Interaction with users

The system needs to accommodate the foreseen demands for storage, serving and on-the-fly computation that are seen as necessary to support interactions with users.

The assumptions about these interactions are:

• All formally released versions of CDRs are accessible to users by http, ftp and OPeNDAP in perpetuity (meaning, serving in perpetuity need not be supported within the SST CCI system)
• The currently released CDR is accessible to users by http, ftp and OPeNDAP in an SST-specific infrastructure that also supports web services [UR-QUF-90; UR-QUF-91; UR-QUF-93]

• Web services will comprise
• Time-series, regional averaging and spatio-temporal gridding tools with download, applicable to L2P [UR-QUF-97]
• Time-series, regional averaging and spatio-temporal regridding tools with download, applicable to L3C and L4 [UR-QUF-97]
• Extraction of data subsets (including whole time-series, limited area requests, and via an interactive map) [UR-QUF-98; UR-QUF-92; UR-DIS-124]
• Visualisation of data, including of quality and uncertainty information [UR-QUF-99]
• Climatology and annual average generation [UR-DIS-130]
• These web services must
• Provide outputs for download in a reasonable time (<1 hour for the maximum request)
• Provide users with information on progress of the request
• Be flexible as regards the selection data sets to include (e.g., a user defined mix of sensors or analyses)
• Be flexible with regard to the climatology assumed when doing manipulations via SST anomalies (user-defined climatology)

• There is a help system for CCI that can be contacted by users. There is a responsibility for the SST CCI science team to answer questions [UR-DIS-119]
• There is a customer relationship management system with tickets to trace communication and to route questions to operations or science team
• There are subscription mechanisms (RSS, ATOM, email) in place to inform interested users about new versions, important data issues etc. [UR-QUF-95; UR-QUF-96; UR-LLP-27]

• Users can access MMD extracts from the MMS with a flexible set of criteria for subsetting
• Users can contribute MMD columns (primarily their SST retrievals) to the MMS (with prior agreement of science team) in order to facility future algorithm inter-comparisons, by a means that is user-friendly but also robust against poor quality ingestions. [UR-REF-23]
• Users can contribute feedback and raise data issues in a tracked manner, with issues raised visible to other users [UR-REF-9]

Further considerations

In the CDR improvement cycle, the science team and authorised scientists will need to work with an efficient Multisensor Matchup System (MMS). The assumed capabilities for this MMS are:

• The MMS will be able to generate a complete MMDB from a new locations list (in situ and dummy), with appropriate quality control and error handling, in a fully automated manner, within 1 month
• The MMS will achieve the routine update of the MMDB with additional in situ observations and locations
• The MMS will support facilities for extraction of small images (for retrieval work) and large images (for classification work)
• MMS tools for subset extract and delivery, plus ingestion tools that ensure validity of ingested data (for science team use, and as an external web service with science team authorisation)

To support the CDR improvement cycle, the system will accommodate an algorithm development environment for use of the science team. In this context, core algorithms for science team will be under version control in a place where the science team can experiment with code, test the effect on MMD statistics, and then do internal reprocessing (prior to fixing code version for formal reprocessing) and assessment of results (using assessment tools).