Telecon with Chris Merchant near end of their discussions on 26 November
GC and BC staff collocated at BC from 24 – 26 Nov in order to lay down the system requirements for the SST Multi-sensor Match-up System (MMS) and to sketch a rough development plan for its first version to be ready by the end of Jan 2011.
On day 1, GC gave an overview about the SST cal/val work and the role of the MMS for the SST_cci project. Various related projects have been discussed, especially the (A)ATSR L2P Project and GHRSST with respect to their Match-up databases.
For subsequent discussions on the MMS requirements, the team identified the possible MMS users, the various datasets it creates and agreed on the following definition of terms:
· MD – Single-sensor match-up dataset. It is usually a NetCDF file comprising single-sensor match-up records. Each record comprises a single reference (in situ, in hi-lat cases “dummy in-situ” with invalid SST value) with a single satellite. MDs are ingested by the MMS and are versioned and archived for full backward traceability.
· MMD – Multi-sensor match-up dataset comprising multi-sensor match-up records. Each record is a coincidence of a single in situ (or dummy) with multiple satellites and auxiliary data. MMDs are extracted from the MMS as NetCDF files in a well-specified format.
· MMS database – The database that stores the in-situ data, the searchable fields of all ingested MDs and all detected coincidences (match-ups of in-situ-satellite and satellite-satellite).
Day 2 was fully dedicated to the gathering of system requirements which have been written down in a SST_cci Toolbox Wiki at https://github.com/bcdev/sst-cci-toolbox/. GC will integrate these in the Product Validation Plan document.
IR SST algorithm improvements are taking place from project kick-off+6 onwards, so a first version of the data set (DBT2) deliverable to support these improvements will be made available to algorithm developers. This will be the Algorithm Testing MMD – an extract from the MMS database. This MMD shall be ready by end of Jan 2011. It will contain prescribed "test" match-ups (for algorithm development) and "training" match-ups (for fair algorithm assessment). It will exclude some match-ups that are designated as "reference" match-ups, which will be used as the Reference MMD later in the project.
The first version of the MMS will have a simple command-line interface for ingestion and extraction of data and database inspection. As part of the ingestion process, the MMS will compute coincidences from MD input files. In addition to the coincidence criteria where there is always an ATSR observation, it has been agreed that triplets of Metop+ SEVIRI+PMW should also be collected. During collocation, the MMS will assign additional satellite and auxiliary data including interpolated NWP values, aerosol and sea-ice data. For each match-up, also a 24h in-situ history (+/-12h from satellite observation time) will be included. Investigations will be made to let the MMS directly call RTTOV-10 (simulation model) and ingest the RTTOV-10 output automatically back into the MMS database.
There are still unknowns regarding the physical deployment of the MMS to the ECDF (the Edinburgh computing cluster). In the next few days, BC will compile the external system requirements (e.g. DBMS, web server, ssh, Java JDK) and send it to Marc Filipak and Owen Embury. A telecon will be held on Nov 13, 10:30 (UK time) dedicated to address the MSS/ ECDF requirements.
Day 3 was used to review the requirements and develop a detailed action list in order to meet the MMD delivery at KO+6, by end of Jan 2011. The immediate actions for Dec 2010 are:
BC: Send external system requirements to Marc and Owen
BC: Update BEAM to also read NOAA AVHRR GAC
BC: Provide existing (A)ATSR childgen tool to Gary
GC: Describe data items to be used for the first month of data
BC: Develop flexible database schema which lets us add fields at any time
BC: Develop ingestion function to convert MD files and compute coincidences
BC: Develop extraction function to generate a MMD from given query
GC: Provide BC with one month of data (all different types)
GC: Convert MD files to monthly, reformat old SEVIRI to NetCDF
GC: Update PVP document with MMS requirements
CM: Provide BC with the RTTOV-10 configuration
BC: Update BEAM to also read NOAA AVHRR GAC
BC: Provide existing (A)ATSR childgen tool to Gary
GC: Describe data items to be used for the first month of data
BC: Develop flexible database schema which lets us add fields at any time
BC: Develop ingestion function to convert MD files and compute coincidences
BC: Develop extraction function to generate a MMD from given query
GC: Provide BC with one month of data (all different types)
GC: Convert MD files to monthly, reformat old SEVIRI to NetCDF
GC: Update PVP document with MMS requirements
CM: Provide BC with the RTTOV-10 configuration
CM discussed the progress with GC and the BC staff in a telecon during day 3.
DONM by telecon, Dec 13, 10:30 (UK time) dedicated to address the MSS/ ECDF requirements
No comments:
Post a Comment