Norman Fomferra (NF), Marco Zulke (MZ), Jon Mittaz of NOAA (JM), Mark Filipiak (MF) and me (CM)
CM explained the purpose of the telecon. Colleagues at Brockmann are currently creating the Multisensor Matchup System, which is generally using BEAM (java) modules. The intention for the main processor is to use the state-of-the-art GAC readers from JM, which are Fortran. How best to ensure equivalence between the MMD and the future processor?
JM explained the context of his work developing a full set of recalibration procedures for all AVHRRs, which he is pursuing within NOAA's Climate Data Record initiative. He explained that the work most probably will lead to a new netCDF archive of AVHRR GAC and Fortran readers that people can apply to their current GAC and LAC data holdings.
CM commented that in the long run, therefore, the SST CCI may have a choice to use netCDF as input, in which case the reader issue would go away. However, as NF noted, this is not due within the current phase of the CCI.
NF explained that, given the information from JM, their preferred approach was to have an option to call the ARC processor (which is the basis of the SST CCI chain and incorporates the GAC data reader) such that it can output just the AVHRR calibrated Level1B in netCDF on an orbit-by-orbit basis as part of MMDB ingestion. A less good option would be a standalone implementation of the same functionality.
It was agreed that CM and colleagues at Edinburgh would confirm the possibility of the preferred approach ASAP (since they want to move to this within a couple of weeks). In addition, MF will supply some netCDF output as currently generated from the Edinburgh processor, and supply some estimate times for processing an orbit in this manner on the Eddie cluster.
CM thanked everyone for their time!
Wednesday 26 January 2011
Monday 17 January 2011
Leicester Telecon 5
Gary Corlett (GC) and me (CM) by skype
Various data issues were discussed.
ECMWF data. Following discussions with David Tan (DT), GC reports that other ECVs are being asked whether they might be better off with model level data, as we hope to use. ECMWF plans for full resolution delivery are for 6 months hence, so we will look at temporary alternative sourcing in the interim, as agreement with DT.
In situ drifting buoys. The AOML data set extends only to June 2010 with no immediate update schedule. We really wanted until end 2010, but will have to put up with thus until AOML do their next update (to be determined).
Multisensor Matchup System strategy. GC and CM agreed that ATSR-series and Metop single matches should not be discarded, in order, to maximise the ATSR1/2 overlap cases and to support geographically discriminating assessments of algorithms. Can't do the equivalent for AVHRR GAC unless we source an independent matchup data set (e.g. Pathfinder). GC expects a reasonable level of ATSR/AVHRR-GAC MMs with the current strategy. Agreed to review the density of ATSR/AVHRR GAC multi-sensor matches once we have created the MMD, in order to assess if additional GAC matches are required for some purposes.
ICOADS metadata. John Kennedy at Met Office has added these to the Hadley Centre's ICOADS data after we sourced it. Agreed GC will follow up with John to ensure, in particular, that metadata about Voluntary Observing Ship measurements are added to the MMS when possible.
Tools. We agreed that the priority is a functioning MMS and that the establishment of tools to operate on MMD output should be addressed only once the RRDP is in good shape.
Various data issues were discussed.
ECMWF data. Following discussions with David Tan (DT), GC reports that other ECVs are being asked whether they might be better off with model level data, as we hope to use. ECMWF plans for full resolution delivery are for 6 months hence, so we will look at temporary alternative sourcing in the interim, as agreement with DT.
In situ drifting buoys. The AOML data set extends only to June 2010 with no immediate update schedule. We really wanted until end 2010, but will have to put up with thus until AOML do their next update (to be determined).
Multisensor Matchup System strategy. GC and CM agreed that ATSR-series and Metop single matches should not be discarded, in order, to maximise the ATSR1/2 overlap cases and to support geographically discriminating assessments of algorithms. Can't do the equivalent for AVHRR GAC unless we source an independent matchup data set (e.g. Pathfinder). GC expects a reasonable level of ATSR/AVHRR-GAC MMs with the current strategy. Agreed to review the density of ATSR/AVHRR GAC multi-sensor matches once we have created the MMD, in order to assess if additional GAC matches are required for some purposes.
ICOADS metadata. John Kennedy at Met Office has added these to the Hadley Centre's ICOADS data after we sourced it. Agreed GC will follow up with John to ensure, in particular, that metadata about Voluntary Observing Ship measurements are added to the MMS when possible.
Tools. We agreed that the priority is a functioning MMS and that the establishment of tools to operate on MMD output should be addressed only once the RRDP is in good shape.
Friday 14 January 2011
Met Office Telecon 7
Me (CM) and Simon Good (SG) by skype
Aim was to decide on what SSTs to include in product outputs where variants retrievals are available. (From a given set of ATSR imagery, for example, SST can be estimated from all views and channels, or just one view and two channels, and variations in between these extremes.)
It had already been agreed at PM2 that the primary SST field in the L2P or L3U/C output would be the "best" available retrieval of skin SST. As optional fields, both a systematic and non-systematic uncertainty will be attached to this (instead of SSES and bias as defined by GHRSST). The time-and-depth adjusted SST (in the case of the long-term climate data record outputs) will appear as an optional field, with an additional (total) uncertainty estimate.
SG confirmed there is no apparent requirement from climate users for variants of SST retrieved. However, we both agreed that we might want to retrieve and keep variants (i) for R&D purposes, and (ii) in case such a requirement emerged. The solution we will propose to ESA is to create additional auxiliary files, each holding only one variant of SST and its two uncertainty components. To use these data, the corresponding L2P/L3U file would also need to be read to get the location etc of each corresponding record, the meta data, etc. But this makes it feasible to access variant types of SST in future without ballooning the size of the principal products.
Aim was to decide on what SSTs to include in product outputs where variants retrievals are available. (From a given set of ATSR imagery, for example, SST can be estimated from all views and channels, or just one view and two channels, and variations in between these extremes.)
It had already been agreed at PM2 that the primary SST field in the L2P or L3U/C output would be the "best" available retrieval of skin SST. As optional fields, both a systematic and non-systematic uncertainty will be attached to this (instead of SSES and bias as defined by GHRSST). The time-and-depth adjusted SST (in the case of the long-term climate data record outputs) will appear as an optional field, with an additional (total) uncertainty estimate.
SG confirmed there is no apparent requirement from climate users for variants of SST retrieved. However, we both agreed that we might want to retrieve and keep variants (i) for R&D purposes, and (ii) in case such a requirement emerged. The solution we will propose to ESA is to create additional auxiliary files, each holding only one variant of SST and its two uncertainty components. To use these data, the corresponding L2P/L3U file would also need to be read to get the location etc of each corresponding record, the meta data, etc. But this makes it feasible to access variant types of SST in future without ballooning the size of the principal products.
Thursday 13 January 2011
Project Meeting 2 at the Met Office
Attended by me (CM), Paul Spinks (PS), Gary Corlett (GC), Nick Rayner (NR), Matt Martin (MM), Norman Fomferra (NF), Martin Boettcher (MB), Simon Good and (partially) several other Met Office colleagues, and (briefly, by teleconference) Jacob Hoyer (JH) and Steinar Eastwood (SE).
I thought I'd note down a few reflections from this project meeting, although this blog in no way supersedes the formal minutes that will be produced by PS. It was quite a detailed technical meeting, focussed on reviewing the project to date and for the next year from a particular view point: in order to verify that the multi-sensor match-up system (MMS) we are creating will be fit for all the purposes we need it for. Several new capabilities for the MMS were identified, and NF and MB were extremely positive in their response to these newly-recognised requirements.
The MMS is key to this project, and to our vision of a system for producing satellite SST that has an effective improvement cycle capability. A multi-sensor match-up (MM) will comprise two or more satellite image extracts matched to an in situ SST observations, together with a powerful set of ancillary data. A system to identify categories of such MMs will powerfully support a lot of science, including algorithm development and multi-way uncertainty characterisation. Moreover, the system being designed (by GC, NF and MB) will include the ability to extended with data originating beyond our own project (e.g., data from other satellite missions we are unable to include).
At the meeting, several enhancements to our planned MMS capability were identified. These include: the flexibility to extract larger image areas (200 km) for classification work at high latitudes; the flexibility to include or exclude in a query multiple image extracts from a single sensor (e.g., from consecutive orbits); and additional categorisations of MMs so that we can easily identify, for example, all the MMs that occur in areas of high SST gradient or aerosol loading. The MMS is very ambitious, but will facilitate SST algorithm development greatly, and I think it will be a lasting legacy from our project (long after our actual data products are superseded by future reprocessing efforts!).
I thought I'd note down a few reflections from this project meeting, although this blog in no way supersedes the formal minutes that will be produced by PS. It was quite a detailed technical meeting, focussed on reviewing the project to date and for the next year from a particular view point: in order to verify that the multi-sensor match-up system (MMS) we are creating will be fit for all the purposes we need it for. Several new capabilities for the MMS were identified, and NF and MB were extremely positive in their response to these newly-recognised requirements.
The MMS is key to this project, and to our vision of a system for producing satellite SST that has an effective improvement cycle capability. A multi-sensor match-up (MM) will comprise two or more satellite image extracts matched to an in situ SST observations, together with a powerful set of ancillary data. A system to identify categories of such MMs will powerfully support a lot of science, including algorithm development and multi-way uncertainty characterisation. Moreover, the system being designed (by GC, NF and MB) will include the ability to extended with data originating beyond our own project (e.g., data from other satellite missions we are unable to include).
At the meeting, several enhancements to our planned MMS capability were identified. These include: the flexibility to extract larger image areas (200 km) for classification work at high latitudes; the flexibility to include or exclude in a query multiple image extracts from a single sensor (e.g., from consecutive orbits); and additional categorisations of MMs so that we can easily identify, for example, all the MMs that occur in areas of high SST gradient or aerosol loading. The MMS is very ambitious, but will facilitate SST algorithm development greatly, and I think it will be a lasting legacy from our project (long after our actual data products are superseded by future reprocessing efforts!).
Subscribe to:
Posts (Atom)