Introduction¶
VMPZ-ID products, used in other LE3 Processing Functions, consist in (with increasing complexity):
reconstructing accurate survey footprint (using MER tiling in HEALPix MOC)
collecting bit Masks derived at previous levels (VIS, NIR, EXT) and construction the coverage mask (footprint+not masked regions in bit masks)
collecting instrumental/environmental information needed to compute the limiting magnitude
estimating the depth maps (i.e. object independent)
estimating the photometric visibility mask and the random catalog for a given selection used for 2PCF/PK-WL.
VMPZ-ID is triggered at the MER tile level, where all products are first produced as (potentially weighted) MOC Healpix maps, at a given MOC order. In the following, this stage is named the stage 1.
A second step, prior to triggering other “client” LE3 processing functions implies “merging” the MOC maps, and a HEALPix partial map or MOC map is produced at the resolution prescribed by the user in the parameter files. Potential mangle maps could be derived from this product. In the following, this stage is named the stage 2.
Note
I am not sure there is a use case for a mangle mask, where here only the compression of information performed by mangle would be used (and not its arbitrary resolution property). Keeping mangle outputs essentially double then number of output products. Besides, it is not sure that the HEALPix partitioning of the sky would be respected sufficiently accurately by such post-processing to produce the random catalog.
Warning
For parameter files, the use cases are important: right now the user specify which band/instrument he wants to process, implying that potentially not-used data is collected from the EAS which is not efficient. So:
do we want one pipeline definition per band/instrument and macro-pipelines for several bands (per instrument)? In this scenario, no need to specify which band we want (we select the pipeline).
Advantage: fine granularity allowing to reprocess single band/instrument
Problem: very large number of pipelines to manage, manual selection of pipeline.
do we want to collect all products and work with a subset of it, specificied in the list of parameters?
Advantage: one (or few) pipelines to manage
Problem: identify the needed products, not efficient if only a few bands need processing
do we want to have a lot of optional inputs in the pipeline def, and only process those that have been selected?
Advantage: one (or few) pipelines to manage
Problem: not very automatic, “manual” selection of products
I would rather go for option1, with macro-pipelines per instrument.
Warning
We have to define the granularity of outputs: single band of an instrument in a file, or all bands? Here we have chosen a single band per file. This is the most flexible solution in terms of processing (e.g. single band reprocessing, different processing for EXT and Euclid) and use cases (we can get the info for any band of any instrument directly from archive), but implies that if a client PF wants to collect all infos it will need to know exactly which ones it needs to use, if all relevant products have been processed, and it will have a variable number of input products over the MER Tiles that it needs to collect and process accordingly. Maybe we need to add a flag indicating in the output which bands are relevant for a given query, and which were processed (not yet implemented)?
If we want to let’s say gather all products in a single one at a Skypatch level (if no use case for individual products), to flag a consistent and complete product, then another pipeline collecting the individual HEALPix objects and outputting a single HEALPix product could be run. The easiest way is to create a data product that is a collection of such individual products (purely metadata).
We can also have here a single self-contained output with all individual products in a FITS file at the skypatch level:
the current HEALPy implementation implies that multiple maps (corresponding to different bands/instruments) are stored as columns. In the XSD data model, the columns need to be fixed, with a name. This implies that we cannot have a variable number of columns for a product. Consequently, using baseline HEALPy all possible bands need to appear as columns whatever the sky area covered (not very efficient)
the alternative is to have one hdu per band with specific keywords, as the multiplicity of hdus is not fixed in the DM. In this scenario the write_map routine from HEALPy needs to be adapted (from what I see, should not be a huge effort). The advantage of this solution is that we keep the flexibility of columns for other information per band (e.g. the bit masks, one column per bit) Several options are possible here if we want a self-contained product:
Finally the concept of “release” introduced recently could “solve” the downstream problem of knowing which consistent products needs be used, if a same release number could be applied to various products at different levels, as we suffer from multiple inputs/outputs potentially (re-)processed at different times by potentially different PFs and we want consistent groups of products. It is however not clear to me how/when this release number is applied to the products.
Warning
It is important to know whether the product are produced on a VIS/NIR/EXT frame or a MER tile or on observation. Here, I assumed MER tile as we discussed.
Note
Output products contain an identifier of the processed sky area, either TILEINDEX as MER does or an identifier defined by MER corresponding to a list of tiles.
Survey footprint¶
For the survey footprint, beside parameters VMPZIDParamsFootprint the footprint requires:
for stage 1 (processing on MER tiles), products containing the list of coordinates describing the geometry of the survey for the set of given band/instrument are first ingested (e.g.: the ReconsFPAPointing in VIS Obs Frame Product for NIR/VIS, the ImgSpatialFootprint in EXT CalibratedFrame if precise enough). The fraction of HEALPix pixel covered is reported (see below the note) in a VMPZIDMOCFootprintMask.
for stage 2 (processing at skypatch level), a set of MOC HEALPix Maps VMPZIDMOCFootprintMask are processed on the different MER tiles overlapping with the prescribed region.
The two stages will produce a VMPZIDHEALPixFootprintMask or VMPZIDMOCFootprintMask. In case of sky patch processing, a mangle file VMPZIDMangleFootprintMask can also be outputed, using VMPZIDParamsMangle parameter files.
Note
The convention for reporting pixel coverage in MER is 1 for fully non observed region and 0 to fully observed regions. For mangle this is the opposite: fully observed regions are set to 0 and fully not observed to 1. This needs to be consistently propagated. Here in the following we assume MER convention: 1=fully not observed.
Bit Masks and coverage mask¶
The coverage masks correspond to regions of the survey footprint that have not been flagged using the bitmask. It is therefore the same format as the survey footprint, but the flagged areas are set to 1. Parameters are set in VMPZIDParamsCoverage.
The two stages will produce the product VMPZIDHEALPixCoverageMask or VMPZIDMOCCoverageMask. In case of sky patch processing, or a mangle file VMPZIDMangleCoverageMask can also be outputted, using VMPZIDParamsMangle parameter files.
For the bit masks, besides parameters VMPZIDParamsBitMask, and a HEALPix VMPZIDMOCFootprintMask, one would need:
for stage 1, the pixel flag maps defined in MER Mosaic Product or (see in human readable format) for the different bands/instruments, to produce VMPZIDMOCBitMask
for stage 2, a set of VMPZIDMOCBitMask computed on the different MER tiles overlapping with the prescribed region to produce VMPZIDHEALPixBitMask or VMPZIDMOCBitMask
In case of sky patch processing, a mangle file VMPZIDMangleCoverageMask can also be outputed, using VMPZIDParamsMangle parameter files.
Note
In the MER Mosaic Product product (including flags), all bands are rebinned in a grid according to the VIS pixel grid. Is this the product we need, or do we need to process also the Calibrated Frame Product?
Instrumental/Environmental Information¶
Using parameters VMPZIDParamsInfoMaps, MOC maps of instrumental and environmental effects are first collected in a product VMPZIDMOCCollectInfoMaps. This implies to have access to all relevant statistics (mean/median/max/min?) on relevant quantities to compute the limiting magnitude, such as PSF/seeing FWHM, background level, RMS map, exposureTime, airmass,solar aspect angle, zodiacal light emissions.
In the table below are listed a series of potential inputs to collect this maps.
Instrumental/ Environmental Parameters |
Statistics |
Instrument |
Input Product |
|---|---|---|---|
Noise |
RMS (pixel) |
VIS |
Calibrated Frame Product (WeightStorage) |
RMS (pixel) |
NIR |
NIR Calibrated Frame (reducedFrameFitsFile) |
|
RMS (pixel) |
MER |
Mosaic Product/Background-subtracted Mosaic Product (RmsStorage) |
|
Weights |
RMS (pixel) |
EXT |
ExtCalibratedFrame (DataStorage) |
Value |
DES |
DesCalibratedFrame (desObservation) |
|
Value |
KIDS |
reducedScienceFrame (Weight)? |
|
RMS (pixel) |
LSST |
ExtLssCalexp (lssCalexpFitsFile) |
|
Exposures |
total number |
VIS |
Calibrated Frame Product (ObservationSequence) |
total number |
NIR |
NIR Calibrated Frame (ObservationSequence) |
|
time |
MER |
Mosaic Product/Background-subtracted Mosaic Product (imgBaseFrame) |
|
time |
EXT |
ExtCalibratedFrame (imgBaseFrame) |
|
time |
DES |
DesCalibratedFrame (imgBaseFrame) |
|
total time |
KIDS |
reducedScienceFrame (ExpTime single exposure) |
|
time |
LSST |
ExtLssCalexp (imgBaseFrame) |
|
PSF |
mean FWHM |
VIS |
Needed? over FOV? For a stack/exposure? Wavelength? |
NIR |
Needed? over FOV? For a stack/exposure? Wavelength? |
||
Center value |
MER |
Mosaic Product or Background-subtracted Mosaic Product (merPsfModelStorage) |
|
FWHM |
EXT |
ExtPSF (extPsfModel, keyword FITS) |
|
Seeing |
median |
DES |
DesCalibratedFrame (PsfModelStorage, FHWM) |
median |
KIDS |
reducedScienceFrame (Seeing) |
|
LSST |
ExtLssCalexp (lssPsfModelFitsFile ?) |
||
Background |
Pixel value |
VIS |
Calibrated Frame Product (BackgroundStorage) |
Pixel value |
NIR |
NIR Calibrated Frame (BackgroundStorage) |
|
Pixel value |
MER |
Background-subtracted Mosaic Product (BackgroundStorage) |
|
Pixel value |
EXT |
ExtCalibratedFrame (ext.detrendedFrame) |
|
Pixel value |
DES |
DesCalibratedFrame (BackgroundStorage) |
|
KIDS |
? |
||
LSST |
ExtLssCalexp (lssCalexpFitsFile) |
||
Airmass |
Value |
DES |
DesCalibratedFrame (desObservation) |
Value |
KIDS |
RawScienceFrame (AirmassStart/AirmassEnd) |
|
LSST |
? (lssVisitInfo) |
||
Atmospheric Extinction |
DES |
? |
|
KIDS |
photometricParameters (Extinction) |
||
LSST |
? |
||
SAA |
VIS |
Calibrated Frame Product (ReconsOrbit) |
|
NIR |
NIR Calibrated Frame (ReconstructedOrbit) |
||
Zodiacal Light |
? |
Warning
This needs to be checked.
Warning
I do not know if PSF for VIS/NIR is needed, and to what accuracy.
Note
What do we need to collect about meanPSF/seeing? only FWHM? mean ellipticity?
Note
Are we going to collect several statistics per environmental/instrumental info?
Warning
The closest information I saw about exposure time in the DM is total number of exposures for VIS/NIR
Warning
Not sure where to collect the information for EXT data: specific instrument frames, EXT stage 1 products, EXT stage 2 products, MER products?
Warning
I did not found any information about zodiacal light in the DM.
Again, with VMPZIDParamsInfoMaps in stage 1 MOC HEALPix masks are produced at the MER level (VMPZIDMOCCollectInfoMaps), while in stage 2 the data is gathered at the sky patch level as a either a partial HEALPix map VMPZIDHEALPixCollectInfoMaps or MOC map VMPZIDMOCCollectInfoMaps, or a mangle mask VMPZIDMangleCollectInfoMaps can be produced using parameters VMPZIDParamsMangle.
Note
I guess EXT data could be directly processed at the skypatch level.
Depth Maps¶
The depth map is then estimated using parameters VMPZIDParamsDepth, and information collected in VMPZIDMOCCollectInfoMaps. In stage 1 MOC HEALPix maps VMPZIDMOCDepth are produced at the MER level, while in stage 2 the data is gathered at the sky patch level in either partial HEALPix maps VMPZIDHEALPixDepth or VMPZIDMOCDepth and a mangle map VMPZIDMangleDepth can also be produced using parameters VMPZIDParamsMangle.
Photometric Visibility Mask and Random Catalog¶
Along with parameters VMPZIDParamsVisibility and the ref:VMPZIDMOCDepth <VMPZIDMOCDepth> maps, the photometric visibility mask can be produced and stored as partial HEALPix map VMPZIDHEALPixVisibility or MOC VMPZIDMOCVisibility. A random catalog VMPZIDRandomCatalog can then be produced using luminosity function representative of the bin/population selected, or a proxy where to draw samples (deep survey).
This photometric visibility mask can also be outputted as a mangle mask VMPZIDMangleVisibility, using parameters VMPZIDParamsMangle.
Note
Producing the random catalog in stage 1 seems difficult, since we would need to use a thinning of spatial Poisson point processes which implies either:
having access beforehand to the probability that a random detected object over the full sky belongs to a given MER tile (at least as difficult as knowing the full visibility mask) and know how many objects we want to simulate, so that we know how many detected objects we should have in a given tile (analytical approach, can be fully distributed).
simulate a large number of (indexed) random points on the footprint (much larger than the final number of wished random objects to account for non-detection), distribute them on the different MER tiles, and keep track whether they are detected or not, merging the lists ultimately to obtain the number of objects desired (Monte-Carlo approach, requiring synchronization and/or some inefficiency).
It seems more efficient to compute the visibility mask on tiles, and compute the random in stage 2.