|badge1| |badge2| |badge3| Introduction ============ VMPZ-ID products, used in other LE3 Processing Functions, consist in (with increasing complexity): * reconstructing accurate :ref:`survey footprint ` (using MER tiling in HEALPix MOC) * collecting :ref:`bit Masks ` derived at previous levels (VIS, NIR, EXT) and construction the coverage mask (footprint+not masked regions in bit masks) * collecting :ref:`instrumental/environmental information ` needed to compute the limiting magnitude * estimating the :ref:`depth maps ` (i.e. object independent) * estimating the :ref:`photometric visibility mask and the random catalog ` for a given selection used for 2PCF/PK-WL. VMPZ-ID is triggered at the MER tile level, where all products are first produced as (potentially weighted) MOC Healpix maps, at a given MOC order. In the following, this stage is named the stage 1. A second step, prior to triggering other "client" LE3 processing functions implies "merging" the MOC maps, and a HEALPix partial map or MOC map is produced at the resolution prescribed by the user in the parameter files. Potential mangle maps could be derived from this product. In the following, this stage is named the stage 2. .. note:: I am not sure there is a use case for a mangle mask, where here only the compression of information performed by mangle would be used (and not its arbitrary resolution property). Keeping mangle outputs essentially double then number of output products. Besides, it is not sure that the HEALPix partitioning of the sky would be respected sufficiently accurately by such post-processing to produce the random catalog. .. warning:: For parameter files, the use cases are important: right now the user specify which band/instrument he wants to process, implying that potentially not-used data is collected from the EAS which is not efficient. So: * do we want one pipeline definition per band/instrument and macro-pipelines for several bands (per instrument)? In this scenario, no need to specify which band we want (we select the pipeline). - Advantage: fine granularity allowing to reprocess single band/instrument - Problem: very large number of pipelines to manage, manual selection of pipeline. * do we want to collect all products and work with a subset of it, specificied in the list of parameters? - Advantage: one (or few) pipelines to manage - Problem: identify the needed products, not efficient if only a few bands need processing * do we want to have a lot of optional inputs in the pipeline def, and only process those that have been selected? - Advantage: one (or few) pipelines to manage - Problem: not very automatic, "manual" selection of products I would rather go for option1, with macro-pipelines per instrument. .. warning:: We have to define the granularity of outputs: single band of an instrument in a file, or all bands? Here we have chosen a single band per file. This is the most flexible solution in terms of processing (e.g. single band reprocessing, different processing for EXT and Euclid) and use cases (we can get the info for any band of any instrument directly from archive), but implies that if a client PF wants to collect all infos it will need to know exactly which ones it needs to use, if all relevant products have been processed, and it will have a variable number of input products over the MER Tiles that it needs to collect and process accordingly. Maybe we need to add a flag indicating in the output which bands are relevant for a given query, and which were processed (not yet implemented)? If we want to let's say gather all products in a single one at a Skypatch level (if no use case for individual products), to flag a consistent and complete product, then another pipeline collecting the individual HEALPix objects and outputting a single HEALPix product could be run. The easiest way is to create a data product that is a collection of such individual products (purely metadata). We can also have here a single self-contained output with all individual products in a FITS file at the skypatch level: - the current HEALPy implementation implies that multiple maps (corresponding to different bands/instruments) are stored as columns. In the XSD data model, the columns need to be fixed, with a name. This implies that we cannot have a variable number of columns for a product. Consequently, using baseline HEALPy all possible bands need to appear as columns whatever the sky area covered (not very efficient) - the alternative is to have one hdu per band with specific keywords, as the multiplicity of hdus is not fixed in the DM. In this scenario the write_map routine from HEALPy needs to be adapted (from what I see, should not be a huge effort). The advantage of this solution is that we keep the flexibility of columns for other information per band (e.g. the bit masks, one column per bit) Several options are possible here if we want a self-contained product: Finally the concept of "release" introduced recently could "solve" the downstream problem of knowing which consistent products needs be used, if a same release number could be applied to various products at different levels, as we suffer from multiple inputs/outputs potentially (re-)processed at different times by potentially different PFs and we want consistent groups of products. It is however not clear to me how/when this release number is applied to the products. .. warning:: It is important to know whether the product are produced on a VIS/NIR/EXT frame or a MER tile or on observation. Here, I assumed MER tile as we discussed. .. note:: Output products contain an identifier of the processed sky area, either TILEINDEX as MER does or an identifier defined by MER corresponding to a list of tiles. .. _intraFootprint: Survey footprint ++++++++++++++++ For the survey footprint, beside parameters :ref:`VMPZIDParamsFootprint ` the footprint requires: * for stage 1 (processing on MER tiles), products containing the list of coordinates describing the geometry of the survey for the set of given band/instrument are first ingested (e.g.: the ReconsFPAPointing in VIS :ref:`VISObsFrame` for NIR/VIS, the ImgSpatialFootprint in EXT `CalibratedFrame `_ if precise enough). The fraction of HEALPix pixel covered is reported (see below the note) in a :ref:`VMPZIDMOCFootprintMask `. * for stage 2 (processing at skypatch level), a set of MOC HEALPix Maps :ref:`VMPZIDMOCFootprintMask ` are processed on the different MER tiles overlapping with the prescribed region. The two stages will produce a :ref:`VMPZIDHEALPixFootprintMask ` or :ref:`VMPZIDMOCFootprintMask `. In case of sky patch processing, a mangle file :ref:`VMPZIDMangleFootprintMask ` can also be outputed, using :ref:`VMPZIDParamsMangle ` parameter files. .. note:: The convention for reporting pixel coverage in MER is 1 for fully non observed region and 0 to fully observed regions. For mangle this is the opposite: fully observed regions are set to 0 and fully not observed to 1. This needs to be consistently propagated. Here in the following we assume MER convention: 1=fully not observed. .. _intraBitMasks: Bit Masks and coverage mask +++++++++++++++++++++++++++ The coverage masks correspond to regions of the survey footprint that have not been flagged using the bitmask. It is therefore the same format as the survey footprint, but the flagged areas are set to 1. Parameters are set in :ref:`VMPZIDParamsCoverage `. The two stages will produce the product :ref:`VMPZIDHEALPixCoverageMask ` or :ref:`VMPZIDMOCCoverageMask `. In case of sky patch processing, or a mangle file :ref:`VMPZIDMangleCoverageMask ` can also be outputted, using :ref:`VMPZIDParamsMangle ` parameter files. For the bit masks, besides parameters :ref:`VMPZIDParamsBitMask `, and a HEALPix :ref:`VMPZIDMOCFootprintMask `, one would need: * for stage 1, the pixel flag maps defined in MER :ref:`MERMosaic` or (see in `human readable format `_) for the different bands/instruments, to produce :ref:`VMPZIDMOCBitMask ` * for stage 2, a set of :ref:`VMPZIDMOCBitMask ` computed on the different MER tiles overlapping with the prescribed region to produce :ref:`VMPZIDHEALPixBitMask ` or :ref:`VMPZIDMOCBitMask ` In case of sky patch processing, a mangle file :ref:`VMPZIDMangleCoverageMask ` can also be outputed, using :ref:`VMPZIDParamsMangle ` parameter files. .. note:: In the MER :ref:`MERMosaic` product (including flags), all bands are rebinned in a grid according to the VIS pixel grid. Is this the product we need, or do we need to process also the :ref:`VISCalibratedFrame`? .. _intraInfo: Instrumental/Environmental Information ++++++++++++++++++++++++++++++++++++++ Using parameters :ref:`VMPZIDParamsInfoMaps `, MOC maps of instrumental and environmental effects are first collected in a product :ref:`VMPZIDMOCCollectInfoMaps `. This implies to have access to all relevant statistics (mean/median/max/min?) on relevant quantities to compute the limiting magnitude, such as PSF/seeing FWHM, background level, RMS map, exposureTime, airmass,solar aspect angle, zodiacal light emissions. In the table below are listed a series of potential inputs to collect this maps. +---------------+-------------+------------+----------------------------------------------------+ | Instrumental/ | | | | | Environmental | Statistics | Instrument | Input Product | | Parameters | | | | +===============+=============+============+====================================================+ | Noise | RMS (pixel) | VIS | :ref:`VISCalibratedFrame` (WeightStorage) | | +-------------+------------+----------------------------------------------------+ | | RMS (pixel) | NIR | :ref:`NIRCalibratedFrame` (reducedFrameFitsFile) | | +-------------+------------+----------------------------------------------------+ | | RMS (pixel) | MER | :ref:`MERMosaic`/:ref:`MERBksMosaic` (RmsStorage) | +---------------+-------------+------------+----------------------------------------------------+ | Weights | RMS (pixel) | EXT | ExtCalibratedFrame (DataStorage) | | +-------------+------------+----------------------------------------------------+ | | Value | DES | DesCalibratedFrame (desObservation) | | +-------------+------------+----------------------------------------------------+ | | Value | KIDS | reducedScienceFrame (Weight)? | | +-------------+------------+----------------------------------------------------+ | | RMS (pixel) | LSST | ExtLssCalexp (lssCalexpFitsFile) | +---------------+-------------+------------+----------------------------------------------------+ | Exposures | total number| VIS | :ref:`VISCalibratedFrame` (ObservationSequence) | | +-------------+------------+----------------------------------------------------+ | | total number| NIR | :ref:`NIRCalibratedFrame` (ObservationSequence) | | +-------------+------------+----------------------------------------------------+ | | time | MER | :ref:`MERMosaic`/:ref:`MERBksMosaic` (imgBaseFrame)| | +-------------+------------+----------------------------------------------------+ | | time | EXT | ExtCalibratedFrame (imgBaseFrame) | | +-------------+------------+----------------------------------------------------+ | | time | DES | DesCalibratedFrame (imgBaseFrame) | | +-------------+------------+----------------------------------------------------+ | | total time | KIDS | reducedScienceFrame (ExpTime single exposure) | | +-------------+------------+----------------------------------------------------+ | | time | LSST | ExtLssCalexp (imgBaseFrame) | +---------------+-------------+------------+----------------------------------------------------+ | PSF | mean FWHM | VIS | Needed? over FOV? For a stack/exposure? Wavelength?| | + +------------+----------------------------------------------------+ | | | NIR | Needed? over FOV? For a stack/exposure? Wavelength?| | +-------------+------------+----------------------------------------------------+ | | Center value| MER | :ref:`MERMosaic` or :ref:`MERBksMosaic` | | | | | (merPsfModelStorage) | | +-------------+------------+----------------------------------------------------+ | | FWHM | EXT | ExtPSF (extPsfModel, keyword FITS) | +---------------+-------------+------------+----------------------------------------------------+ | Seeing | median | DES | DesCalibratedFrame (PsfModelStorage, FHWM) | | +-------------+------------+----------------------------------------------------+ | | median | KIDS | reducedScienceFrame (Seeing) | | +-------------+------------+----------------------------------------------------+ | | | LSST | ExtLssCalexp (lssPsfModelFitsFile ?) | +---------------+-------------+------------+----------------------------------------------------+ | Background | Pixel value | VIS | :ref:`VISCalibratedFrame` (BackgroundStorage) | | +-------------+------------+----------------------------------------------------+ | | Pixel value | NIR | :ref:`NIRCalibratedFrame` (BackgroundStorage) | | +-------------+------------+----------------------------------------------------+ | | Pixel value | MER | :ref:`MERBksMosaic` (BackgroundStorage) | | +-------------+------------+----------------------------------------------------+ | | Pixel value | EXT | ExtCalibratedFrame (ext.detrendedFrame) | | +-------------+------------+----------------------------------------------------+ | | Pixel value | DES | DesCalibratedFrame (BackgroundStorage) | | +-------------+------------+----------------------------------------------------+ | | | KIDS | ? | | +-------------+------------+----------------------------------------------------+ | | | LSST | ExtLssCalexp (lssCalexpFitsFile) | +---------------+-------------+------------+----------------------------------------------------+ | Airmass | Value | DES | DesCalibratedFrame (desObservation) | | +-------------+------------+----------------------------------------------------+ | | Value | KIDS | RawScienceFrame (AirmassStart/AirmassEnd) | | +-------------+------------+----------------------------------------------------+ | | | LSST | ? (lssVisitInfo) | +---------------+-------------+------------+----------------------------------------------------+ | Atmospheric | | DES | ? | | Extinction +-------------+------------+----------------------------------------------------+ | | | KIDS | photometricParameters (Extinction) | | +-------------+------------+----------------------------------------------------+ | | | LSST | ? | +---------------+-------------+------------+----------------------------------------------------+ | SAA | | VIS | :ref:`VISCalibratedFrame` (ReconsOrbit) | | +-------------+------------+----------------------------------------------------+ | | | NIR | :ref:`NIRCalibratedFrame` (ReconstructedOrbit) | +---------------+-------------+------------+----------------------------------------------------+ | Zodiacal | | | ? | | Light | | | | +---------------+-------------+------------+----------------------------------------------------+ .. warning:: This needs to be checked. .. warning:: I do not know if PSF for VIS/NIR is needed, and to what accuracy. .. note:: What do we need to collect about meanPSF/seeing? only FWHM? mean ellipticity? .. note:: Are we going to collect several statistics per environmental/instrumental info? .. warning:: The closest information I saw about exposure time in the DM is total number of exposures for VIS/NIR .. warning:: Not sure where to collect the information for EXT data: specific instrument frames, EXT stage 1 products, EXT stage 2 products, MER products? .. warning:: I did not found any information about zodiacal light in the DM. Again, with :ref:`VMPZIDParamsInfoMaps ` in stage 1 MOC HEALPix masks are produced at the MER level (:ref:`VMPZIDMOCCollectInfoMaps `), while in stage 2 the data is gathered at the sky patch level as a either a partial HEALPix map :ref:`VMPZIDHEALPixCollectInfoMaps ` or MOC map :ref:`VMPZIDMOCCollectInfoMaps `, or a mangle mask :ref:`VMPZIDMangleCollectInfoMaps ` can be produced using parameters :ref:`VMPZIDParamsMangle `. .. note:: I guess EXT data could be directly processed at the skypatch level. .. _intraDepth: Depth Maps ++++++++++ The depth map is then estimated using parameters :ref:`VMPZIDParamsDepth `, and information collected in :ref:`VMPZIDMOCCollectInfoMaps `. In stage 1 MOC HEALPix maps :ref:`VMPZIDMOCDepth ` are produced at the MER level, while in stage 2 the data is gathered at the sky patch level in either partial HEALPix maps :ref:`VMPZIDHEALPixDepth ` or :ref:`VMPZIDMOCDepth ` and a mangle map :ref:`VMPZIDMangleDepth ` can also be produced using parameters :ref:`VMPZIDParamsMangle `. .. _intraVisibility: Photometric Visibility Mask and Random Catalog ++++++++++++++++++++++++++++++++++++++++++++++ Along with parameters :ref:`VMPZIDParamsVisibility ` and the ref:`VMPZIDMOCDepth ` maps, the photometric visibility mask can be produced and stored as partial HEALPix map :ref:`VMPZIDHEALPixVisibility ` or MOC :ref:`VMPZIDMOCVisibility `. A random catalog :ref:`VMPZIDRandomCatalog ` can then be produced using luminosity function representative of the bin/population selected, or a proxy where to draw samples (deep survey). This photometric visibility mask can also be outputted as a mangle mask :ref:`VMPZIDMangleVisibility `, using parameters :ref:`VMPZIDParamsMangle `. .. note:: Producing the random catalog in stage 1 seems difficult, since we would need to use a thinning of spatial Poisson point processes which implies either: 1) having access beforehand to the probability that a random **detected** object over the full sky belongs to a given MER tile (at least as difficult as knowing the full visibility mask) and know how many objects we want to simulate, so that we know how many detected objects we should have in a given tile (analytical approach, can be fully distributed). 2) simulate a large number of (indexed) random points on the footprint (much larger than the final number of wished random objects to account for non-detection), distribute them on the different MER tiles, and keep track whether they are detected or not, merging the lists ultimately to obtain the number of objects desired (Monte-Carlo approach, requiring synchronization and/or some inefficiency). It seems more efficient to compute the visibility mask on tiles, and compute the random in stage 2. .. |badge1| image:: https://img.shields.io/badge/EDEN-2.1-yellow .. |badge2| image:: https://img.shields.io/badge/DM-8.0.5-red .. |badge3| image:: https://img.shields.io/badge/ML-ML2B-yellow