.. _NIRCalibratedFrame: NIR Calibrated Frame ==================== $SetSchemaPath nir/euc-nir-CalibratedFrame.xsd Data product name _________________ .. DataProductNameStart $PrintDataProductName .. DataProductNameEnd Data product custodian ______________________ .. DataProductCustodianStart $PrintDataProductCustodian .. DataProductCustodianEnd Name of the Schema File ________________________ .. NameSchemaStart $PrintSchemaFilename .. NameSchemaEnd Last Edited for DPDD Version ____________________________ .. DpddVersionlTagStart 1.1 .. DpddVersionTagEnd Processing Elements Creating / Updating / Using the Product ___________________________________________________________ .. PECreatorStart **Creators:** * The full :ref:`NIRPipeline` **Consumers:** * NIR_Resampling .. PECreatorEnd Processing Function using the products ______________________________________ .. PFUsingStart .. Indicate here the PF you are aware of using the product. Can be the custodian PF itself for internal product. Can also be Science Working Group for final products. The list does not need to be exhaustive. * PF-NIR, PF-MER, PF-SIR .. PFUsingEnd Proposed for inclusion in EAS/SAS _________________________________ .. IncludeInSASStart This product is proposed for inclusion in the SAS: **yes** This is one of the main PF-NIR products and represents a single :ref:`LE1NispRawFrame` image fully calibrated. .. IncludeInSASEnd Data Product Elements _____________________ .. DataProductElementsStart $PrintDataProductElements .. DataProductElementsEnd Detailed Description of the Data Product ________________________________________ .. DetailedDescStart The **DpdNirCalibratedFrame** is the OU-NIR Data Product produced by the execution of PF-NIR scientific pipeline on a :ref:`LE1NispRawFrame` image. Pipeline tasks correct and flag the image for instrumental signatures and provides a fully astrometric and photometric calibrated image. This Data Product is used by the :term:`Processing Function` MER as input for its pipeline. .. _NIR_calibrated_frame_: .. figure:: ../figures/nirCalibratedFrame.png :scale: 100 % :align: center :figclass: align-center : Graphical representaton of ``Data`` section structure for **DpdNirCalibratedFrame** The **DpdNirCalibratedFrame** ``Data`` section, reported on image :numref:`NIR_calibrated_frame_`, is structured in classes with increasing specificity on the data product. * The most generic ``imgBaseFrame`` collects all metadata relative to the image structure itself as the ``Imgtype`` and data size. * Metadata contained in ``img:singleExposure`` report the information on Filters, Spatial Footprint and the Euclid bit Mask adopted. ``DetectorList`` hosts the characterization of each detector, as the detector Id, the WCS parameters and the photometrci zero point. * Metadata in the ``nir:nirCalibratedFrame`` structure contain: * the ``Observation Sequence``, that are numbers needed to identify the specific pointing, the date, the ``ReadoutMode`` adopted and the ``TargetPointing`` * the name of the 4 files in the ``Storage`` metadata (see below) * a section dedicated to the ``QualityParams``, a set of statistical values and validation parameters useful to evaluate the data quality and identify anomalies The **DpdNirCalibratedFrame** Data Product references to 4 files: 1. **Scientific Image** in ``Data.DataStorage``: a multi-extension FITS (MEF) file, with FitsFormat id = "nir.calibratedScienceFrame", containing three extensions per detector representing calibrated science image, RMS, and Data Quality flags. Hence the final MEF file has 49 layers: a primary header + 3 extensions for each of the 16 detectors. 2. **Background model** in ``Data.BackgroundStorage``: a MEF file with the same structure as the Scientific Image, containing the estimated background. 3. a **PSF Model** in ``Data.PsfStorage.PsfModelFile``: a Point Spread Function (PSF) model as provided by PSFEx software (.psf). 4. a **PSF Image** in ``Data.PsfStorage.PsfFITSFile``: a single extension FITS file containing the PSF image associated to the science image. It can be extended to include wavelength and focal plane dependency. Each file is provided toghether with a dedicated Quality Parameter section; therefore the ``Data.QualityParams`` is structured in 3 sections: ``ScienceQualityParams``, ``BackgroundQualityParams`` and ``PsfQualityParams`` for science, background and PSF respectively. Scientific image FITS file ^^^^^^^^^^^^^^^^^^^^^^^^^^ The main FITS file associated to this Data Product, linked on ``Data.DataStorage``, contains the scientific image derived from the application of PF-NIR pipeline :ref:`NIRPipeline` on the :ref:`LE1NispRawFrame` data product. The FITS file structure, therefore, is an evolution of the :ref:`LE1NispRawFrame` one. It has a primary header with main informations on poitings and 3 layer for each detector, 49 layers in total. The layer name, in *EXTNAME* keyword, follow the convention ``DETxy.SCI``, ``DETxy.RMS``, and ``DETxy.DQ``, for Science, RMS, and Data Quality layers respectively. ``xy`` are the detector coordinated on the NISP Detector System Assembly as illustrated on :numref:`NISP_detector_position_ids`image. ============= ================= =============== ======================== Layer name Data Format Units Content ============= ================= =============== ======================== **PRIMARY** Main headers **DETxy.SCI** *float32* electron Scientific data **DETxy.RMS** *float32* electron RMS data **DETxy.DQ** *int32* Data Quality mask ============= ================= =============== ======================== .. _calframe: .. figure:: ../figures/calframe.png : Graphical representation of NIR calibrated frames 3-layer per detector structure Note that background is not subtracted from science images, and pixel values are still in native units (electrons). Relative and absolute calibration factors for each detector, for conversion to microJy, are included in the metadata, as well as zero-points for conversion to AB magnitudes. Reference pixels, the border of 4 not exposed pixels around the image, are removed from all detector, which size is therefore 2040x2040 pixels. Primary layer """"""""""""" The ``PRIMARY`` header keywords report main observational pointing and instrumental information and other parameters relative to the NIR pipeline execution. $PrintPrimaryHeader fit/euc-nir.xml nir.calibratedScienceFrame Scientific layer """""""""""""""" The ``DETxy.SCI`` layer contains the input image corrected from instrumental effects and converted in electrons. The astrometrical and photometrical correction factors are reported on detector header. $PrintExtensionHeader fit/euc-nir.xml nir.calibratedScienceFrame DETn.SCI Standard Deviation layer """""""""""""""""""""""" The pixel value stored into the ``DETxy.RMS`` data layer is derived from a variance layer intenally used by NIR Pipeline tasks. It is initially instantiated by the `NIR Init `_ task application according the formula 12 of `Kubik (2016) `_ paper. The final variance layer value is the result of the calibration file contributions added during the application of :ref:`NIRPipeline` tasks. Data Quality layer """""""""""""""""" The ``DETxy.DQ`` layer is a bit maks containing flags associated to each pixel during the pipeline execution. The first bit `INVALID` identifies pixels that are considered not to be used. It is activated by any pipeline task that identify a critical feature (i.e. bad pixels, cosmic hits and saturaded pixels). The single bitmask flag description is available in dedicated `wiki `_ and shown in :numref:`NIR_bitmask_table` .. _NIR_bitmask_table: .. figure:: ../figures/NIR_DQ_bitmask_table.png :scale: 40 % :align: center :figclass: align-center : NIR Data Quality Bit value, flag name and description. Background FITS file ^^^^^^^^^^^^^^^^^^^^ The Background FITS file contains the background value estimated from images by the `NIR Background estimation `_ task. The FITS file structure mirrors the scientific file one with 49 layers: a primary header with main information and 3 layers (SCI, RMS and DQ) for each detector. PSF model and image files ^^^^^^^^^^^^^^^^^^^^^^^^^ The `NIR PSF estimation `_ task estimates a PSF model from scientific images thanks a `PSFEx `_ run. The software produces a ``.psf`` file containing model parameters and a ``.fits`` with an image of the PSF model. This last product is used by PF-MER. Data Quality Parametes ^^^^^^^^^^^^^^^^^^^^^^ The ``Data.QualityParams`` section provide a set of Quality Parameters derived from each of the files. The ``Data.QualityParams.ScienceQualityParams`` contains a set of statistical values (*min, max, mean, median std* and *masked pixel fraction*) for the whole image and some Quality Parameters correlated to the astrometric and photometric calibration. For each detector are indeed provided the same set of statistics and the numbers of pixel flagged for each bit mask value. The ``Data.QualityParams.BackgroundQualityParams`` contains the set of statistical values for each detector. Despite the background image pixel value is in ``[electron]``, the background detector statistical values are converted in ``[Myr/sr]``. .. The DQ flags follow a common dictionary described `here `_. .. **Extension headers** $PrintExtensionHeader **HDU Table** $PrintHDUTableContents .. DetailedDescEnd