NIR Calibrated Frame

Data product name

DpdNirCalibratedFrame

Data product custodian

NIR

Name of the Schema File

euc-nir-CalibratedFrame.xsd

Last Edited for DPDD Version

1.1

Processing Elements Creating / Updating / Using the Product

Creators:

• The full NIR Pipeline

Consumers:

• NIR_Resampling

Processing Function using the products

• PF-NIR, PF-MER, PF-SIR

Proposed for inclusion in EAS/SAS

This product is proposed for inclusion in the SAS: yes

This is one of the main PF-NIR products and represents a single NISP Raw Frame Product image fully calibrated.

Data Product Elements

Header:

object of type sys:genericHeader

Data:

object of type nir:nirCalibratedFrame

QualityFlags:

object of type dqc:sqfPlaceHolder

Parameters:

object of type ppr:genericKeyValueParameters

Detailed Description of the Data Product

The DpdNirCalibratedFrame is the OU-NIR Data Product produced by the execution of PF-NIR scientific pipeline on a NISP Raw Frame Product image. Pipeline tasks correct and flag the image for instrumental signatures and provides a fully astrometric and photometric calibrated image.

This Data Product is used by the Processing Function MER as input for its pipeline.

As most of the other Euclid Dpds, the DpdNirCalibratedFrame Data section is designed extending base classes with increasing specificity on the data product. Because of the DpdNirCalibratedFrame represents an image, the base class, on top of the data product (see image Fig. 15), is the generic class imgBaseFrame.

Fig. 15 : Graphical representation of first section of Data section: the imgBaseFrame.

This class collects all metadata relative to the image parameters itself.

• ImgType contains metadata describing the type of Image (i.e. photometric or spectroscopic, calibration).

• AxisNumber and AxisLengths describe the FITS file structure axis

• DataSize and DataLength refers to the size of file itself.

Fig. 16 : Graphical representation of second section of Data section: the img:singleExposure.

The higher class structure img:singleExposure (see fig Fig. 16) stores metadata needed for the description of an Euclid single exposure image.

• Instrument is the name of Euclid instrument used for the exposure.

• Filter hosts the name and some basic information of filter used.

• ImgSpatialFootprint contains the polygon coordinates of image sky coverage.

• Masks is the name of bit mask adopted.

• DetectorList contains a list of Detector, a structure dedicated to each detector metadata as:

  • DetectorId: the single detector coordinates on grid (i.e. 11,`12`, … ,`34`, 44) used as unique identifier

  • WCS: single detector WCS

  • ZeroPoint: the photometric Zeropoint value in AB mag

Fig. 17 : Graphical representation of third section of Data section: the nir:nirCalibratedFrame.

Metadata dedicated exclusively to the DpdNirCalibratedFrame are contained in the nir:nirCalibratedFrame structure (see fig Fig. 17).

• ObservationDateTime is the date and the time of the observation according the on-board clock, on UTC time and in julian date

• ObservationSequence is a set of metadata needed to identify the single observation

• ReadoutMode

• TargetPointing

Metadata in the nir:nirCalibratedFrame structure (see fig Fig. 17) contain:

• the Observation Sequence, that are numbers needed to identify the specific pointing, the date, the ReadoutMode adopted and the TargetPointing

• the name of the 4 files in the Storage metadata (see below)

• a section dedicated to the QualityParams, a set of statistical values and validation parameters useful to evaluate the data quality and identify anomalies

The DpdNirCalibratedFrame Data Product contains references to some files:

1. Data.DataStorage contains the Scientific Image: a multi-extension FITS (MEF) file, with FitsFormat id = “nir.calibratedScienceFrame”, containing three extensions per detector representing calibrated science image (SCI), RMS, and Data Quality flags (DQ). Hence the final MEF file has 49 layers: a primary header + 3 extensions for each of the 16 detectors.

2. Data.BackgroundStorage contains the Background model: a MEF file with the same structure as the Scientific Image, containing the estimated background.

3. Data.PsfStorage contains the PSF measured combining all the pipeline input images. The PSF is stored in a double format:

   1. Data.PsfStorage.PsfModelFile stores the PSF Model: a Point Spread Function (PSF) model as provided by PSFEx software (.psf).

   2. Data.PsfStorage.PsfFITSFile stores a PSF Image that is a single extension FITS file containing the PSF image associated to the science image. It can include wavelength and focal plane dependency.

4. Data.PsfSingleStorage is an optional metadata to store the PSF estimated from each single dither. It has the same structure of Data.PsfStorage.

Each file is provided together with a dedicated Quality Parameter section; therefore the Data.QualityParams is structured in 4 sections: ScienceQualityParams, BackgroundQualityParams, PsfQualityParams and PsfSingleQualityParams for science, background and PSF (global and single) respectively.

Scientific image FITS file

The main FITS file associated to this Data Product, linked on Data.DataStorage, contains the scientific image derived from the application of PF-NIR pipeline NIR Pipeline on the NISP Raw Frame Product data product.

The FITS file structure, therefore, is an evolution of the NISP Raw Frame Product one. It has a primary header with main informations on poitings and 3 layer for each detector, 49 layers in total. The layer name, in EXTNAME keyword, follow the convention DETxy.SCI, DETxy.RMS, and DETxy.DQ, for Science, RMS, and Data Quality layers respectively. xy are the detector coordinated on the NISP Detector System Assembly as illustrated on :numref:`NISP_detector_position_ids`image.

Layer name	Data Format	Units	Content
PRIMARY			Main headers
DETxy.SCI	float32	electron	Scientific data
DETxy.RMS	float32	electron	RMS data
DETxy.DQ	int32		Data Quality mask

Fig. 18 : Graphical representation of NIR calibrated frames 3-layer per detector structure

Note that background is not subtracted from science images, and pixel values are still in native units (electrons). Relative and absolute calibration factors for each detector, for conversion to microJy, are included in the metadata, as well as zero-points for conversion to AB magnitudes.

Reference pixels, the border of 4 not exposed pixels around the image, are removed from all detector, which size is therefore 2040x2040 pixels.

Primary layer

The PRIMARY header keywords report main observational pointing and instrumental information and other parameters relative to the NIR pipeline execution.

HDU PRIMARY: header
	Name	Description	Value
Keywords	FITS_DEF	FITS name definition	string = nir.calibratedScienceFrame
	FITS_VER	FITS version	string = 0.3
	TELESCOP		string = Euclid
	INSTRUME	or NISPsim for simulations	string = NISP
	VERSION	Data Release version	string
	DATE	UT date when this file was created	string
	ORIGIN	FITS file originator	string
	SOFTVERS	Version of the processing function	string
	IMG_CAT	ImgType:Category DP category (SCIENCE, CALIB, …)	string
	IMG_T1	ImgType:FirstType DP type (OBJ, SRD, DARK, …)	string
	IMG_T2	ImgType:SecondType DP type (LAMP, SKY, …)	string
	OBSMODE	ImgType:ThirdType DP type (WIDE, DEEP, CALIBRATION)	string
	OBSTYPE	ImgType:Technique DP technique (IMAGE, SPECTROIMAGE)	string
	DATE-OBS	img:spaceObservationDateTime:OBT Date of observation (start), e.g. 2014-03-15T09:30:09.313	string
	UTC-OBS	img:spaceObservationDateTime:UTC UTC time correlated to the on-board time by the Mission Operation Centre, e.g. 2014-03-15T09:30:09.313Z	string
	MJD-OBS	img:spaceObservationDateTime:MJD MJD start time, e.g. 56731.39594113	double
	READMODE	ReadoutMode:ReadoutModeMethod Readout Mode Method: (Multiaccum, UpTheRamp, FowlerSamples)	string
	NR	ReadoutMode:MACCValues:NRead Number of reads	integer
	NG	ReadoutMode:MACCValues:NGroup Number of Group	integer
	ND	ReadoutMode:MACCValues:NDrop Number of drops	integer
	FRTIME	ReadoutMode:FrameTime Exposure time in seconds for single readout sample. Default value =1.41 sec	double
	EXPTIME	ExposureTime Effective integration time in seconds, e.g. 60.0005	double
	ELAPTIME	Elapsed time of the observation in seconds	double
	RA	TargetPointing:RA Center of field of view right ascension (deg)	double
	DEC	TargetPointing:DEC Center of field of view right declination (deg)	double
	PA	TargetPointing:Orientation Position angle (deg)	double
	EQUINOX	Standard FK5 (years), e.g. 2000.	double
	RADESYS	Coordinate reference frame, e.g. FK5	string
	OBS_ID	eso:observationSequence:ObservationId Running Sequence number of the Observed block	integer
	DITHOBS	eso:observationSequence:DitherObservation Number sequence of dithering i.e. 1-4 for WIDE survey	integer
	PTGID	eso:observationSequence:PointingID Integer number defining exposure inside observation	integer
	EXPNUM	eso:observationSequence:Exposure Internal integer counter for single images	integer
	TOTEXP	eso:observationSequence:TotalExposure Total number of Exposures in specific pointing	integer
	FILTER	Filter name, e.g. NIR_J	string
	FWA_POS	FWA position (J,H,Y,OPEN,CLOSED)	string
	GWA_POS	GWA position (RGS270,RGS000,RGS180,BGS000,OPEN)	string
	FWA_ANG	Commanded Filter Wheel Angle in the range 0,360	integer
	FWA_REF	Filter Wheel Angle Reference position to compute angle	string
	GWA_ANG	Commanded Grism Angle in the range 0,360	integer
	GWA_REF	Grism Wheel Angle Reference position to compute angle	string
	GWA_TILT	Commanded Grism Tilt Angle	integer
	PHRELOB	Correction factor for relative photometric calibration for observation	double
	PHRELOBE	Uncertainty in PHRELOB	double
	PHRELEX	Correction factor for relative photometric calibration	double
	PHRELEXE	Uncertainty in PHRELEX	double
	CALBPIX	Bad Pixel Mask calibration file (productId and name)	string
	CALMDARK	Master Dark calibration file (productId and name)	string
	CALMFLAT	Master Flat (Small scale) calibration file (productId and name)	string
	CALLFLAT	Large Scale Flat calibration file (productId and name)	string
	CALSAT	Saturation calibration file (productId and name)	string
	CALNL	Non-linearity coefficients file	string
	CRTHRES	Threshold value for Cosmic Ray Rejection	double
	CALCATNA	Name of Input Catalog	string
	CALCATOR	Origin of Input catalog e.g. SIM, VIS, GAIA, …	string

Scientific layer

The DETxy.SCI layer contains the input image corrected from instrumental effects and converted in electrons. The astrometrical and photometrical correction factors are reported on detector header.

HDU DETn.SCI: header
	Name	Description	Value
Keywords	EXTNAME	e.g. DET11.SCI, meaning Detector number 0, Science data	string
	DET_ID	Detector:DetectorId e.g. Detector ID, e.g. 11	string
	BITPIX	array data type	integer
	NAXIS	e.g. number of array dimensions	integer
	NAXIS1	e.g. pixel numbers on first axis	integer
	NAXIS2	e.g. pixel numbers on second axis	integer
	BUNIT	physical units of the array values	string
	RPIX_PRC	Reference Pixel processing mode (0=no_px_correction,1=horiz_px_correction,2=vertical_px_correction)	integer
	GAIN	Detector:Gain e- to ADU conversion factor, e.g. 1.5	double
	EQUINOX	Standard FK5 (years), e.g. 2000.	double
	RADESYS	Coordinate reference frame, e.g. FK5	string
	CRVAL1	Right ascension at ref pixel	double
	CRVAL2	Declination at ref pixel	double
	CRPIX1	Reference pixel x coordinate	double
	CRPIX2	Reference pixel y coordinate	double
	CTYPE1	Coordinate 1 type	double
	CTYPE2	Coordinate 2 type	double
	CUNIT1	Axis 1 unit	string
	CUNIT2	Axis 2 unit	string
	CD1_1	Translation matrix element	double
	CD1_2	Translation matrix element	double
	CD2_1	Translation matrix element	double
	CD2_2	Translation matrix element	double
	PV1_0	Projection distortion parameter	double
	PV1_1	Projection distortion parameter	double
	PV1_2	Projection distortion parameter	double
	PV1_4	Projection distortion parameter	double
	PV1_5	Projection distortion parameter	double
	PV1_6	Projection distortion parameter	double
	PV1_7	Projection distortion parameter	double
	PV1_8	Projection distortion parameter	double
	PV1_9	Projection distortion parameter	double
	PV1_10	Projection distortion parameter	double
	PV2_0	Projection distortion parameter	double
	PV2_1	Projection distortion parameter	double
	PV2_2	Projection distortion parameter	double
	PV2_4	Projection distortion parameter	double
	PV2_5	Projection distortion parameter	double
	PV2_6	Projection distortion parameter	double
	PV2_7	Projection distortion parameter	double
	PV2_8	Projection distortion parameter	double
	PV2_9	Projection distortion parameter	double
	PV2_10	Projection distortion parameter	double
	ASTIRMS1	Astrom. dispersion RMS (intern., high S/N)	double
	ASTIRMS2	Astrom. dispersion RMS (intern., high S/N)	double
	ASTRRMS1	Astrom. dispersion RMS (ref., high S/N)	double
	ASTRRMS2	Astrom. dispersion RMS (ref., high S/N)	double
	ZPAB	zero point for conversion to AB magnitude	double
	ZPABE	error in ZPAB determination	double
	ZPVEGA	zero point for conversion to Vega magnitude	double
	ZPVEGAE	error in ZPVEGA determination	double
	PHRELDT	Correction factor for relative photometric calibration for detector	double
	PHRELDTE	Uncertainty in PHRELDT	double
	CRSNGALG	Algorithm used in finding cosmic rays	string
	CRMULALG	Algorithm used in finding cosmic rays	string
	DARKFILL	Dark fill value in e-	double
	NBADPIXT	Number of bad pixels for detector	integer
	NREJNL	Number of high noise pixels not corrected for non-linearity	integer
	NSATPIX	Number of pixels flagged for saturation	integer
	NDFILL	Number of high noise pixels with filled dark values	integer
	NCRPIXS	Number of pixels masked as CR pixels in the single frame rejection	integer
	NCRPIXM	Number of pixels masked as CR pixels in the multi frame rejection	integer

Standard Deviation layer

The pixel value stored into the DETxy.RMS data layer is derived from a variance layer intenally used by NIR Pipeline tasks. It is initially instantiated by the NIR Init task application according the formula 12 of Kubik (2016) paper.

The final variance layer value is the result of the calibration file contributions added during the application of NIR Pipeline tasks.

Data Quality layer

The DETxy.DQ layer is a bit maks containing flags associated to each pixel during the pipeline execution.

The first bit INVALID identifies pixels that are considered not to be used. It is activated by any pipeline task that identify a critical feature (i.e. bad pixels, cosmic hits and saturaded pixels).

The single bitmask flag description is available in dedicated wiki and shown in Fig. 19

Fig. 19 : NIR Data Quality Bit value, flag name and description.

Background FITS file

The Background FITS file contains the background value estimated from images by the NIR Background estimation task.

The FITS file structure mirrors the scientific file one with 49 layers: a primary header with main information and 3 layers (SCI, RMS and DQ) for each detector.

PSF model and image files

The NIR PSF estimation task estimates a PSF model from scientific images thanks a PSFEx run.

The software produces a .psf file containing model parameters and a .fits with an image of the PSF model. This last product is used by PF-MER.

Data Quality Parameters

The Data.QualityParams section provide a set of Quality Parameters derived from each of the files.

The Data.QualityParams.ScienceQualityParams contains a set of statistical values (min, max, mean, median std and masked pixel fraction) for the whole image and some Quality Parameters correlated to the astrometric and photometric calibration. For each detector are indeed provided the same set of statistics and the numbers of pixel flagged for each bit mask value.

Fig. 20 : NIR scientific calibrated frame DQC

The Data.QualityParams.BackgroundQualityParams contains the set of statistical values for each detector. Despite the background image pixel value is in [electron], the background detector statistical values are converted in [Myr/sr].

Data.QualityParams.PsfQualityParams and Data.QualityParams.PsfSingleQualityParams contain some quality information on PSF as the FWHM measured on the 2 axes, the number of sources used for PSF measurement and the size of radius encicling energy.

Fig. 21 : NIR PSF DQC