Vis Calibrated Quad Frame Product¶
Data product name¶
DpdVisCalibratedQuadFrame
Data product custodian¶
VIS
Name of the Schema file¶
euc-vis-CalibratedQuadFrame.xsdLast Edited for DPDD Version¶
2.0
Processing Element(s) creating/using the data product¶
VIS_science_xml_out
Processing function using the data product¶
SHE and MER processing functions.
Proposed for inclusion in EAS/SAS¶
Yes
Data product elements¶
- Header:
object of type sys:genericHeader
- Data:
object of type vis:visCalibratedQuadFrame
- QualityFlags:
object of type dqc:sqfDpdCalibratedFrame
- Parameters:
object of type ppr:genericKeyValueParameters
Detailed description of the data product¶
The VIS processing function provides a CalibratedQuadFrame product consisting of 3 FITS files per exposure: a calibrated VIS individual exposure, and the corresponding Background map and Weight MAP.
A PSF file is also provided. This corresponds to the PSF part of the VisPSFModel.
In all products, pixels are delivered at the native pixel scale and all extensions have the same pixel scale and size. Pixel data is in 32-bit floating point format.
The unit of the pixel is ADU for all product. To convert the image to electrons you need to multiply it by the GAIN, which is expressed in e-/ADU.
To convert the image into mJy, you need to use
Some exposures cannot be used for scientific purposes, or can only be used with certain precautions. These include exposures affected by, stray light, X-rays or guidance errors. For selected useful data, some metrics are included in the .xml file that follows each FITS file. See the VisAnalysisResults for how to interpret these metrics.
Changes introduced by the PF version¶
PF 14.0¶
This data product is added in replacement of DpdVisCalibratedFrame.
Flags definition changes.
Contents of VIS Noise map updated.
Invalidate quadrants which contain too many cosmic rays.
Add PSF statistics and quality plots.
PF 15.0¶
Flags definition changes.
Quadrants containing too many cosmic rays are not invalidated.
New FITS keyword COSMICPC containing the percentage of COSMIC flagged pixels in extension.
Astrometry: if pointing reconstruction fails, use commanded pointing instead of crashing, set PTG_RECD to false and DpdVisCalibratedQuadFrame.Header.AutomatedValidationStatus set to INVALID.
PF 17.0¶
No major change in this product.
Calibrated VIS individual exposure¶
The calibrated VIS individual exposure is a multi-extension FITS file containing 3 extensions per quadrant, one for pixel data, one for the associated RMS map and one for the associated Flagmap, making 432 extensions for the full 144 quadrants which constitutes an exposure.
Each extension contains the keyword EXTNAME, which is <detector id>.SCI, <detector id>.RMS or <detector id>.FLG for Science, RMS map and Flagmap respectively. The “detector id” describes the extension following the scheme given in the VIS FPA ICD (EUCL-SAP-ICD-6-001).
SCI extension¶
The SCI extension contains calibrated pixel data for each of the VIS instrument quadrants. The individual images contain both an astrometric and photometric solution written in the FITS header.
No flux scaling is applied, but the results of the photometric solution computed by VIS are written in the FITS header keywords.
The images are not background subtracted but a background map is provided as a FITS file with the same pixel size as the individual images.
The astrometric solution is described as a PV projection. Images are not resampled and are delivered at the native VIS pixel scale (which of course means that the optical distortion of the instrument is still present). WCSFITS is used to compute the astrometric solution. It uses a DpdVisDistortionModel of the FPA to compute the astrometric solution (pointing and position angle). The pointing position is the centre of rotation of the FOV and is defined as the centre of the VIS FPA.
Fig. 11 : CCD numbering and location at detector plane level (Front view) from VIS FPA ICD (EUCL-SAP-ICD-6-001) issue 2.5 2014/20/10¶
RMS extension¶
The RMS map contains the noise at each pixel in the corresponding Science image, expressed as the absolute standard deviation and are the ones to use for photometry. It is calculated from the quadratic sum of the signal plus the readout noise.
Please note that the RMS map is not flagged/masked, it contains the invalid pixels values.
This RMS map has undergone several improvements along the different versions of the PF, which are summarised below.
PF-13.0
RMS maps are generated directly from the output image using the BACKGROUND_RMS SExtractor checkimage (Bertin & Arnouts 1996). These maps are generated by calculating the standard deviation of pixels in a sliding window on the input images, after object removal.
Note that these RMS maps only include the noise contribution from the background, and do not contain a contribution from the Poisson noise of each individual source.
PF-14.0 and after
The RMS map now contains the quadratic sum of the signal (background+objects in the science image) and the readout noise.
FLG extension¶
The Flagmap data format is defined in the data model and presented in VisFlagMap in a human readable format.
Data Header¶
Primary Header |
|
|---|---|
SIMPLE |
File does conform to FITS standard |
BITPIX |
Number of bits per data pixel |
NAXIS |
Number of data axes |
FITS_DEF |
FITS definition name |
EXTEND |
FITS dataset may contain extensions |
FITS_DEF |
FITS definition name |
FITS_VER |
FITS definition version |
TIMESYS |
Time scale of the time-related keywords |
PLAN_ID |
32-bit SOC Operation ID |
BUNIT |
Pixel data unit, e.g. ‘electrons’ or ‘adu’ |
OBASW |
On board application software version: version. |
DATE |
UT date when this file was created |
SOFTVERS |
Version of the simulator or version of the LE1 |
EXPTIME |
Commanded integration time in seconds, e.g. 565.0 |
IMG_CAT |
Data product category: SCIENCE, CALIB, TECHNICA |
IMG_T1 |
values in {BIAS, DARK, FLAT, LINEARITY, CHARGE_INJ |
IMG_T2 |
values in {LAMP, SKY, DOME, OTHER…} from SEQU |
OBSMODE |
Observation Mode (WIDE, DEEP, CALIBRATION) |
OBSTYPE |
DP technique (IMAGE, SPECTROIMAGE) |
INSTRUME |
VIS instrument |
TELESCOP |
Telescope name: EUCLID |
OBT_STA1 |
Integer, number of seconds of OBT (START_TIME) |
OBT_STA2 |
Integer, number of milliseconds of OBT (START_TIME) |
DATE-OBS |
UTC Date of observation (start), e.g. ‘2014-03-15T09:30:09.313’ |
MJD-OBS |
Inst. seq. start time in MJD |
FIELDID |
Field Id |
OBS_ID |
OSS_obsid |
DITHOBS |
OSS_inObsId |
PTGID |
OSS_pointingID |
EXPNUM |
Exposure Id in dither |
TOTEXP |
Total number of Exposures in the specific Point |
INSTRMOD |
VIS intrument mode: Science or Manual |
COMPALGO |
Science TM compression mode: NoCompression, Co |
COMPPARA |
Science TM compression parameters: pixels/block |
RA |
OSS_attEqLon |
DEC |
OSS_attEqLat |
PA |
OSS_attEqPosAngle |
EQUINOX |
Standard FK5 (years), e.g. 2000. |
SEQID |
VIS sequences as defined in EUCL-IFS-ICD-6-002 |
DATE-END |
Inst. seq. end time in UTC |
OBT_END1 |
Integer, number of seconds of OBT (VIS sequence) |
OBT_END2 |
Integer, number of milliseconds of OBT (VIS seq) |
NOMCHAIN |
VDGT2771 |
VSTART |
Index of first readout line (starting at 0) |
VEND |
Index of last readout line + 1 |
ROECFGID |
ROE configuration table ID |
RSUSINID |
Table version of RSU SINE table used in the RSU |
RSUFRID |
Table version of RSU FREQ table used in the RSU |
RSUSFRID |
Table version of RSU short FREQ table used in t |
RSUCFSTA |
Shutter unit configuration status: CLOSED_LOOP, |
TP_STAT |
Trap pumping status: off, parallel, serial, mul |
CUSTATUS |
Calibration unit status: On or Off |
CI_STAT |
Charge injection status: On or Off |
IDLTIME |
Time (s) between end of shutter opening or end |
OBT_STA |
OBT at start time |
DATE-ENH |
UTC of LE1 Enhancement |
AUX_VERS |
LE1 Enh. (AUX) Processor name & version |
ELAT |
OSS_attEclipLat |
ELONG |
OSS_attEclipLon |
POS |
OSS_attEclipPosAngle |
ALPHA |
OSS_alpha |
BETA |
OSS_beta |
SAA |
OSS_saa |
PATCH_ID |
OSS_obsPatchId |
FOV_LAT1 |
OSS_fovLat1 |
FOV_LAT2 |
OSS_fovLat2 |
FOV_LAT3 |
OSS_fovLat3 |
FOV_LAT4 |
OSS_fovLat4 |
FOV_LON1 |
OSS_fovLon1 |
FOV_LON2 |
OSS_fovLon2 |
FOV_LON3 |
OSS_fovLon3 |
FOV_LON4 |
OSS_fovLon4 |
VS1T2043 |
VS1T2043 |
VS1T2050 |
VS1T2050 |
VS1T2020 |
VS1T2020 |
VS1T2021 |
VS1T2021 |
VS1T2028 |
VS1T2028 |
VS1T2029 |
VS1T2029 |
VS1T2036 |
VS1T2036 |
VS1T2037 |
VS1T2037 |
VS2T2043 |
VS2T2043 |
VS2T2050 |
VS2T2050 |
VS2T2020 |
VS2T2020 |
VS2T2021 |
VS2T2021 |
VS2T2028 |
VS2T2028 |
VS2T2029 |
VS2T2029 |
VS2T2036 |
VS2T2036 |
VS2T2037 |
VS2T2037 |
VS3T2043 |
VS3T2043 |
VS3T2050 |
VS3T2050 |
VS3T2020 |
VS3T2020 |
VS3T2021 |
VS3T2021 |
VS3T2028 |
VS3T2028 |
VS3T2029 |
VS3T2029 |
VS3T2036 |
VS3T2036 |
VS3T2037 |
VS3T2037 |
VS4T2043 |
VS4T2043 |
VS4T2050 |
VS4T2050 |
VS4T2020 |
VS4T2020 |
VS4T2021 |
VS4T2021 |
VS4T2028 |
VS4T2028 |
VS4T2029 |
VS4T2029 |
VS4T2036 |
VS4T2036 |
VS4T2037 |
VS4T2037 |
VS5T2043 |
VS5T2043 |
VS5T2050 |
VS5T2050 |
VS5T2020 |
VS5T2020 |
VS5T2021 |
VS5T2021 |
VS5T2028 |
VS5T2028 |
VS5T2029 |
VS5T2029 |
VS5T2036 |
VS5T2036 |
VS5T2037 |
VS5T2037 |
VS6T2043 |
VS6T2043 |
VS6T2050 |
VS6T2050 |
VS6T2020 |
VS6T2020 |
VS6T2021 |
VS6T2021 |
VS6T2028 |
VS6T2028 |
VS6T2029 |
VS6T2029 |
VS6T2036 |
VS6T2036 |
VS6T2037 |
VS6T2037 |
VS7T2043 |
VS7T2043 |
VS7T2050 |
VS7T2050 |
VS7T2020 |
VS7T2020 |
VS7T2021 |
VS7T2021 |
VS7T2028 |
VS7T2028 |
VS7T2029 |
VS7T2029 |
VS7T2036 |
VS7T2036 |
VS7T2037 |
VS7T2037 |
VS8T2043 |
VS8T2043 |
VS8T2050 |
VS8T2050 |
VS8T2020 |
VS8T2020 |
VS8T2021 |
VS8T2021 |
VS8T2028 |
VS8T2028 |
VS8T2029 |
VS8T2029 |
VS8T2036 |
VS8T2036 |
VS8T2037 |
VS8T2037 |
VS9T2043 |
VS9T2043 |
VS9T2050 |
VS9T2050 |
VS9T2020 |
VS9T2020 |
VS9T2021 |
VS9T2021 |
VS9T2028 |
VS9T2028 |
VS9T2029 |
VS9T2029 |
VS9T2036 |
VS9T2036 |
VS9T2037 |
VS9T2037 |
VSAT2043 |
VSAT2043 |
VSAT2050 |
VSAT2050 |
VSAT2020 |
VSAT2020 |
VSAT2021 |
VSAT2021 |
VSAT2028 |
VSAT2028 |
VSAT2029 |
VSAT2029 |
VSAT2036 |
VSAT2036 |
VSAT2037 |
VSAT2037 |
VSBT2043 |
VSBT2043 |
VSBT2050 |
VSBT2050 |
VSBT2020 |
VSBT2020 |
VSBT2021 |
VSBT2021 |
VSBT2028 |
VSBT2028 |
VSBT2029 |
VSBT2029 |
VSBT2036 |
VSBT2036 |
VSBT2037 |
VSBT2037 |
VSCT2043 |
VSCT2043 |
VSCT2050 |
VSCT2050 |
VSCT2020 |
VSCT2020 |
VSCT2021 |
VSCT2021 |
VSCT2028 |
VSCT2028 |
VSCT2029 |
VSCT2029 |
VSCT2036 |
VSCT2036 |
VSCT2037 |
VSCT2037 |
POS_X |
OEM_x |
POS_Y |
OEM_y |
POS_Z |
OEM_z |
VEL_X |
OEM_vx |
VEL_Y |
OEM_vy |
VEL_Z |
OEM_vz |
CALBLKID |
OSS_obsType |
CALBLKVR |
OSS_variant |
DATASETR |
PPO DataSetRelease |
N_CCD |
Number of CCD |
BIASSEC |
Offset area relative to readout, 1-based |
MAGZEROP |
zero-point |
PHOT_ERR |
self calibration photometric error |
PHOTIRMS |
self calibration mag dispersion RMS |
APERCORR |
self calibration aperture correction MAG_50pix - MAG_13pix |
APCORRMS |
self calibration aperture correction rms error |
Extension Header For CalibratedFrame |
|
|---|---|
XTENSION |
IMAGE extension |
BITPIX |
Number of bits per data pixel |
NAXIS |
Number of data axes |
NAXIS1 |
Length of data axis 1 |
NAXIS2 |
Length of data axis 2 |
PCOUNT |
Required keyword; must = 0 |
GCOUNT |
Required keyword; must = 1 |
EXTNAME |
Format: CCD row-CCD column.quadrant id, e.g. 2.3-H |
DETID |
CCD-ID field from science TM (0-35) |
CCDID |
e.g. Detector ID, e.g. ‘0-0’, ‘1-1’ … ‘6-6’ |
ROEID |
ROE ID: 1-12 |
ROECTV |
ROE configuration table version |
BUNIT |
Pixel data unit, e.g. ‘electrons’ or ‘adu’ |
PRESCANX |
number of serial prescan pixels, e.g 51 |
OVRSCANX |
number of serial overscan pixels, e.g 29 |
OVRSCANY |
number of parallel overscan pixels, e.g 20 |
EXPDUR1 |
Nominal/Short, nominal with CI, Flat: duration |
EXPDUR2 |
Nominal/Short, nominal with CI, Flat: duration |
CRVAL1 |
Right ascension at ref pixel |
CRVAL2 |
Declination at ref pixel |
CRPIX1 |
Reference pixel x coordinate |
CRPIX2 |
Reference pixel y coordinate |
CD1_1 |
Translation matrix element |
CD1_2 |
Translation matrix element |
CD2_1 |
Translation matrix element |
CD2_2 |
Translation matrix element |
CTYPE1 |
Coordinamte 1 type |
CTYPE2 |
Coordinamte 2 type |
CUNIT1 |
Physical units of CRVAL1 |
CUNIT2 |
Physical units of CRVAL2 |
WCSAXES |
Number of coordinate axes |
CMPRTSCI |
Quadrant image compression ratio obtained on-board |
GAIN |
Maximum equivalent gain (e-/ADU) |
RDNOISE |
Read out Noise |
RA |
Commanded FPA pointing right ascension (deg) |
DEC |
Commanded FPA pointing declination (deg) |
PA |
Commanded FPA pointing position angle (deg) |
EXPTIME |
Commanded integration time in seconds, e.g. 565.0 |
SATLEVEL |
Pixel saturation level in e- |
SATURATE |
Pixel saturation level in ADU |
BIAS_TOP |
bias_cfg[“main_bias_line_end”] |
BIAS_BTM |
bias_cfg[“main_bias_line_start”] |
BIAS_LFT |
bias_cfg[“main_bias_col_start”] |
BIAS_RGT |
bias_cfg[“main_bias_col_end”] |
GAINCORR |
ref_gain/cal_gain |
LSFCORR |
True if the LargeScaleFlat corection was applied |
COSMICPC |
percent of COSMIC flagged pixels in extension |
STDCRMS |
|
NUMBRMS |
|
AVGRESID |
VIS mean residual |
PV1_1 |
|
PV2_1 |
|
PV1_2 |
|
PV2_2 |
|
PV1_4 |
|
PV2_4 |
|
PV1_5 |
|
PV2_5 |
|
PV1_6 |
|
PV2_6 |
|
PV1_7 |
|
PV2_7 |
|
PV1_8 |
|
PV2_8 |
|
PV1_9 |
|
PV2_9 |
|
PV1_10 |
|
PV2_10 |
|
RA_COMM |
commanding pointing RA |
DEC_COMM |
commanding pointing DEC |
PA_COMM |
commanding position angle |
REFCAT |
reference star catalogue file |
FPAMODEL |
fpa model file |
MAGZEROP |
zero-point |
PHOT_ERR |
self calibration photometric error |
PHOTIRMS |
self calibration mag dispersion RMS |
APERCORR |
self calibration aperture correction MAG_50pix - MAG_13pix |
APCORRMS |
self calibration aperture correction rms error |
FLXSCALE |
Flux scaling for stacked exposure |
Weight Map¶
When combining multiple dithered exposures from separate detectors with gaps it becomes important to assign a “confidence” to each detector pixel so that overlapping exposures can be combined and the noise level at each pixel can be correctly estimated. When combining individual exposures into stacks, this is the product to use as a weight map.
Internally, the VIS processing function uses relative “weight maps”, which are defined as \(w_j \propto \frac{1}{{\sigma_{j}}^{2}}\), where \(\sigma_j\) is the standard deviation (or “RMS”) of the \(j_{th}\) pixel.
At the end it corresponds to the small scale flat (PRNU) with all pixels having the invalid flag set to 0. This corresponds only to the background, not the object.
Data Header¶
Primary Header for Weight Map |
|
|---|---|
SIMPLE |
conforms to FITS standard |
BITPIX |
array data type |
NAXIS |
number of array dimensions |
EXTEND |
|
Extension Header for Weight Map |
|
|---|---|
XTENSION |
Image extension |
BITPIX |
array data type |
NAXIS |
number of array dimensions |
NAXIS1 |
|
NAXIS2 |
|
PCOUNT |
Number of parameters |
GCOUNT |
Number of groups |
EXTNAME |
|
CCDID |
|
WCSAXES |
|
CTYPE1 |
Coordinamte 1 type |
CTYPE2 |
Coordinamte 2 type |
CUNIT1 |
Physical units of CRVAL1 |
CUNIT2 |
Physical units of CRVAL2 |
CRPIX1 |
|
CRPIX2 |
|
CD1_1 |
|
CD1_2 |
|
CD2_1 |
|
CD2_2 |
|
CRVAL1 |
|
CRVAL2 |
|
STDCRMS |
|
NUMBRMS |
|
AVGRESID |
|
PV1_1 |
|
PV2_1 |
|
PV1_2 |
|
PV2_2 |
|
PV1_4 |
|
PV2_4 |
|
PV1_5 |
|
PV2_5 |
|
PV1_6 |
|
PV2_6 |
|
PV1_7 |
|
PV2_7 |
|
PV1_8 |
|
PV2_8 |
|
PV1_9 |
|
PV2_9 |
|
PV1_10 |
|
PV2_10 |
|
RA_COMM |
commanding pointing RA |
DEC_COMM |
commanding pointing DEC |
PA_COMM |
commanding position angle |
RA |
reconstructed pointing RA |
DEC |
reconstructed pointing DEC |
PA |
reconstructed position angle |
REFCAT |
reference star catalogue file |
FPAMODEL |
fpa model file |
Background Map¶
The background map delivered is calculated using NoiseChisel.
In PF V15.0.4 the background calculation changes, using NoiseChisel instead of SExtractor background check image.
NoiseChisel seems to capture the straylight very well.
As NoiseChisel is more sensitive to cosmic we have implemented some fall-back in case of crash. The type of background calculation can be found in the VisAnalysisResults associated with these different values:
NC1 : NoiseChisel methode 1
NC2 : NoiseChisel methode 2
HISTMODE : Mod of the image
SExtractor : SourceExtractor background
Data Header¶
Primary Header for Background Map |
|
|---|---|
SIMPLE |
This is a FITS file |
BITPIX |
|
NAXIS |
|
EXTEND |
This file may contain FITS extensions |
NEXTEND |
Number of extensions |
MAGZEROP |
zero-point |
PHOT_ERR |
self calibration photometric error |
PHOTIRMS |
self calibration mag dispersion RMS |
APERCORR |
self calibration aperture correction MAG_50pix - MAG_13pix |
APCORRMS |
self calibration aperture correction rms error |
EXPTIME |
Commanded integration time in seconds, e.g. 565.0 |
Extension Header for Background Map |
|
|---|---|
XTENSION |
Image extension |
BITPIX |
array data type |
NAXIS |
number of array dimensions |
NAXIS1 |
|
NAXIS2 |
|
PCOUNT |
Number of parameters |
GCOUNT |
Number of groups |
EXTNAME |
|
DETID |
CCD-ID field from science TM (0-35) |
CCDID |
Detector ID, e.g. ‘0-0’, ‘1-1’ … ‘6-6’ |
ROEID |
ROE ID: 1-12 |
ROECTV |
ROE configuration table version |
BUNIT |
Pixel data unit, e.g. ‘electrons’ or ‘adu’ |
PRESCANX |
Number of serial prescan pixels, e.g 51 |
OVRSCANX |
Number of serial overscan pixels, e.g 29 |
OVRSCANY |
Number of parallel overscan pixels, e.g 20 |
EXPDUR1 |
Nominal/Short, nominal with CI, Flat: duration |
EXPDUR2 |
Nominal/Short, nominal with CI, Flat: duration |
CRVAL1 |
|
CRVAL2 |
|
CRPIX1 |
|
CRPIX2 |
|
CD1_1 |
|
CD1_2 |
|
CD2_1 |
|
CD2_2 |
|
CTYPE1 |
Coordinamte 1 type |
CTYPE2 |
Coordinamte 2 type |
CUNIT1 |
Physical units of CRVAL1 |
CUNIT2 |
Physical units of CRVAL2 |
WCSAXES |
|
CMPRTSCI |
Quadrant image compression ratio obtained on-board |
GAIN |
|
RDNOISE |
|
RA |
reconstructed pointing RA |
DEC |
reconstructed pointing DEC |
PA |
reconstructed position angle |
EXPTIME |
Commanded integration time in seconds, e.g. 565.0 |
SATLEVEL |
pixel saturation level in e- |
SATURATE |
pixel saturation level in ADU |
BIAS_TOP |
|
BIAS_BTM |
|
BIAS_LFT |
|
BIAS_RGT |
|
GAINCORR |
ref_gain/cal_gain |
BIASLVLE |
|
BIASRONE |
|
BIASLVLF |
|
BIASRONF |
|
BIASLVLG |
|
BIASRONG |
|
BIASLVLH |
|
BIASRONH |
|
LSFCORR |
|
COSMICPC |
percent of COSMIC flagged pixels in extension |
STDCRMS |
|
NUMBRMS |
|
AVGRESID |
|
PV1_1 |
|
PV2_1 |
|
PV1_2 |
|
PV2_2 |
|
PV1_4 |
|
PV2_4 |
|
PV1_5 |
|
PV2_5 |
|
PV1_6 |
|
PV2_6 |
|
PV1_7 |
|
PV2_7 |
|
PV1_8 |
|
PV2_8 |
|
PV1_9 |
|
PV2_9 |
|
PV1_10 |
|
PV2_10 |
|
RA_COMM |
Commanding pointing RA |
DEC_COMM |
Commanding pointing DEC |
PA_COMM |
Commanding position angle |
REFCAT |
reference star catalogue file |
FPAMODEL |
|
MAGZEROP |
zero-point |
PHOT_ERR |
self calibration photometric error |
PHOTIRMS |
self calibration mag dispersion RMS |
APERCORR |
self calibration aperture correction MAG_50pix - MAG_13pix |
APCORRMS |
self calibration aperture correction rms error |
FLXSCALE |
Flux scaling for stacked exposure |
CCD_RA |
CCD center right ascension |
CCD_DEC |
CCD center declination |
BKG_MED |
CCD bkg median value in MJY/sr |
BKG_MAD |
CCD bkg median absolute deviation in MJy/sr |