.. _HKTMProduct: HK/TM Product --------------------- $SetSchemaPath soc/euc-soc-HktmProduct.xsd Data product name ================= .. DataProductNameStart $PrintDataProductName .. DataProductNameEnd Data product custodian ====================== .. DataProductCustodianStart $PrintDataProductCustodian .. DataProductCustodianEnd Name of the Schema file ======================= .. NameSchemaStart $PrintSchemaFilename .. NameSchemaEnd DPDD version ============ .. DpddVersionlTagStart 1.1 .. DpddVersionlTagEnd Processing Element(s) creating/using the data product ===================================================== .. PECreatorStart Creators: * SOC Consumers: * VIS_PF, NISP_PF .. PECreatorEnd Processing function using the data product ========================================== .. PFUsingStart VIS processing function, NISP processing function. .. PFUsingEnd Data product elements ===================== .. DataProductElementsStart $PrintDataProductElements .. DataProductElementsEnd Detailed description of the data product ======================================== .. DetailedDescStart The HK/TM Product is produced by SOC from the HK/TM data received by MOC and ingested into the SOC ARES System. This is intended to be used primarily, but not exclusively, by the LE1 Teams. The current approach for the HKTM products generation is done according to the following guidelines: - The HKTM data is extracted from the ARES system (Hadoop cluster). - The HKTM product data time span is 1 hour. That is, it contains parameter data with their time stamp included in the 1 hour period corresponding to that product. - The HKTM product is compossed of: - A series of FITS files, each of them with: - 1 Primary HDU (Header Data Units) with a general metadata header - 1 Extension layer (additional HDU) per parameter, each of them with one binary table with a number of rows (one per sample), and 2 columns, the first one with the sample time stamp, and the second one with the parameter value. - One XML file with the appropriate metadata information for the HKTM product. This contains a two-way index of the parameters: (parameter) → (file, hdu) and (file, hdu) → (parameter). - For each parameter, the type used in the corresponding table in the FITS file is stated. This could be removed, since this information already appears in the table HDU (TBD). - The set of data is split into several FITS files. The current splitting criteria is: #. 1 FITS file with VIS parameters #. 1 FITS file with NISP parameters #. 1 FITS file with AOCS parameters #. 1 FITS file with Flight Dynamics related parameters #. I FITS file with the rest of parameters - All the tables have a first column with a time stamp. It has been suggested to group all this information in a common time column. But this is a problem with a number of aspects: - First, there are lots of different sampling frequencies, so in order to have a common TIMESTAMP column, we should make an effort on grouping the different parameters by a common frequency. Grouping per frequency/period several thousands of parameters is a LOT of work. - This will not be valid for those non-periodic parameters, though; those parameters that get updated only from time to time. You could think about putting them in separate files. - Then, even if one succeeds with the grouping, you will have the problem of having in separate FITS files parameters that have a common nature/origin (although a different sampling frequency). Finally, even with a common sampling frequency, that doesn't mean that they share a common timestamp (imagine a very simplistic scenario of two 2-hour period parameters; one could be generated at odd hours, the other at even hours). In summary: what we get from the system is not parameter values for a given set of timestamps; what we obtain are sets of pairs (timestamp, param.value). So, storing them as pairs in the FITS files is not only faster, convenient and easier to use; it is actually how the data is generated and retrieved. Here you can see an example of the list of files generated for 1 hour that will conform the HKTM Product: :: Length Date Time Name --------- ---------- ----- ---- 52709760 01-27-2022 19:10 EUC_SOC_HKTM-AOCS_20211029T070000-20211029T080000.fits 11687040 01-27-2022 19:10 EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits 26786880 01-27-2022 19:10 EUC_SOC_HKTM-NISP_20211029T070000-20211029T080000.fits 10327680 01-27-2022 19:10 EUC_SOC_HKTM-FGS_20211029T070000-20211029T080000.fits 106574400 01-27-2022 19:11 EUC_SOC_HKTM-others_20211029T070000-20211029T080000.fits 2084434 01-28-2022 07:55 EUC_SOC_HKTM_20211029T070000-20211029T080000.xml --------- ------- 210170194 6 files Below, a (reduced) view of the content of the XML Metadata file: .. code-block:: xml
EUC_SOC_HKTM_20211029T070000-20211029T080000 DpdHKTMProduct HKTMProductGenerator 3.2.1 SOC TEST TEST NA VALID 2100-01-01T00:00:00.000Z 0 SOC 2022-01-27T18:11:59.802Z
2021-10-29T07:00:00.000000Z 2021-10-29T08:00:00.000000Z . . . . . . . . . . . . EUC_SOC_HKTM-AOCS_20211029T070000-20211029T080000.fits EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits EUC_SOC_HKTM-NISP_20211029T070000-20211029T080000.fits EUC_SOC_HKTM-FGS_20211029T070000-20211029T080000.fits EUC_SOC_HKTM-others_20211029T070000-20211029T080000.fits Finally, you can see below an excerpt of the metadata keywords for the different table extensions, each corresponding to a different parameter, in the FITS files. :: # HDU 0 in EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits: SIMPLE = T / conforms to FITS standard BITPIX = 8 / array data type NAXIS = 0 / number of array dimensions EXTEND = T CREATOR = 'HKTMProductGenerator 3.2.1' OBSERVER= 'Euclid Operator' COMMENT Multitable FITS file, with data retrieved from ARES system COMMENT File contains data for parameters in VIS group # HDU 1 in EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits: XTENSION= 'BINTABLE' / binary table extension BITPIX = 8 / array data type NAXIS = 2 / number of array dimensions NAXIS1 = 16 / length of dimension 1 NAXIS2 = 360 / length of dimension 2 PCOUNT = 0 / number of group parameters GCOUNT = 1 / number of groups TFIELDS = 2 / number of table fields TTYPE1 = 'TIMESTAMP' TFORM1 = 'K ' TTYPE2 = 'VMGT3035' TFORM2 = '8A ' EXTNAME = 'VMGT3035' / extension name . . . # HDU 46 in EUC_SOC_HKTM-VIS_20211029T070000-20211029T080000.fits: XTENSION= 'BINTABLE' / binary table extension BITPIX = 8 / array data type NAXIS = 2 / number of array dimensions NAXIS1 = 16 / length of dimension 1 NAXIS2 = 1800 / length of dimension 2 PCOUNT = 0 / number of group parameters GCOUNT = 1 / number of groups TFIELDS = 2 / number of table fields TTYPE1 = 'TIMESTAMP' TFORM1 = 'K ' TTYPE2 = 'VMGT3022' TFORM2 = 'D ' EXTNAME = 'VMGT3022' / extension name . . . .. DetailedDescEnd