Test dataset disassembly with ci_hsc_gen3
Description
is triggering
Checklist
Lucidchart Diagrams
Issue Matrix
hideActivity
Simon Krughoff August 3, 2020 at 11:53 PM
Looks good.
Tim Jenness July 31, 2020 at 3:27 AM
I'm sorry. See the stack trace in an early comment involving meas_astrom and filter determination. Whatever code was using the results of that meas_astrom filter determination failed when it thought the filter was "r" but worked fine when it thought it was HSC-R even without the filters for HSC being registered. It seemed like a part of the code you were familiar with. I don't know exactly which part of the pipeline was using it but it was reading the filter from an icExp.
John Parejko July 31, 2020 at 1:17 AM
I don't understand your point 1) above. "using HSC-R" - using it how and where?
Tim Jenness July 31, 2020 at 12:25 AM
Follow up on this, if I clone the Exposure read/write behavior with Filter (by using the FILTER keyword and ignoring what happens to be in Filter) everything works. This raises two issues:
The code in meas_astrom using HSC-R works but using r does not. I assume this is what we expect (?)
There is a lot of metadata manipulation in
Exposure.writeFits
that is not reproducible if each component inExposure
is written out separately. In many cases this does not matter because we are explicitly callingwriteFits
on each component so we don't need to rely on storing the values in metadata and then regenerating them from metadata. Somewhere in the pipeline the Filter is set to r but never gets over-ridden by the FILTER header value.
I'm not sure how much of this is a problem. For now I will recreate how FILTER is handled in Exposure.writeFits inside ExposureAssembler. That can be removed when can be relied upon.
Tim Jenness July 30, 2020 at 11:33 PM
All of the changes on this ticket are good and should be reviewed and ci_hsc_gen3 does complete with them. A fundamental problem though is that when multiprocessing is enabled (with the -j
option) it breaks because the FILTER singleton is not initialized. Normally this is done when an Instrument
is instantiated but in multi processing this never happens.
The error is:
and in single process mode the filter yaml look like:
but in multiprocessing they look like:
Composite disassembly is not properly tested anywhere for afw Exposures. Run ci_hsc_gen3 with disassembly turned on and see what breaks.