Update Gen2 calibration policy formats in obs_subaru
Description
Attachments
blocks
relates to
Checklist
Lucidchart Diagrams
Issue Matrix
hideActivity
So...the answer is no. Having checked that the same CALIBs would be in play:
rootDirGen2 = "/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654"
butlerGen2 = dafPersist.Butler(rootDirGen2)
ccd = 49
dataType = "bias"
for visit in [1228, 1220, 1194, 1882,]:
print(butlerGen2.getUri(dataType, visit=visit, ccd=ccd))
for visit in [1228, 1220, 1194, 1882,]:
print(butlerGen2.getUri("calexp", visit=visit, ccd=ccd))
/datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits
/datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits
/datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits
/datasets/hsc/repo/CALIB/BIAS/2014-03-29/NONE/BIAS-2014-03-29-049.fits
/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-I/corr/CORR-0001228-049.fits
/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-R/corr/CORR-0001220-049.fits
/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00817/HSC-Z/corr/CORR-0001194-049.fits
/datasets/hsc/repo/rerun/RC/w_2021_06/DM-28654-sfm/00823/HSC-Y/corr/CORR-0001882-049.fitsI ran those visits on the ticket branch and made the same comparison plots against the w_20201_06 RC2 run.
The same general pattern of some outlier CCDs is present, but it seems to be a different set of CCDs each time. The good news is that all differences in the above new runs were all at the same very low levels.
Jenkins was indeed happy, so merged and done!
Thanks for the speedy turnaround, Chris! No, I don't know if the outlier detectors are always as such as I have only run the one visit. It's a good question though, so I'll run a few more visits (and in different bands) and have a look. It could also vary for different CALIBs, so, to start, I'll try to pick visits that will all use the same master bias and dark frames to take that element out of the equation.
This all looks fine to me (especially when I double checked scales). Mostly for my own curiosity, do you know if the outlier detectors in the final two focal plane plots are always different, or if the set changes depending on visit?
I don't have any concerns, so feel free to merge once Jenkins is happy.
Would you mind giving this a look? I think you were present at the meeting this morning where this issue was discussed (at some length!), but do let me know if you would like any further detail.
Jenkins + ci_hsc is running.
As a validation of this fix, I ran two Gen2 processCcd processing of a single HSC visit (1228 which is HSC-I and in the COSMOS 9813 tract) with the latest w_2021_09 master stack, and against this change. I also ran the same visit with Gen3. The latter Gen2 and Gen3 visits now compare bitwise identical for all image planes (image, variance, mask) in both the icExp and calexp outputs for all 103 CCDs. The src catalogs also compare almost identically, but for the minor differences noted in https://rubinobs.atlassian.net/browse/DM-28858#icft=DM-28858 (based on ci_hsc_gen2 vs. ci_hsc_gen3 comparisons) and currently under investigation (but blocked by this ticket) on https://rubinobs.atlassian.net/browse/DM-28936#icft=DM-28936.
I also compared the two Gen2 runs. While there are very small differences in the data products due to the minor differences in the variance planes (resulting in slight thresholding differences --> slightly different numbers of CRs detected and source footprint sizes --> slight differences in centroids for deblended sources, etc...), these were discussed during Princeton's regular Monday Meeting on Mar 1, 2021, and it was agreed they were indeed at too small a level to cause any concern and that this change should be made. Direct comparison plots can be perused (for the time being!) here and I've attached a few examples to the ticket for posterity (make particular note of the scales/units on these comparison plots...differences are tiny!).
Details
Details
Assignee
Reporter
Labels
Reviewers
Story Points
RubinTeam
Components
Sprint
Priority
UndefinedChecklist
Open Checklist
Checklist
The Gen2
obs_subarupolicy filepolicy/HscMapper.yamldefines bias and dark calibrations to be of typeImageF, which, by definition, does not have mask or variance planes. On read, these are passed through a standardization step that convertsImageFto anExposureFby adding empty mask and variance planes. However, when running a comparison test using Gen3 middleware (on DM-28936), it was revealed that these files do have variance and masks planes, it turns out these calibration files do have mask and variance planes and uses them when updating the variance planes throughout the processing. So, as it stands, Gen2 and Gen3 are behaving slightly differently leading to small (and very likely scientifically insignificant) differences, putting a dent in our efforts towards acheiving Gen2/Gen3 parity through Single Frame Processing. The fix is to update the bias and dark formats in the policy file fromImageFtoMaskedImageF, which will be done here.