Make sure data IDs are expanded when adding Datasets and filling templates
Description
blocks
is blocked by
is duplicated by
Issue Matrix
hideChecklist
Lucidchart Diagrams
Activity
Nate Lust January 2, 2019 at 3:08 PM
looks fine to me
Jim Bosch January 2, 2019 at 2:29 PM
@Nate Lust, I've just added a ci_hsc PR with a unit test to check that data IDs are now expanded in gen2convert: https://github.com/lsst/ci_hsc/pull/67.
Nate Lust January 2, 2019 at 11:46 AM
Minor things on pr. I mostly followed the logic, but there were a few places I trusted you not knowing the context of the code. All the new code looked sound in its syntax.
Jim Bosch December 27, 2018 at 11:11 AM
@Nate Lust, as we discussed this is not urgent, so continue to enjoy your holidays, but it should fix the bug with the filename templates that frequently caused outputs to be written to the same file. It's also, I'm afraid, a rather substantial change. I'd recommend reviewing commit-by-commit, to separate the shallow+broad API change commits from the deep+narrow implementation change commits.
Main PR is for daf_butler, with a totally trivial doc update in obs_base.
Jim Bosch December 10, 2018 at 9:39 AM
https://rubinobs.atlassian.net/browse/DM-15034#icft=DM-15034 includes a new DataId
object that has a place for these extra metadata fields and a Registry
API for populating them; this ticket now just involves a Butler-level and Datastore-level logic to find out what a template needs and request it.
A data ID with (camera, visit, sensor) keys may be enough to uniquely identify a Dataset, but we should expand it to include physical_filter (implied by visit) and abstract_filter (implied by physical_filter), both so we can consistently populate the Dataset table and so we can use the expanded keys in templates.
Note that we will also need to expand data IDs before transferring them (e.g. via a QuantumGraph after preflight) to a limited Registry, as the limited Registry doesn't have the necessary information to expand them itself. That implies that the expansion should include attaching the region, because PipelineTask execution may want that region and limited Registry can't attach it either.