Make sure data IDs are expanded when adding Datasets and filling templates

Description

A data ID with (camera, visit, sensor) keys may be enough to uniquely identify a Dataset, but we should expand it to include physical_filter (implied by visit) and abstract_filter (implied by physical_filter), both so we can consistently populate the Dataset table and so we can use the expanded keys in templates.

Note that we will also need to expand data IDs before transferring them (e.g. via a QuantumGraph after preflight) to a limited Registry, as the limited Registry doesn't have the necessary information to expand them itself.  That implies that the expansion should include attaching the region, because PipelineTask execution may want that region and limited Registry can't attach it either.

Issue Matrix

hide

Checklist

Lucidchart Diagrams

Activity

Show:

Nate Lust January 2, 2019 at 3:08 PM

looks fine to me

Jim Bosch January 2, 2019 at 2:29 PM

, I've just added a ci_hsc PR with a unit test to check that data IDs are now expanded in gen2convert: https://github.com/lsst/ci_hsc/pull/67.

Nate Lust January 2, 2019 at 11:46 AM

Minor things on pr. I mostly followed the logic, but there were a few places I trusted you not knowing the context of the code. All the new code looked sound in its syntax.

Jim Bosch December 27, 2018 at 11:11 AM

, as we discussed this is not urgent, so continue to enjoy your holidays, but it should fix the bug with the filename templates that frequently caused outputs to be written to the same file. It's also, I'm afraid, a rather substantial change. I'd recommend reviewing commit-by-commit, to separate the shallow+broad API change commits from the deep+narrow implementation change commits.

Main PR is for daf_butler, with a totally trivial doc update in obs_base.

Jim Bosch December 10, 2018 at 9:39 AM

https://rubinobs.atlassian.net/browse/DM-15034#icft=DM-15034 includes a new DataId object that has a place for these extra metadata fields and a Registry API for populating them; this ticket now just involves a Butler-level and Datastore-level logic to find out what a template needs and request it.

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Jim Bosch

Reporter

Reviewers

Nate Lust

Story Points

RubinTeam

Data Release Production

Components

Checklist

Created September 7, 2018 at 2:21 PM
Updated February 4, 2019 at 9:44 AM
Resolved January 2, 2019 at 6:38 PM