Fix lengths and datatypes on sized columns, primarily in ObsCore-related schemas

Description

The https://rubinobs.atlassian.net/browse/DM-43371 update will make the length field a requirement for columns with datatypes that are considered sized, namely char, string, unicode, and binary. This change would cause validation errors in sdm_schemas, as there are currently some columns with these types that are missing an explicit length field. For many of these, e.g., columns in the ObsCore schemas, an arraysize is present, which should instead be assigned to length.

A related issue is that there are a number of columns in the ObsCore-related schemas that have been assigned an incorrect type of char but should instead be using string. This was confirmed by looking at DDL dumped from the live version of an ObsCore database, where all such columns are VARCHAR , defined in Felis as string.

Tables affected by the above problems with lengths and datatypes include:

  • dp02_obscore

  • obsloctap

  • oga_live_obscore

  • imsim (There is just a minor fix needed here to add the length field to a single column.)

In the ObsCore-related schemas, there is an additional special case where access_url should be using the Felis text type rather than char, as the live ObsCore database uses TEXT in MySQL.

Activity

Show:
Gregory Dubois-Felsmann
July 29, 2024 at 11:51 PM

These were welcome fixes. We still have to clean up both:

  • the actual choices of Felis `char` versus Felis `string`, which I am convinced were not made consciously in order to make some strings fixed-length and others variable-length; AND

  • the appropriate default mappings to VOTable `arraysize` for fixed- and variable-length strings.

See, among others, https://rubinobs.atlassian.net/browse/DM-45375#icft=DM-45375 .

Gregory Dubois-Felsmann
June 17, 2024 at 11:39 PM

This cannot be reviewed yet, because it has not been deployed in production. It’s in sdm_schemas 2.6.2, which is still awaiting release through Phalanx.

Gregory Dubois-Felsmann
May 7, 2024 at 12:09 AM
(edited)

PR has been reviewed and merged. Code is in sdm_schemas 2.6.2; not deployed yet as of today.

Gregory Dubois-Felsmann
May 1, 2024 at 10:37 PM

Confirming that the intent for the ObsCore attributes is for the string-valued ones to be variable-length strings, i.e., Felis string.

Will review PR later today.

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Reviewers

Gregory Dubois-Felsmann

Story Points

RubinTeam

Components

Checklist

Created April 17, 2024 at 10:03 PM
Updated July 29, 2024 at 11:56 PM
Resolved July 29, 2024 at 11:56 PM