Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Switch to using an explicit export of conda packages

Description

Using conda env export is susceptible to not being resolvable if a package is marked as broken. We should switch to using conda list --explicit to avoid this issue.

This triggers a few different modifications:
1. newinstall/lsstsw need to be modified to install using conda install. This is a different workflow than the existing workflow.
2. newinstall/lsstsw should probably be backwards compatible with previous methods
3. pip packages should be moved into conda-forge (pytest-subtests, peigen) as conda list --explicit does not seem to spit out packages installed by pip
4. Some additional work with the conda-forge team is necessary to ensure broken packages are NOT removed for the CDN the way the currently are, but are removed from the metadata. This ensure broken packages are still installable

Checklist

Lucidchart Diagrams

Issue Matrix

hide

Activity

Show:
Gabriele Comoretto [X]
June 5, 2020 at 5:30 PM

If everything has been merge, can the status of this issue be changed to DONE, so I can start with the 20.0.0 release candidate 1?

Kian-Tat Lim
June 4, 2020 at 11:59 PM

Since the DMCCB has decided to implement this for the release to ensure reproducibility, this ticket should block the release.

Kian-Tat Lim
June 4, 2020 at 8:21 PM

Brian will use the same lock files that were generated a while back, as that is the environment we actually want to freeze. He will update the _packages file to correspond, even though we won't actually use it anymore. He will also add new versions of the lock, bleed, and packages files to the qserv branch for building that product.

Kian-Tat Lim
June 4, 2020 at 8:19 PM

I was worried about removing the python version option (because we might need to go to Py4 in the future), but Brian convinced me that this was only for the base miniconda environment and that it didn't affect our environment.

I was also questioning the conda clean commands, but they seem to be quite useful to minimize space utilization, so they can stay.

There are a couple of other trivial comments; the rest looks great.

John Swinbank
June 4, 2020 at 5:05 PM

Please remember to update the README.md in scipipe_conda_env to describe how to generate these lockfiles. (Or remove that readme, and have the whole process documented somewhere central.)

Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Reviewers

Gabriele Comoretto [X]
Kian-Tat Lim

RubinTeam

Components

Checklist

Created May 18, 2020 at 7:23 PM
Updated June 19, 2020 at 1:28 AM
Resolved June 5, 2020 at 5:40 PM