Done
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Tim Jenness
Tim JennessReporter
Tim Jenness
Tim JennessReviewers
Simon Krughoff
Story Points
1
RubinTeam
Architecture
Components
Sprint
None
Priority
Checklist
Checklist
Created January 7, 2020 at 8:30 PM
Updated February 3, 2020 at 10:57 PM
Resolved January 8, 2020 at 6:01 PM
I'm testing the science pipelines with numpy1.17/astropy4/matplotlib3.1 and validate_drp fails:
====================================================================== ERROR: testPlotMetricsFromJsonJob (__main__.ParseJsonJob) Does plotting the metrics run and produce the correct filenames? ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/axis.py", line 1550, in convert_units ret = self.converter.convert(x, self.units, self) File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/astropy/visualization/units.py", line 101, in convert elif isinstance(val, list) and isinstance(val[0], u.Quantity): IndexError: list index out of range The above exception was the direct cause of the following exception: Traceback (most recent call last): File "tests/test_load_json.py", line 93, in testPlotMetricsFromJsonJob plot_metrics(job, filterName) File "/Users/timj/work/lsst/tmp/lsstsw/build/validate_drp/python/lsst/validate/drp/validate.py", line 455, in plot_metrics outputPrefix=outputPrefix) File "/Users/timj/work/lsst/tmp/lsstsw/build/validate_drp/python/lsst/validate/drp/plot.py", line 314, in plotPhotometryErrorModel histtype='stepfilled', orientation='horizontal') File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/__init__.py", line 1601, in inner return func(ax, *map(sanitize_sequence, args), **kwargs) File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 6694, in hist x = [self.convert_xunits(xi) for xi in x] File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 6694, in <listcomp> x = [self.convert_xunits(xi) for xi in x] File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/artist.py", line 180, in convert_xunits return ax.xaxis.convert_units(x) File "/Users/timj/work/lsst/tmp/lsstsw/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/matplotlib/axis.py", line 1553, in convert_units f'units: {x!r}') from e matplotlib.units.ConversionError: Failed to convert value(s) to axis units: []
The root cause seems to be that in
plotPhotometryErrorModel
themmagRmsHighSnr
variable is empty because there are no high signal to noise measurements. For the current astropy/numpy/matplotlib this triggers:/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice. out=out, **kwargs) /Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount)
but continues regardless. With newer versions it fails badly in Astropy with an index lookup gone bad. This is arguably a bug in Astropy that should be a bit more defensive, but it also seems to me that we shouldn't be asking matplotlib to calculate a histogram with no data in it.
The quick fix would seem to me to be to not try to plot the histogram if there are no data. Is that acceptable? Are we expecting there to be any numbers from the bright mask?
This does make the tests pass:
diff --git a/python/lsst/validate/drp/plot.py b/python/lsst/validate/drp/plot.py index 36a3cff..3ab6c58 100644 --- a/python/lsst/validate/drp/plot.py +++ b/python/lsst/validate/drp/plot.py @@ -118,8 +118,9 @@ def plotAstrometryErrorModel(dataset, astromModel, outputPrefix=''): ax[0].hist(dist, bins=100, color=color['all'], histtype='stepfilled', orientation='horizontal') - ax[0].hist(dist[bright], bins=100, color=color['bright'], - histtype='stepfilled', orientation='horizontal') + if len(dist[bright]): + ax[0].hist(dist[bright], bins=100, color=color['bright'], + histtype='stepfilled', orientation='horizontal') ax[0].set_ylim([0., 500.]) ax[0].set_ylabel("Distance [{unit:latex}]".format(unit=dist.unit)) @@ -307,10 +308,11 @@ def plotPhotometryErrorModel(dataset, photomModel, ax[0][0].hist(mmagRms, bins=100, range=(0, 500), color=color['all'], histtype='stepfilled', orientation='horizontal') - ax[0][0].hist(mmagRmsHighSnr, - bins=100, range=(0, 500), - color=color['bright'], - histtype='stepfilled', orientation='horizontal') + if len(mmagRmsHighSnr): + ax[0][0].hist(mmagRmsHighSnr, + bins=100, range=(0, 500), + color=color['bright'], + histtype='stepfilled', orientation='horizontal') plotOutlinedAxline( ax[0][0].axhline, mmagrms_median.value,