lsst.validate.base — Metric measurement framework

lsst.validate.base is a framework for packages that measure software and data quality metrics. A metric can be any measurable scalar quantity; some examples are in the LSST Science Requirements Document (LPM-17), though packages can also define ad hoc metrics. Measurements made through lsst.validate.base can be uploaded to LSST’s SQUASH monitoring dashboard to help you see how code development affects performance.

Features

lsst.validate.base helps you build packages that make and report measurements to SQUASH:

• Define metrics and specifications (milestones) using a YAML schema, and access those definitions through Metric and Specification classes.
• Create semantically-rich measurement classes that record not only a value but also metadata like input parameters and measurement by-products using the MeasurementBase base class.
• Package input datasets as blobs that can power drill-down visualizations of measurements on the SQUASH dashboard.
• Record self-documenting datasets: values have units (though Astropy Quantity) as well as plot labels and descriptions (see the Datum class).
• Build a JSON document of measurements and blobs that’s ready to submit to the SQUASH web API using the Job class.

Using lsst.validate.base

• Defining metrics and specifications in YAML
  • Metric objects in YAML
  • Defining specifications in YAML
• Using metrics and specifications in Python
  • Creating Metric objects in Python
  • Checking a measurement against a Specification
  • Accessing Specification objects of a Metric
• Creating measurement classes
  • A minimal measurement class
  • Storing a measurement quantity
  • Registering measurement parameters
  • Storing extra measurement outputs
• Providing datasets to measurements through blobs
  • Template for a blob class
• Using Jobs to collect and serialize measurements
  • Collecting measurements in a Job
  • Getting measurements from a Job
  • Serializing to JSON
  • Uploading lsst.validate.base’s JSON to SQUASH

Python API reference

lsst.validate.base Package

Framework for measuring and defining performance metrics that can be submitted to the SQUASH service.

Functions

load_metrics(yaml_path)

Load metric from a YAML document into an ordered dictionary of Metrics.

Classes

BlobBase()	Base class for blobs: flexible containers of data that are serialized to JSON.
Datum([quantity, unit, label, description])	A value annotated with units, a plot label and description.
DeserializedBlob(name, id_, datums)	A concrete Blob deserialized from JSON.
DeserializedMeasurement([quantity, id_, ...])	Measurement deserialized from JSON.
Job([measurements, blobs])	A Job wraps all measurements and blob metadata associated with a validation run.
MeasurementBase()	Base class for Measurement classes.
Metric(name, description, operator_str[, ...])	Container for the definition of a metric and its specification levels.
QuantityAttributeMixin	Mixin with common attributes for classes that wrap an astropy.units.Quantity.
Specification(name, quantity[, unit, ...])	A specification level, or threshold, associated with a Metric.
ValidateError	Base error for validate_base.
ValidateSpecificationError	Error accessing or using requirement specifications.

Class Inheritance Diagram

lsst.validate.base.jsonmixin Module

Classes

JsonSerializationMixin

Mixin that provides JSON serialization support to subclasses.

lsst.validate.base.datummixin Module

Classes

DatumAttributeMixin

Mixin that provides a Datum-like API to non-Datum classes.