DMTN-158

DM Milestone Status#

Provenance#

This document was generated based on the contents of the lsst-dm/milestones repository, version 5e3f9da0, dated 2024-12-18. This corresponds to the status recorded in the project controls system for September 2024.

Notation#

Throughout this document, the identifiers of completed milestones are set in italics; those of milestones which are still pending, in roman.

Summary#

The DM Subsystem is currently tracking 388 milestones: 7 at Level 1, 84 at Level 2, 272 at Level 3, and 23 at Level 4. 2 have no level defined. Of these, 359 have been completed. Of the incomplete milestones, 22 are late relative to the baseline schedule, while the remainder are scheduled for the future.

_images/burndown.png

Fig. 1 Milestone completion as a function of date.#

Currently overdue milestones#

There are 22 milestones overdue as of 2024-12-18 23:21:02.008865.

  • DLP-526: Archive Center Integration Complete at SLAC [Due 2024-12-09]

  • LDM-503-12: Ops rehearsal for commissioning #3 [Due 2024-11-01]

  • LDM-503-12a: LSSTcam On Sky Readiness [Due 2024-11-01]

  • LDM-503-14: Test report: DM Readiness for Science Verification Surveys [Due 2024-11-01]

  • LDM-503-16: Ops rehearsal for data processing #3 [Due 2024-11-01]

  • LDM-503-17: Final operations rehearsal [Due 2024-12-09]

  • LDM-503-18q: Qserv construction complete [Due 2024-01-11]

  • LDM-503-19a: All P1a DM requirements verified [Due 2024-11-01]

  • LDM-503-EFDc: EFD data is queriable through TAP in the Science Platform [Due 2024-11-01]

  • LDM-503-NET: Network Verification End-to-End [Due 2024-04-15]

  • LDM-503-RSPb: Science Platform ready for Science Verification [Due 2024-12-09]

  • DM-AP-17: Solar System Processing pipeline (SSP) available. [Due 2024-11-01]

  • DM-DRP-24: Physically motivated PSF model, including separate characterization of contributions from the atmosphere and the telescope system. [Due 2024-11-01]

  • DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects. [Due 2021-08-31]

  • DM-DRP-31: A photometric redshift is now provided for each object. [Due 2024-11-01]

  • DM-DRP-36: Solar System Processing integrated to data release processing. [Due 2024-11-01]

  • DM-DRP-37: Artifact rejection and background matching during coadd construction. [Due 2018-11-30]

  • DM-DAX-15: DAX Construction Complete [Due 2024-01-11]

  • DM-DAX-9: Butler Provenance System [Due 2024-11-01]

  • DM-SQRE-4: Notebook service ready for verification & validation [Due 2023-02-28]

  • DMTC-021101-1155: COMP: Encryption [Due 2023-12-07]

  • DMTC-021102-1175: COMP: Firewalls and Physical Security [Due 2023-12-06]

Milestones by due date#

Due in March 2025#

Due in February 2025#

  • LDM-503-17a: Test report: Final Pipelines Delivery

  • DM-READY: DM is ready for ORR

  • LDM-503-19: All 1a,1b DM requirements verified

  • DM-AP-14: Alert filtering system available.

  • DM-AP-16: Full integration of the Alert Production system within the operational environment.

Due in January 2025#

  • No milestones due.

Due in December 2024#

  • DLP-526: Archive Center Integration Complete at SLAC

  • LDM-503-17: Final operations rehearsal

  • LDM-503-RSPb: Science Platform ready for Science Verification

Due in November 2024#

  • DM-NCSA-21: L1 Offline Processing Service, Raft Scale, ComCam

  • LDM-503-12: Ops rehearsal for commissioning #3

  • LDM-503-14: Test report: DM Readiness for Science Verification Surveys

  • LDM-503-16: Ops rehearsal for data processing #3

  • LDM-503-12a: LSSTcam On Sky Readiness

  • LDM-503-EFDc: EFD data is queriable through TAP in the Science Platform

  • RSP-1200: RSP handles 1200 users

  • LDM-503-19a: All P1a DM requirements verified

  • DM-AP-15: Alert distribution system fully integrated.

  • DM-AP-17: Solar System Processing pipeline (SSP) available.

  • DM-DRP-24: Physically motivated PSF model, including separate characterization of contributions from the atmosphere and the telescope system.

  • DM-DRP-31: A photometric redshift is now provided for each object.

  • DM-DRP-36: Solar System Processing integrated to data release processing.

  • DM-DAX-9: Butler Provenance System

Due in October 2024#

  • No milestones due.

Due in September 2024#

  • No milestones due.

Due in August 2024#

  • No milestones due.

Due in July 2024#

  • LDM-503-15: Ops rehearsal for data release processing #2

Due in June 2024#

  • No milestones due.

Due in May 2024#

  • LDM-503-16a: Test report: Pipelines Release Spring 2023

Due in April 2024#

Due in March 2024#

  • No milestones due.

Due in February 2024#

Due in January 2024#

Due in December 2023#

Due in November 2023#

Due in October 2023#

  • No milestones due.

Due in September 2023#

  • No milestones due.

Due in August 2023#

  • No milestones due.

Due in July 2023#

  • No milestones due.

Due in June 2023#

  • LDM-503-EFDa: Engineering Facility Database at the summit capturing and enabling access to all telemetry with M1/M3 active

  • LDM-503-EFDb: EFD/telemetry data replicated at the LDF, stored in InfluxDB. aggregated into parquet files

Due in May 2023#

  • No milestones due.

Due in April 2023#

  • No milestones due.

Due in March 2023#

  • No milestones due.

Due in February 2023#

  • DM-SQRE-4: Notebook service ready for verification & validation

Due in January 2023#

  • No milestones due.

Due in December 2022#

  • No milestones due.

Due in November 2022#

  • No milestones due.

Due in October 2022#

  • A9440: START: Data Security

Due in September 2022#

  • LDM-503-RSPa: Science Platform ready for DP0.2 with Image Services

Due in August 2022#

  • No milestones due.

Due in July 2022#

  • No milestones due.

Due in June 2022#

  • No milestones due.

Due in May 2022#

  • S22F: S22 Complete

  • LDM-503-15a: Test report: Pipelines Release Spring 2022

Due in April 2022#

  • No milestones due.

Due in March 2022#

  • No milestones due.

Due in February 2022#

  • No milestones due.

Due in January 2022#

  • No milestones due.

Due in December 2021#

  • S22S: S22 Start

  • DM-PORTAL: Resume development of LSP portal aspect

  • DM-CLOUD: Decide on final data facility design for DR1

Due in November 2021#

  • F21F: F21 Complete

Due in October 2021#

  • No milestones due.

Due in September 2021#

Due in August 2021#

  • LDM-503-13a: Test report: Pipelines Release Fall 2020

  • DM-AP-12: Difference imaging is now agnostic to the PSF of the template image.

  • DM-AP-13: Trailed source measurement on difference images.

  • DM-DRP-26: Overlap resolution at tract & patch boundaries.

  • DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects.

  • DM-DRP-28: Difference images are now a first-class data product during data release processing.

  • DM-DRP-35: Simultaneous measurement across a suite of coadds representing different bandpasses, epocs, and flavors.

Due in July 2021#

  • No milestones due.

Due in June 2021#

  • DMTC-8100-2210: 100G ring operating at 99.99% availability - Complete

  • F21S: F21 Start

  • DM-DRP-23: Atmospheric characterization based on data from the Auxiliary Telescope now available.

Due in May 2021#

Due in April 2021#

Due in March 2021#

  • No milestones due.

Due in February 2021#

  • No milestones due.

Due in January 2021#

Due in December 2020#

  • DM-NET-5: Base - Archive Network Functional 100 Gbps

  • DLP-464: Install networking for normal Base Center operations for ComCam scale

  • DLP-466: Install networking for ComCam-scale Chilean DAC serving on order 20 people

  • LDM-503-09: Test report: Ops rehearsal for commissioning #1

  • LDM-503-11b: Test report: Pipelines Release Fall 2019

  • LDM-503-14a: Science Platform ready for DP0

Due in November 2020#

  • DM-NCSA-15: Batch Processing Service for offline spectrograph data processing

  • DM-NCSA-22: Information in consolidated database available to QA portal

  • DLP-465: Install networking for normal Base Center operations for full-camera scale

  • DLP-468: Provide networking for formal Chilean DAC production at full-camera scale

  • DLP-467: Install networking for full-camera-scale Chilean DAC serving on order 80 people

  • F20F: F20 Complete

  • LDM-503-08: Test report: Spectrograph data acquisition

  • LDM-503-10: Test report: DAQ validation

  • LDM-503-11a: ComCam Ops Readiness

  • DM-DRP-12: Background estimation over the full visit.

  • DM-DRP-13: PSF estimation over the full visit.

  • DM-DRP-15: All varieties of coadd required for object detection and characterization are now produced during normal pipeline operation (although not necessarily at the ultimately required level of fidelity).

  • DM-DRP-21: Integrated image characterization pipeline for Data Release Production.

  • DM-DRP-30: Forced photometry is now performed on individual processed visit images during data releases.

  • DM-DRP-34: Selection maps are generated during data releases.

  • DM-DAX-7: Generation 3 Butler, running on a POSIX file-system provided by NCSA

  • DM-DAX-8: Scatter-gather support in PipelineTask

  • DM-DAX-12: Generation 3 middleware at feature parity with Generation 2

  • DM-DAX-14: Provenance system review completed

  • DM-SQRE-6: IVOA Server-side Operations for Data Access API

  • DM-SQRE-7: IVOA-compliant image metadata services

Due in October 2020#

  • No milestones due.

Due in September 2020#

  • DLP-46: Full Capacity over Diverse path Ready

  • DMTC-8000-0990: Full capacity La Serena to Santiago over Primary link AND Secondary link - Complete

  • DMTC-8100-2410: FIU-CIARA Management Contract Delivery #6 Complete

  • DLP-33: Santiago - Boca Raton Full Integration Ready

  • DMTC-6800-1325: Network Verification Pre-Tests

Due in August 2020#

  • No milestones due.

Due in July 2020#

  • DMTC-8100-2170: 100G ring operating at 99.0% availability - Complete

Due in June 2020#

  • DM-NCSA-14: Data Backbone endpoints in Chile for ingestion and access, distribution over WAN, ingest at NCSA into custodial file sto

  • F20S: F20 Start

  • DM-AP-10: Advanced single frame measurement pipeline for Alert Production.

  • DM-DRP-22: Template generation integrated with Data Release Production pipelines.

Due in May 2020#

  • DLP-533: Base Center Integration Complete

  • S20F: S20 Complete

  • LDM-503-05: Test report: Alert distribution validation

  • LDM-503-06: Test report: DM ComCam interface verification readiness

  • LDM-503-10b: Large Scale CCOB Data Access

  • LDM-503-09a: Test report: Pipelines Release Fall 2018

  • DM-AP-6: Alert format defined & queue system available.

  • DM-AP-18: Alert packet generation

Due in April 2020#

  • No milestones due.

Due in March 2020#

  • No milestones due.

Due in February 2020#

  • DM-SQRE-3: Notebook service stable for commissioning and other internal project users

Due in January 2020#

  • A6150: Start Milestone: January 1st, 2020

  • DM-DRP-8: Calibration product generation for the Auxiliary Telescope.

Due in December 2019#

Due in November 2019#

  • F19F: F19 Complete

  • DM-STAFF: Staffing Checkpoint

  • LDM-503-08b: Small Scale CCOB Data Access

  • DM-AP-11: Difference imaging includes noise decorrelation and correction for differential chromatic refraction.

  • DM-DRP-20: Refined set of LSST calibration products.

Due in October 2019#

  • A1300: Start Milestone: October 1st, 2019

  • DLP-70: Miami - Chicago Early Integration Ready

  • DLP-69: Boca Raton - Chicago Early Integration Ready

  • DMTC-8200-0600: ESNet Implementation Phase 1- Complete

Due in September 2019#

Due in August 2019#

  • A4670: June 2019 Travel to Chile - Travel Start

  • A4680: June 2019 Travel to Chile - Travel End

  • A4750: Summit AA Systems Installed at the Summit

  • A4860: ComCam Systems in Tucson ready for NCSA

  • DLP-552: LSST Software Release 9.1 Complete, Ready for Full Camera

Due in July 2019#

  • DM-NCSA-11: Verified acquisition of raw and crosstalk-corrected exposures at raft scale, incl. correct metadata

  • A4369.9: Ship Hardware to BDC

  • IT-725-M: Base Data Center Network Complete

  • DM-NET-4: Base LAN installed

  • LDM-503-10a: Science Platform: TAP service with federated SSO A&A

Due in June 2019#

  • A2330: Ship AA System to Chile

  • A4640: Ship ComCam Hardware to Tucson

  • DMTC-8000-0950: Technical Implementation Plan for 40Gbps diverse path - Complete

  • F19S: F19 Start

Due in May 2019#

  • DM-NCSA-13: Header Writing Service for Spectrograph use case

  • A5090: Expected Arrival of ComCam Systems to NCSA

  • IT-716-M2: Furniture installed in Offices, purchased cabling received

  • DMTC-8100-2130: 100G ring operating at 98.0% availability - Complete

  • S19F: S19 Complete

  • DM-SUIT-7: Mapping between SUIT systems & NCSA auth system

  • DM-SUIT-10: SUIT deployment procedure

Due in April 2019#

  • DLP-36: Santiago - Miami Early Integration Ready

  • DMTC-8100-2112: Miami - Boca Raton path diverse fiber - Complete

Due in March 2019#

  • DM-NCSA-16: Perform ISR processing on ComCam-scale data.

  • DMTC-8000-0870: Report on functional fiber, including uptime, 1st year functional milestone - Complete

Due in February 2019#

  • DM-NCSA-3: Provide database for metadata, provenance, location and demonstrate ingest at small scale

  • DM-NCSA-4: Minimal support for the small operational schema including file metadata and provenance for every file, and record of in

  • IT-718-M: Base Center Full Occupancy

  • LDM-503-02: Test report: HSC reprocessing

  • LDM-503-04: Test report: Aux Tel DAQ integration functionality test

  • LDM-503-07: Test report: Camera data processing

  • LDM-503-04b: Test report: Aux Tel DAQ interface Integration Verification and Spectrograph Operations Rehearsal

  • DM-SUIT-5: Search and display processed HSC data

  • DM-DRP-2: Basic visualization and quality assessment tools operational on HSC-scale data volumes.

  • DM-DRP-11: Pipelines code provides supports for database ingestion of results.

  • DM-DAX-5: Database ingest in support of HSC reprocessing (ie, large catalog ingest)

Due in January 2019#

  • DM-AP-8: Advanced ISR, including ghost and linear feature masking, correction for the Brighter-Fatter effect and compensation for pixel response non-uniformity.

Due in December 2018#

Due in November 2018#

  • IT-716-M: Base Center Beneficial Occupancy

  • F18F: F18 Complete

  • DM-SUIT-8: SUIT portal integrated with workspace

  • DM-DRP-19: QA metrics are generated during pipeline execution.

  • DM-DRP-37: Artifact rejection and background matching during coadd construction.

  • DM-DRP-29: Moving point source model fitting now available.

  • DM-DAX-10: Per-object gather support in Butler

  • DM-SQRE-2: Notebook service capabilities are suitable for the commissioning team to develop notebooks for its needs

Due in October 2018#

  • DMTC-8100-2330: FIU-CIARA Management Contract Delivery #4 Complete

  • DMTC-8100-2530: Data passing test report - Complete

  • DM-DRP-9: Data reduction pipeline for the Auxiliary Telescope.

Due in September 2018#

Due in August 2018#

  • DM-NCSA-9: Test instance of alert distribution hosting service and L1 database in Development & Integration Enclave

  • DLP-551: LSST Software Release 8.1 Complete, Ready for Commissioning Camera

Due in July 2018#

  • DM-AP-9: JOINTCAL1: Jointcal at a functional level

  • DM-DRP-7: Coordinate transformation tool provided for use with the Collimated Beam Projector.

  • DM-DRP-17: Simultaneous photometric and astrometric fitting to multiple exposures.

Due in June 2018#

  • DMTC-8100-2090: 100G ring operating at 97.0% availability - Complete

  • F18S: F18 Start

  • DM-AP-7: Basic instrument signature removal (ISR) capability.

  • DM-DRP-3: PSF-homogenized coadd construction.

  • DM-DRP-5: Camera package supporting the LSST Camera.

  • DM-DRP-38: Camera package supporting the Commissioning Camera.

Due in May 2018#

  • DMTC-8000-0836: La Serena to Santiago with live traffic and uptime over LSST Lambda Ready - Complete

  • S18F: S18 Complete

Due in April 2018#

Due in March 2018#

  • DM-NCSA-5: Level 1 archiving system able to acquire pixel data from the Aux Tel DAQ, header metadata via OCS, assemble FITS image,

  • DM-NCSA-6: Ability to transfer files originating from Tucson to NCSA and ingest files at NCSA, including metadata and provenance

  • DM-NCSA-7: Capability to paint displays for Tucson and NCSA

  • DM-NET-2: Mountain - Base Network Functional 2 x 100 Gbps

  • DLP-61: La Serena – Santiago link Ready

  • DLP-44: La Serena to Santiago with test traffic AURA DWDM LSST Fiber

  • DMTC-8000-0826: Intermediate traffic testing and report - Complete

  • DLP-40: Summits-Gatehouse Initial Capability Ready

  • DLP-65: Mountain - Base LSST and AURA DWM End Node Installation and Test Ready

  • DM-DRP-32: Object classification system available.

  • DM-DAX-11: Transformed EFD schema

Due in February 2018#

  • No milestones due.

Due in January 2018#

  • DM-DRP-16: Global photometric fitting (e.g. Burke et al. Forward Global Calibration Method).

Due in December 2017#

Due in November 2017#

  • DLP-864: Coordination with NCSA/CC-IN2P3 JCC

  • DLP-368: Initial AP DMCS

  • DLP-459: Qserv capability to hold data reduced at NCSA

  • DLP-458: Ingest of bulk data from real cameras to ComCam scale

  • DLP-457: Support runs on real cameras data and ImSim data

  • DLP-442: Procure infrastructure for Base Center to meet FY18 goals

  • DLP-433: Procure infrastructure for Archive Center to FY18 scale

  • DMTC-8000-0818: Acceptance Test with Equipment Installed - Complete

  • DMTC-7400-2510: REUNA La Serena - Santiago Optical Channel in operation - Complete

  • DMTC-7400-2130: End of commissioning. Acceptance of fiber connection - Complete

  • F17F: F17 Complete

  • LDM-503-01: Test report: Science Platform with WISE data in PDAC

  • LDM-503-03: Test report: Alert generation validation

  • DM-SUIT-1: Search and display WISE sources (objects) in PDAC

  • DM-SUIT-4: Multiple data traces in chart space

  • DLP-207: Extra LSST query capabilities in web UI

  • DLP-205: LSST web UI search external catalog

  • DLP-222: Provide time-domain data view

  • DM-DRP-14: Insertion of simulated sources into the data stream to check pipeline performance.

  • DM-DRP-18: Initial multi-band deblending algorithm available.

  • DM-DAX-1: WISE data ingest to PDAC

  • DM-DAX-6: Prototype level 1 database

  • DLP-473: Async Queries in Qserv

  • DLP-725: Working Prototype of AP/L1 Database

  • DLP-344: Release SQuaSH-in-a-box automated deployment platform

  • DLP-352: Metrics of Interest integrated in running stack or post-tests

Due in October 2017#

Due in September 2017#

Due in August 2017#

  • DMTC-7400-2090: Report on functional fiber connections, including AURA equipment - Complete

  • DLP-558: LSST Software Release 7.1 Complete

  • DM-AP-1: Basic single frame measurement pipeline.

  • DM-AP-2: Alard & Lupton-style image differencing.

  • DM-AP-3: Point source & dipole measurement on difference images.

  • DM-AP-4: DIASource association

  • DM-AP-5: DIAObject generation

  • DM-DRP-6: Camera package supporting the Auxiliary Telescope.

  • DM-SQRE-1: Project internal Jupyter notebook service

Due in July 2017#

  • DMTC-7400-2420: REUNA La Serena - Santiago DWDM equipment - Complete

  • DM-SUIT-2: Search WISE coaded data single exposure images in PDAC (the images are from IRSA at IPAC, not NCSA)

  • DM-DAX-2: Query service supporting IVOA TAP protocol, w/ support for asynchronous queries

Due in June 2017#

Due in May 2017#

  • DM-NCSA-1: Provide regular reprocessing service for HSC data

  • DM-NCSA-2: Provide access to results of regular reprocessing (NB the form this takes depends upon available DAX functionality)

  • DLP-538: LSE-130 Data items Exchanged between Cam & DM - Phase 3

  • DMTC-0000-3000: LSST Software Release S17 Complete

  • S17F: S17 Complete

  • DM-DRP-1: HSC merger complete: all functionality deployed for the most recent HSC data release processing is now available within the LSST stack.

  • DM-DRP-4: Calibration product generation in support of basic ISR.

  • DLP-736: Performance and capability enhancements in support of catalog use cases

  • DLP-152: Initial PSF-matched image coaddition available

  • DLP-472: Qserv Data Distribution

Due in April 2017#

Due in March 2017#

  • No milestones due.

Due in February 2017#

  • DLP-37: Summits-Gatehouse Fiber Ready

  • DMTC-7400-2210: Fiber delivery by Telefonica - Complete

  • DLP-567: LSST Software Release 7.0 Complete

Due in January 2017#

  • DMTC-8000-0806: 3T Fiber Installation Certification and Acceptance Test - Complete

Due in December 2016#

Due in November 2016#

  • DLP-847: Meet camera DAQ/OCS milestones for FY16

  • DLP-366: Mid-scale processing of eligible camera data

  • DLP-412: Monitor development cluster

  • DMTC-0000-0030: LSST Software Release F16 Complete

  • DLP-871: Firefly server side code refactor

  • DLP-802: Alert Production Database (APDB) design

Due in October 2016#

  • No milestones due.

Due in September 2016#

  • DMTC-8100-2050: 100G ring operating at 95.0% availability - Complete

  • DMTC-8100-2250: FIU-CIARA Management Contract Delivery #2 Complete

  • DLP-544: LSE-72 OCS-DM master interface - Phase 3

  • DM-SUIT-3: Time series analysis tool for WISE data

Due in August 2016#

  • DLP-557: LSST Software Release 6.1 Complete

  • DLP-579: Usability and developer efficiency

  • DLP-346: SQuaSH Prototype using existing architectural components deployed

Due in July 2016#

  • DLP-541: LSE-68 Camera-DM Image data interfaces - Phase 3

Due in June 2016#

Due in May 2016#

  • DMTC-0000-0020: LSST Software Release X16 Complete

  • DLP-694: Prototype Brighter-Fatter correction available

Due in April 2016#

  • DLP-45: Delivery of “functional Pre-operations Link” (at least 4Gbps), Ready

  • DMTC-8000-0910: Delivery of 4Gbs link from La Serena to Santiago Ready - Complete

Due in March 2016#

Due in February 2016#

  • DLP-469: Coordination with NCSA/CC-IN2P3 JCC

  • DLP-358: Refinement of facility integration plan and traceability of science and operational requirements

  • DLP-364: Build prototype production sequence for the DMCS

  • DLP-850: Prototype authentication/authorization system complete

  • DLP-405: Investigate CAS integration with Internet2 AAA tools

  • DLP-411: Monitor OpenStack

  • DLP-424: Integration of DevOps to OpenStack

  • DLP-471: Refresh sizing model

  • DLP-431: Refinement of facility integration plan and traceability of science and operational requirements

  • DLP-566: LSST Software Release 6.0 Complete

  • DLP-872: Convert major portion of GWT in Firefly to pure JavaScript

  • DLP-486: Prototype merge & deduplication in tract & patch overlaps

  • DLP-103: Prototype multi-coadd peak association available

  • DLP-100: Consistent multi-band deblending available

  • DLP-649: Shared Scans

  • DLP-345: Prototype metrics collection system deployed

  • DLP-351: Intermediate Performance Metrics routinelly calculated and published

  • DLP-337: Release new stack documentation infrastructure

Due in January 2016#

  • DLP-452: Support datasets on basic file system infrastructure with apropos policies

  • DLP-41: Mountain - Base Fiber Path Ready

  • DMTC-7400-2170: Gatehouse to Summit contract signed; Design Ready - Complete

  • DLP-276: Dipole measurement suite

Due in December 2015#

  • DLP-43: Fiber Delivery Ready

  • DMTC-8000-0760: Technical Implementation plan of fiber connection (primary link) - Complete

  • DMTC-7400-0815: La Serena-Gatehouse path design Ready

  • DMTC-7400-2010: Gatehouse to La Serena path design Ready - Complete

  • DLP-288: HSC merger complete

  • DLP-700: Galaxy shear fitting parameters available

  • DLP-342: Publish updated introductory software guide to developing with the stack

  • DLP-341: Deploy stack user support and discussion forum

Due in November 2015#

  • DLP-101: Prototype coadd detection available

  • DLP-73: Footprints redesigned and refactored

  • DLP-635: Qserv Query Cancellation

Due in October 2015#

  • DMTC-8100-2010: First Data passed on 100G ring without errors - Complete

  • DLP-528: Archive Site Ready for Equipment Configuration

  • DLP-275: Deliver document on effects of DCR

  • DLP-805: Distributed Database and Table Management

Due in September 2015#

  • DLP-357: Initial refinement of facility integration plan

  • DLP-389: Implement advanced logging functionality, including event-based logging

  • DLP-42: La Serena to Santiago Path Design Ready

Due in August 2015#

  • DLP-356: Set up Joint Coordination Council (JCC) for coordination with satellite processing center

  • DLP-355: Establish MOA with CC-IN2P3 satellite processing center

  • DLP-361: Build prototype AP DMCS, including events

  • DLP-404: Identify candidate security packages

  • DLP-410: Implement Puppet for system configuration management

  • DLP-415: Evaluate iRODS, Datacat, and similar file system management technologies

  • DLP-423: DevOps assessment

  • DLP-430: Initial refinement of facility integration plan

  • DLP-556: LSST Software Release 5.1 Complete

  • DLP-218: Provide API for image display in Firefly

  • DLP-216: Provide basic APIs to use Firefly visualization components

  • DLP-503: Prototype direct coaddition

  • DLP-502: Prototype image artifact identification available

  • DLP-25: Prototype Web Service for SUI

  • DLP-340: Operating system testing for public stack releases increased

  • DLP-334: SQuaRE release improved or changed CI system

Due in July 2015#

  • DLP-451: Deploy OpenStack capability

  • DLP-256: First draft definition of workspace

  • DLP-564: Able to use some data-access APIs developed by database team

Due in June 2015#

  • DM-NET-1: Rubin Network Re-engineering

  • DLP-559: Mountain - Base Network Functional 1 Gbps

Due in May 2015#

  • DLP-400: Refactor basic event services

  • DLP-441: Base Center requirements document

  • DLP-817: Multi-node Multi-query Integration Testing Harness

  • DLP-800: Data Ingest into MySQL

Due in April 2015#

  • No milestones due.

Due in March 2015#

  • DLP-518: Evaluate best-of-breed tools. Technology exploration.

  • DLP-521: Define the data access APIs with database team

Due in February 2015#

  • DLP-450: Support lsst-dev for developers

  • DLP-449: Upgrade storage server hardware

  • DLP-555: LSST Software Release 5.0 Complete

  • DLP-644: Generalized Single Frame Astrometry Pipeline

Due in January 2015#

  • DLP-422: Evaluate OpenStack as a development environment

Due in December 2014#

  • DLP-520: Access Qserv for catalog search in SUI prototype

  • DLP-642: Camera Team using DM code

Due in November 2014#

  • DLP-360: Prepare for OCS Middleware integration with AP DMCS

  • DLP-359: Improve fault tolerance in AP simulator

  • DLP-818: CSS

  • DLP-801: Qserv uses C++-based Geometry Code

Due in October 2014#

  • DLP-519: Research the development framework for SUI

Due in September 2014#

  • DLP-554: LSST Software Release 4.0 Complete

  • DLP-724: Forced per-epoch photometry

  • DLP-716: Fundamental flux measurements

Due in August 2014#

  • DLP-578: Propose a workspace structure for discussion.

  • DLP-574: Evaluate best-of-breed tools. Technology exploration.

  • DLP-575: Evaluate best-of-breed tools. Technology exploration.

  • DLP-576: Evaluate best-of-breed tools. Technology exploration.

  • DLP-577: Evaluate best-of-breed tools. Technology exploration.

  • DLP-639: Re-design camera geometry classes

  • DLP-638: Refactor AFW R&D Code

  • DLP-588: AFW R&D code refactored

  • DLP-715: Fundamental centroid and shape measurements

  • DLP-697: Single epoch CModel minimization

  • DLP-696: Single epoch Sersic model fitting

  • DLP-836: New logging for DM

  • DLP-835: New xrootd client

  • DLP-834: Automated Testbed for Qserv

Milestones by WBS#

02C.00: Data Management Level 2 Milestones#

_images/graph_02C.00.png

Fig. 2 Relationships between milestones in WBS 02C.00 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-554: LSST Software Release 4.0 Complete#

  • WBS: 02C.00

  • Level: 2

  • Successors: DLP-556

  • Due: 2014-09-02

  • Completed: 2014-09-30

Warning

No description available

DLP-555: LSST Software Release 5.0 Complete#

  • WBS: 02C.00

  • Level: 2

  • Due: 2015-02-26

  • Completed: 2015-03-06

Warning

No description available

DLP-559: Mountain - Base Network Functional 1 Gbps#

  • WBS: 02C.00

  • Level: 1

  • Successors: DM-NET-2

  • Due: 2015-06-02

  • Completed: 2015-06-01

Warning

No description available

DLP-556: LSST Software Release 5.1 Complete#

Warning

No description available

DLP-528: Archive Site Ready for Equipment Configuration#

  • WBS: 02C.00

  • Level: 1

  • Due: 2015-10-01

  • Completed: 2015-10-01

This is complete in. data/pmcs/201904-ME.xls.

DLP-566: LSST Software Release 6.0 Complete#

Warning

No description available

DMTC-0000-0020: LSST Software Release X16 Complete#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: DLP-694

  • Due: 2016-05-31

  • Completed: 2016-07-12

Warning

No description available

DLP-541: LSE-68 Camera-DM Image data interfaces - Phase 3#

  • WBS: 02C.00

  • Level: 2

  • Successors: S17S

  • Due: 2016-07-05

  • Completed: 2016-10-31

Complete with LCR-1788 June 2019.

DLP-557: LSST Software Release 6.1 Complete#

Warning

No description available

DLP-579: Usability and developer efficiency#

  • WBS: 02C.00

  • Level: 3

  • Successors: DLP-557

  • Due: 2016-08-31

  • Completed: 2017-12-01

Warning

No description available

DLP-544: LSE-72 OCS-DM master interface - Phase 3#

  • WBS: 02C.00

  • Level: 2

  • Successors: DLP-558

  • Due: 2016-09-30

  • Completed: 2016-10-31

Warning

No description available

DMTC-0000-0030: LSST Software Release F16 Complete#

Warning

No description available

S17S: S17 Start#

Warning

No description available

DLP-567: LSST Software Release 7.0 Complete#

  • WBS: 02C.00

  • Level: 2

  • Due: 2017-02-28

  • Completed: 2017-02-28

Warning

No description available

DLP-538: LSE-130 Data items Exchanged between Cam & DM - Phase 3#

approval of LCR-1494 in June 2019 allows this to be closed, LSE-400 [Lim, 2019] covers the items.

DMTC-0000-3000: LSST Software Release S17 Complete#

Warning

No description available

S17F: S17 Complete#

Warning

No description available

F17S: F17 Start#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: DLP-736

  • Successors: F17F

  • Due: 2017-06-01

  • Completed: 2017-06-01

Warning

No description available

DLP-558: LSST Software Release 7.1 Complete#

Warning

No description available

F17F: F17 Complete#

Warning

No description available

LDM-503-01: Test report: Science Platform with WISE data in PDAC#

This test demonstrates the deployment of the Portal and API Aspects of the LSST Science Platform (LSE-319 [Jurić et al., 2019], LDM-542 [Dubois-Felsmann et al., 2019], LDM-554 [Dubois-Felsmann et al., 2019]) to serve all-sky precursor data, including catalogs totaling of order 50–100 billion rows, in a prototype LSST Data Access Center hosted in the LSST Data Facility at NCSA.

LDM-503-03: Test report: Alert generation validation#

This test demonstrates the successful execution of a prototype Alert Generation science payload (LDM-148 [Lim et al., 2020], LDM-151 [Swinbank et al., 2020]), processing data from precursor surveys at a relatively small scale using compute resources at the LSST Data Facility at NCSA.

S18S: S18 Start#

Warning

No description available

LSST-1160: Header Service Integration#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: DM-NCSA-27

  • Successors: LSST-1190

  • Due: 2017-12-29

  • Completed: 2018-03-31

Warning

No description available

LSST-1190: Control Interfaces Verification: Single Visit#

Warning

No description available

LSST-1200: Interface Verification: Single Visit#

Warning

No description available

LSST-1220: EFD ETL Under DM Control#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: LSST-1190

  • Successors: LSST-1580

  • Due: 2017-12-29

  • Completed: 2019-03-29

Warning

No description available

S18F: S18 Complete#

Warning

No description available

F18S: F18 Start#

Warning

No description available

DLP-551: LSST Software Release 8.1 Complete, Ready for Commissioning Camera#

  • WBS: 02C.00

  • Level: 1

  • Predecessors: DLP-558

  • Successors: DLP-552

  • Due: 2018-08-31

  • Completed: 2018-07-12

Warning

No description available

F18F: F18 Complete#

Warning

No description available

S19S: S19 Start#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S18S

  • Successors: S20S

  • Due: 2018-12-01

  • Completed: 2018-12-01

Warning

No description available

LDM-503-02: Test report: HSC reprocessing#

This test demonstrates the successful execution of a prototype Data Release Production science payload (LDM-148 [Lim et al., 2020], LDM-151 [Swinbank et al., 2020]), processing data from precursor surveys at a relatively small scale using compute resources at the LSST Data Facility at NCSA.

LDM-503-04: Test report: Aux Tel DAQ integration functionality test#

This test demonstrates that LSST data acquisition system can capture raw data from instrumentation and construct a well-formed image with proper headers.

LDM-503-07: Test report: Camera data processing#

This test demonstrates that the LSST Science Pipelines can be successfully used to load and perform basic processing of data from the LSST Camera test systems. In the process, it establishes that data is properly made available at the LSST Data Facility at NCSA, and is accessible through the LSST Science Platform.

LDM-503-04b: Test report: Aux Tel DAQ interface Integration Verification and Spectrograph Operations Rehearsal#

This test demonstrates that the well-formed image generated in LDM-503-04 can be written to the permanent record of the survey.

S19F: S19 Complete#

Time-based milestone marking the completion of the Spring 2019 development cycle.

F19S: F19 Start#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: F18S

  • Successors: F19F

  • Due: 2019-06-01

  • Completed: 2019-06-01

Warning

No description available

LDM-503-10a: Science Platform: TAP service with federated SSO A&A#

This test demonstrates the successful integration of a single-sign-on federated authentication system, and a basic authorization system, with the three Aspects of the LSST Science Platform (Portal, Notebook, and API), with the API Aspect containing at least a TAP service.

DLP-552: LSST Software Release 9.1 Complete, Ready for Full Camera#

This pipeline software release will precede LDM-503-12a. This will have all functionality required to process data from LSSTCam. It will not have the full functionality for data release processing.

LDM-503-08b: Small Scale CCOB Data Access#

This test demonstrates the integration of the CCOB with the permanent record of the survey.

F19F: F19 Complete#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: F19S

  • Successors: F20S, S20S, DM-STAFF

  • Due: 2019-11-27

  • Completed: 2019-11-29

Time-based milestone marking the completion of the Fall 2019 development cycle.

DM-STAFF: Staffing Checkpoint#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: F19F

  • Successors: LDM-503-13

  • Due: 2019-11-27

  • Completed: 2019-11-29

DM Management evaluate current DM staffing against project schedule and budget. If there is a shortfall through mid-2022 (the “ops gap”), this is the point at which we need to fill it.

S20S: S20 Start#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S19S, S19F, F19F

  • Successors: S21S, S20F

  • Due: 2019-12-02

  • Completed: 2019-12-02

Warning

No description available

DLP-533: Base Center Integration Complete#

  • WBS: 02C.00

  • Level: 1

  • Successors: COMC-1564, COMC-0620-2090

  • Due: 2020-05-13

  • Completed: 2020-02-28

This milestone represents the availability of the base center in Las Serena. This includes DM staging machines, the commissioning cluster. It also includes the Chilean Data Access center machines. The later will only be deployed after the US DAC is up.

LDM-503-06: Test report: DM ComCam interface verification readiness#

This milestone records successful transfer of an image equivalent to one raft from the DAQ at the summit to reliable storage in the LSST Data Facility at NCSA, from where it will be made available for scientific evaluation through the LSST Science Platform.

S20F: S20 Complete#

Time-based milestone marking the completion of the Spring 2020 development cycle.

LDM-503-05: Test report: Alert distribution validation#

This test demonstrates that a full-LSST-scale (10,000 messages every 39 seconds) alert stream can be distributed to end users. It further shows that simple filters based on alert contents can be applied to that stream.

LDM-503-10b: Large Scale CCOB Data Access#

Demonstrate the ability to transfer data from the CCOB with 21 rafts from SLAC and ingested at NCSA and make available through an instance of the LSP.

LDM-503-09a: Test report: Pipelines Release Fall 2018#

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2018 and early 2019. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

F20S: F20 Start#

Time-based milestone marking the start of the Fall 2020 development cycle.

F20F: F20 Complete#

Time-based milestone marking the completion of the Fall 2020 development cycle.

LDM-503-08: Test report: Spectrograph data acquisition#

This test demonstrates data acquisition from the Auxiliary Telescope spectrograph, transmission of all image files with associated metadata to NCSA for screening, and then further processing if needed.

LDM-503-10: Test report: DAQ validation#

Test of the extended data acquisition (DAQ) network in Chile from the Summit to the Base Data Center.

LDM-503-11a: ComCam Ops Readiness#

Demonstrate that all DM functionality required to support ComCam operations has been delivered.

LDM-503-09: Test report: Ops rehearsal for commissioning #1#

This milestone represents the completion of a multi-day exercise in which (simulated) observations are performed and data is processed in as close to an operational mode as is possible at this stage in construction. In particular, it will demonstrate transfer, archiving and ingestion of raw data; offline processing of calibration and science data; and curation of the resultant data products.

LDM-503-11b: Test report: Pipelines Release Fall 2019#

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2019 and early 2020. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-503-14a: Science Platform ready for DP0#

The Rubin Science Platform is ready to support Data Preview 0.1 as deployed on the IDF. The RSP capabilities are largely those demonstrated under previous milestones, but ported to the IDF and with deployment and authentication/authorization infrastructure improvements that support the scale of DP0.1 usage. The Notebook Aspect provides access to the Rubin Science Pipelines stack and to data exposed in a cloud-deployed Gen3 Butler repository; the API Aspect provides TAP queries against a Qserv deployment of catalog tables, but not image metadata; and the Portal Aspect provides a UI for visual and ADQL construction of queries against those tables. Image and image metadata access services in the API and Portal Aspects are not included.

LDM-503-11: Ops rehearsal for commissioning #2#

Template generation, including code required to build templates in both Data Release and Alert Production science payloads, can now product templates which are corrected for the effects of Differential Chromatic Refraction (DCR), as described in LDM-151 [Swinbank et al., 2020] §6.18. Further, noise in image differences can be “decorrelated”, as described in LDM-151 [Swinbank et al., 2020] §6.19.1. Both of these represents initial capabilities, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

S21S: S21 Start#

Warning

No description available

S21F: S21 Complete#

Time-based milestone marking the completion of the Spring 2021 development cycle.

DM-IT-FIB: Camera Fibers Ready#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S21F

  • Successors: COMC-060200-20100

  • Due: 2021-05-31

  • Completed: 2021-10-29

Camera fibers ready deployed from the Computer Room to the top of the TMA.

F21S: F21 Start#

Warning

No description available

LDM-503-13a: Test report: Pipelines Release Fall 2020#

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2020 and early 2021. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-GEN3: Gen3 parity demonstrated#

Show parity of Butler Gen3 with Butler Gen2 systems. We should be able to run processing pipelines with Gen3 which previously ran on Gen2. Demonstrate the ability to write new pipelines using the framework. Framework should be ready to support Preops processing, specifically DP0.2 (RTN-001 [O'Mullane, 2024]).

F21F: F21 Complete#

Time-based milestone marking the completion of the Fall 2021 development cycle.

S22S: S22 Start#

Warning

No description available

DM-PORTAL: Resume development of LSP portal aspect#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S22S

  • Successors: LSST-1580

  • Due: 2021-12-01

  • Completed: 2022-01-31

Warning

No description available

DM-CLOUD: Decide on final data facility design for DR1#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S22S

  • Successors: LSST-1580

  • Due: 2021-12-01

  • Completed: 2022-01-31

Warning

No description available

S22F: S22 Complete#

  • WBS: 02C.00

  • Level: 2

  • Successors: LSST-1590, DM11-, LDM-503-15a

  • Due: 2022-05-05

  • Completed: 2022-05-31

Time-based milestone marking the completion of the Spring 2022 development cycle.

LDM-503-15a: Test report: Pipelines Release Spring 2022#

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2021 and early 2022. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-503-RSPa: Science Platform ready for DP0.2 with Image Services#

The Rubin Science Platform is ready to support Data Preview 0.2 as deployed on the IDF. The RSP serves an end-to-end Rubin-Gen3-processed dataset with a substantial set of image and SDM-ified catalog data products. Beyond the capabilities provided under LDM-503-14a, this deployment provides: initial IVOA-oriented image metadata and image services in the API Aspect, specifically an ObsTAP service and a basic SODA-compatible image cutout service; Portal Aspect support for accessing these services and browsing images from the released dataset; and access to a User File Workspace from all three Aspects.

LDM-503-EFDa: Engineering Facility Database at the summit capturing and enabling access to all telemetry with M1/M3 active#

An instance of the Engineering and Facility Database at the summit capturing and enabling access to all telemetry, including from an active M1M3. This may be distinct from the Auxiliary Telescope EFD instance. Demonstrate the ability to run for five days, allowing data retrieval and plotting via notebooks.

LDM-503-EFDb: EFD/telemetry data replicated at the LDF, stored in InfluxDB. aggregated into parquet files#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-26344]

  • Predecessors: LDM-503-EFDa, COMC-0030

  • Successors: LSST-1520

  • Due: 2023-06-16

  • Completed: 2024-03-01 [DM-26347]

Demonstrate that EFD data from the Summit appears at a Data Facility. Show that the Parquet files generated contain all low frequency EFD values and summaries of high frequency values.

LDM-503-18q: Qserv construction complete#

  • WBS: 02C.00

  • Level: 4

  • Test specification: Undefined [DM-38758]

  • Successors: DM-READY

  • Due: 2024-01-11

  • Completion pending [DM-38759]

This milestone represents the point at which Qserv completes construction and enters full operation and maintenance. A testplan should be executed showing the requirements are verified.

NET-neutral: Cisco ACI replaced#

  • WBS: 02C.00

  • Level: Undefined

  • Successors: LDM-503-NET

  • Due: 2024-02-16

  • Completed: 2023-01-31

Warning

No description available

LDM-503-NET: Network Verification End-to-End#

Network verified.

LDM-503-13: Ops rehearsal for data release processing #1#

This milestone builds upon LDM-503-12, but with its scope extended beyond procedures which are necessary in commissioning to cover the operational era.

LDM-503-16a: Test report: Pipelines Release Spring 2023#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-32560]

  • Predecessors: COMC-0240

  • Successors: LDM-503-17a

  • Due: 2024-05-01

  • Completed: 2023-08-31 [DM-32561]

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2023 and early 2023. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-503-15: Ops rehearsal for data release processing #2#

This milestone builds upon LDM-503-13, and will demonstrate the production of a data release under simulated operational conditions using data from science verification activities.

LDM-503-12: Ops rehearsal for commissioning #3#

This milestone builds upon LDM-503-11, and serves to demonstrate that all necessary procedures and workflows are in place to support commissioning activities.

LDM-503-14: Test report: DM Readiness for Science Verification Surveys#

This milestone represents the point at which all code and/or services which are supplied by DM in support of science verification will be tested and demonstrated to meet their requirements.

LDM-503-16: Ops rehearsal for data processing #3#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-17138]

  • Predecessors: LSST-1520, LDM-503-15, SITCOM-130

  • Successors: DM7-, LDM-503-17, SITCOM-119

  • Due: 2024-11-01

  • Completion pending [DM-17139]

This milestone demonstrates our ability to do multi-site processing with at least one international site.

LDM-503-12a: LSSTcam On Sky Readiness#

Demonstrate that all DM functionality required to support operations with the full LSST Camera has been delivered.

LDM-503-EFDc: EFD data is queriable through TAP in the Science Platform#

Demonstrate that engineering data for a specific observation time can be retrieved via TAP on the Science Platform or directly to a TAP tool like Topcat. Test DMS-REQ-0102 (Priority 1a) to permanently archive metadata from OCS and make available to rights holders. Also test DMS-REQ-0358 (Priority 1a), demonstrating support of simultaneous queries.

RSP-1200: RSP handles 1200 users#

  • WBS: 02C.00

  • Level: 4

  • Predecessors: LSST-1520

  • Successors: LDM-503-17

  • Due: 2024-11-01

  • Completed: 2023-09-29

Warning

No description available

LDM-503-19a: All P1a DM requirements verified#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-38727]

  • Predecessors: LSST-1520

  • Successors: LSST-1580, LDM-503-14, SITCOM-114

  • Due: 2024-11-01

  • Completion pending [DM-38728]

This milestone represents the point at which all code and/or services needed for commissioning (i.e. 1a requirements) are verified.

DLP-526: Archive Center Integration Complete at SLAC#

  • WBS: 02C.00

  • Level: 1

  • Successors: DM2-, COMC-1664

  • Due: 2024-12-09

  • Completion pending

This milestone represents the availability of an archive center to host LSST DR1 data. This includes an science platform interface with the implied services for Data Access, a portal and Notebooks.

LDM-503-17: Final operations rehearsal#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-17140]

  • Predecessors: DM2-, DM11-, DM7-, LDM-503-16, RSP-1200, COMC-0620-9000

  • Successors: LSST-1580

  • Due: 2024-12-09

  • Completion pending [DM-17141]

This milestone builds upon LDM-503-16 to demonstrate successful operation of all aspects of the DM subsystem under simulated operational conditions.

LDM-503-RSPb: Science Platform ready for Science Verification#

The Rubin Science Platform is ready to support Science Verification activities in the final phase of Rubin Observatory commissioning. In additional to the capabilities demonstrated in Milestone-2 (LDM-503-RSPa), the RSP will be scalable to the user load required to support Science Verification; provide access to a regularly updating stream of newly-acquired LSSTCam data; provide access to next-to-data computing and user batch computing to support intensive data analysis; provide access to a User Database Workspace; provide a means for integrating with a TBD focal-plane-scale visualization tool; provide capabilities in all three Aspects for following links between related datasets and exploiting provenance information; provide access to the EFD for recently acquired data, extending the capability demonstrated for LDM-503-EFDc; and provide substantial user-facing documentation, with links between the data and documentation.

LDM-503-17a: Test report: Final Pipelines Delivery#

This milestone describes the final Science Pipelines release from the DM Construction Project. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

DM-READY: DM is ready for ORR#

Warning

No description available

LDM-503-19: All 1a,1b DM requirements verified#

This milestone represents the point at which all code and/or services needed for operations (i.e. 1a and 1b requirements) are verified.

02C.03: Alert Production#

_images/graph_02C.03.png

Fig. 3 Relationships between milestones in WBS 02C.03 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-639: Re-design camera geometry classes#

  • WBS: 02C.03.01

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-13

Warning

No description available

DLP-638: Refactor AFW R&D Code#

  • WBS: 02C.03.05

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-13

Warning

No description available

DLP-642: Camera Team using DM code#

  • WBS: 02C.03.01

  • Level: 3

  • Due: 2014-12-10

  • Completed: 2014-12-10

Warning

No description available

DLP-644: Generalized Single Frame Astrometry Pipeline#

  • WBS: 02C.03.08

  • Level: 3

  • Due: 2015-02-24

  • Completed: 2015-02-24

Warning

No description available

DLP-275: Deliver document on effects of DCR#

  • WBS: 02C.03.04

  • Level: 3

  • Successors: DLP-276

  • Due: 2015-10-29

  • Completed: 2016-04-08

Warning

No description available

DLP-276: Dipole measurement suite#

  • WBS: 02C.03.04

  • Level: 3

  • Predecessors: DLP-275

  • Due: 2016-01-22

  • Completed: 2016-07-12

Warning

No description available

DM-AP-1: Basic single frame measurement pipeline.#

Warning

No description available

DM-AP-2: Alard & Lupton-style image differencing.#

Warning

No description available

DM-AP-3: Point source & dipole measurement on difference images.#

Warning

No description available

DM-AP-4: DIASource association#

  • WBS: 02C.03

  • Level: 3

  • Successors: LDM-503-03

  • Due: 2017-08-31

  • Completed: 2017-11-01

Warning

No description available

DM-AP-5: DIAObject generation#

  • WBS: 02C.03

  • Level: 3

  • Successors: LDM-503-03

  • Due: 2017-08-31

  • Completed: 2017-11-01

Warning

No description available

DM-AP-7: Basic instrument signature removal (ISR) capability.#

Warning

No description available

DM-AP-9: JOINTCAL1: Jointcal at a functional level#

Warning

No description available

DM-AP-8: Advanced ISR, including ghost and linear feature masking, correction for the Brighter-Fatter effect and compensation for pixel response non-uniformity.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: F18F

  • Successors: LDM-503-11b

  • Due: 2019-01-04

  • Completed: 2019-01-17

Warning

No description available

DM-AP-11: Difference imaging includes noise decorrelation and correction for differential chromatic refraction.#

Template generation, including code required to build templates in both Data Release and Alert Production science payloads, can now product templates which are corrected for the effects of Differential Chromatic Refraction (DCR), as described in LDM-151 [Swinbank et al., 2020] §6.18. Further, noise in image differences can be “decorrelated”, as described in LDM-151 [Swinbank et al., 2020] §6.19.1. Both of these represents initial capabilities, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-AP-6: Alert format defined & queue system available.#

Warning

No description available

DM-AP-18: Alert packet generation#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: S20F

  • Successors: LDM-503-05, DM-AP-6

  • Due: 2020-05-29

  • Completed: 2020-05-29

The AP Pipeline is now capable of generating per-DIASource alert packets, which comply with the current version of the LSST alert schema. This milestone only covers packet generation; success involves only those serialized alerts being e.g. written to disk, and does not imply that an alert transport system is available.

DM-AP-10: Advanced single frame measurement pipeline for Alert Production.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: F20S, DM-AP-1

  • Successors: LSST-1580

  • Due: 2020-06-01

  • Completed: 2023-05-31

The Single Frame Processing system is now substantially complete and provides all functionality described in LDM-151 [Swinbank et al., 2020] §3.1. It supports all expected interfaces, data-flows, and products, but the details of the algorithms are expected to undergo continued refinement.

DM-AP-12: Difference imaging is now agnostic to the PSF of the template image.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: DM-AP-11

  • Successors: LDM-503-15a

  • Due: 2021-08-31

  • Completed: 2023-05-31

The Alert Generation science payload is now agnostic to the PSF of the template, as described in LDM-151 [Swinbank et al., 2020] §3.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-AP-13: Trailed source measurement on difference images.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: DM-AP-3

  • Successors: LDM-503-13a

  • Due: 2021-08-31

  • Completed: 2021-10-29

The Alert Generation science payload now provides the capability to fit a trailed point source model to detections made on difference images, as described in LDM-151 [Swinbank et al., 2020] §3.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-AP-15: Alert distribution system fully integrated.#

  • WBS: 02C.03

  • Level: 3

  • Successors: DM16-, DM11-, DM-AP-16

  • Due: 2024-11-01

  • Completed: 2024-06-28

The Alert Distribution system (LDM-151 [Swinbank et al., 2020] §3.3) is now available for deployment at the LSST Data Facility. The system is substantially complete, and supports all expected interfaces, data-flows, and products, but the details of the algorithms may undergo continued refinement.

DM-AP-17: Solar System Processing pipeline (SSP) available.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: LDM-503-12a, SITCOM-122

  • Successors: DM-DRP-36

  • Due: 2024-11-01

  • Completion pending

All of the algorithmic components which comprise the Solar System Processing (SSP) pipeline, as described in LDM-151 [Swinbank et al., 2020] §§6.25–6.28, are available for integration into science payloads. These components must provide realistic interfaces, data-flows and products, but it is expected that the details of the algorithms will undergo continued refinement.

DM-AP-14: Alert filtering system available.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: LDM-503-17a, DM-AP-6

  • Successors: LSST-1580

  • Due: 2025-02-27

  • Completion pending

The Alert Filtering Service (LDM-151 [Swinbank et al., 2020] §3.3.4; LSE-163 [Jurić et al., 2023] §3.5.2) can now be deployed as part of a prototype Alert Distribution system (LDM-151 [Swinbank et al., 2020] §3.3). This service is substantially complete, and supports all expected interfaces, data-flows, and products, but may not yet provide the complete set of filtering capabilities and/or may not perform at the level required to distribute the full LSST alert stream.

DM-AP-16: Full integration of the Alert Production system within the operational environment.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: LDM-503-17a, DM-AP-15, COMC-0240

  • Successors: LSST-1580

  • Due: 2025-02-27

  • Completion pending

The Alert Production system (comprising Single Frame Processing, LDM-151 [Swinbank et al., 2020] §3.1; Alert Generation, LDM-151 [Swinbank et al., 2020] §3.3; and Precovery and Forced Photometry, LDM-151 [Swinbank et al., 2020] §3.4) is now available for deployment at the LSST Data Facility. The system is substantially complete, and supports all expected interfaces, data-flows and products, but the details of the algorithms may undergo continued refinement.

02C.04: Data Release Production#

_images/graph_02C.04.png

Fig. 4 Relationships between milestones in WBS 02C.04 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-588: AFW R&D code refactored#

  • WBS: 02C.04.01

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-715: Fundamental centroid and shape measurements#

  • WBS: 02C.04.06

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2015-12-04

Warning

No description available

DLP-697: Single epoch CModel minimization#

  • WBS: 02C.04.06

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-696: Single epoch Sersic model fitting#

  • WBS: 02C.04.06

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-724: Forced per-epoch photometry#

  • WBS: 02C.04.06

  • Level: 3

  • Due: 2014-09-12

  • Completed: 2014-09-12

Warning

No description available

DLP-716: Fundamental flux measurements#

  • WBS: 02C.04.06

  • Level: 3

  • Due: 2014-09-12

  • Completed: 2014-09-12

Warning

No description available

DLP-503: Prototype direct coaddition#

  • WBS: 02C.04.04

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-08-31

Warning

No description available

DLP-502: Prototype image artifact identification available#

  • WBS: 02C.04.04

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-08-31

Warning

No description available

DLP-101: Prototype coadd detection available#

  • WBS: 02C.04.05

  • Level: 3

  • Due: 2015-11-24

  • Completed: 2016-04-11

Warning

No description available

DLP-73: Footprints redesigned and refactored#

  • WBS: 02C.04.01

  • Level: 3

  • Due: 2015-11-24

  • Completed: 2017-05-09

Warning

No description available

DLP-700: Galaxy shear fitting parameters available#

  • WBS: 02C.04.06

  • Level: 3

  • Due: 2015-12-04

  • Completed: 2016-05-31

Warning

No description available

DLP-288: HSC merger complete#

  • WBS: 02C.04.01

  • Level: 3

  • Due: 2015-12-29

  • Completed: 2016-12-07

Warning

No description available

DLP-486: Prototype merge & deduplication in tract & patch overlaps#

  • WBS: 02C.04.05

  • Level: 3

  • Successors: DLP-566

  • Due: 2016-02-29

  • Completed: 2016-04-11

Warning

No description available

DLP-103: Prototype multi-coadd peak association available#

  • WBS: 02C.04.05

  • Level: 3

  • Successors: DLP-566

  • Due: 2016-02-29

  • Completed: 2016-04-11

Warning

No description available

DLP-100: Consistent multi-band deblending available#

  • WBS: 02C.04.05

  • Level: 3

  • Successors: DLP-566

  • Due: 2016-02-29

  • Completed: 2016-04-11

Warning

No description available

DLP-694: Prototype Brighter-Fatter correction available#

  • WBS: 02C.04.02

  • Level: 3

  • Successors: DMTC-0000-0020

  • Due: 2016-05-31

  • Completed: 2016-05-31

Warning

No description available

DM-DRP-1: HSC merger complete: all functionality deployed for the most recent HSC data release processing is now available within the LSST stack.#

Warning

No description available

DM-DRP-4: Calibration product generation in support of basic ISR.#

Warning

No description available

DLP-736: Performance and capability enhancements in support of catalog use cases#

  • WBS: 02C.04.01

  • Level: 3

  • Predecessors: DMTC-0000-3000, S17F

  • Successors: F17S

  • Due: 2017-05-31

  • Completed: 2016-12-19

Warning

No description available

DLP-152: Initial PSF-matched image coaddition available#

  • WBS: 02C.04.04

  • Level: 3

  • Predecessors: S17F

  • Due: 2017-05-31

  • Completed: 2016-12-19

Warning

No description available

DM-DRP-6: Camera package supporting the Auxiliary Telescope.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-08

  • Due: 2017-08-31

  • Completed: 2018-01-31

Warning

No description available

DM-DRP-14: Insertion of simulated sources into the data stream to check pipeline performance.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-11b

  • Due: 2017-11-30

  • Completed: 2017-12-01

Warning

No description available

DM-DRP-18: Initial multi-band deblending algorithm available.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-11b

  • Due: 2017-11-30

  • Completed: 2017-12-01

Warning

No description available

DM-DRP-16: Global photometric fitting (e.g. Burke et al. Forward Global Calibration Method).#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-09a

  • Due: 2018-01-31

  • Completed: 2018-05-31

Warning

No description available

DM-DRP-32: Object classification system available.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-09a

  • Due: 2018-03-30

  • Completed: 2018-10-31

Warning

No description available

DM-DRP-3: PSF-homogenized coadd construction.#

Warning

No description available

DM-DRP-5: Camera package supporting the LSST Camera.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F

  • Successors: LDM-503-09a

  • Due: 2018-06-29

  • Completed: 2018-06-29

Warning

No description available

DM-DRP-38: Camera package supporting the Commissioning Camera.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F

  • Successors: LDM-503-09a

  • Due: 2018-06-29

  • Completed: 2018-06-29

Warning

No description available

DM-DRP-7: Coordinate transformation tool provided for use with the Collimated Beam Projector.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-09a, T&SC-2100-2252

  • Due: 2018-07-12

  • Completed: 2018-07-05

Warning

No description available

DM-DRP-17: Simultaneous photometric and astrometric fitting to multiple exposures.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F, DM-AP-9

  • Successors: LDM-503-09a

  • Due: 2018-07-20

  • Completed: 2018-11-29

Warning

No description available

DM-DRP-9: Data reduction pipeline for the Auxiliary Telescope.#

Warning

No description available

DM-DRP-19: QA metrics are generated during pipeline execution.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F

  • Successors: LDM-503-11b

  • Due: 2018-11-29

  • Completed: 2018-12-13

Warning

No description available

DM-DRP-37: Artifact rejection and background matching during coadd construction.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F18F, DM-DRP-3

  • Successors: LSST-1580

  • Due: 2018-11-30

  • Completion pending

The Data Release Production science payload now includes code for generating final estimates of the sky background and artifact masks, as described in LDM-151 [Swinbank et al., 2020] §5.2.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-29: Moving point source model fitting now available.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F18F

  • Successors: LSST-1580

  • Due: 2018-11-30

  • Completed: 2022-12-31

The Data Release Production science payload now includes the capability of fitting moving a moving point source model that includes parallax and proper motion as free parameters to all objects, as described in LDM-151 [Swinbank et al., 2020] §6.7.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-2: Basic visualization and quality assessment tools operational on HSC-scale data volumes.#

Warning

No description available

DM-DRP-11: Pipelines code provides supports for database ingestion of results.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F

  • Successors: F20F, DM-DAX-5

  • Due: 2019-02-28

  • Completed: 2019-12-31

The Data Release Production science payload now generates results which are compliant with the LSST Science Data Model, and which are suitable for database ingest.

DM-DRP-20: Refined set of LSST calibration products.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DRP-4

  • Successors: LSST-1580, SITCOM-120

  • Due: 2019-11-27

  • Completed: 2021-07-29

The Calibration Products Production science payload is now capable of generating all calibration products which are required to enter commissioning.

DM-DRP-8: Calibration product generation for the Auxiliary Telescope.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F, COMC-060200-10000

  • Successors: LSST-1580

  • Due: 2020-01-21

  • Completed: 2021-07-31

The Calibration Products Production science payload is now capable of generating those products which are required to calibrate the Auxiliary Telescope. This represent an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-22: Template generation integrated with Data Release Production pipelines.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F20S

  • Successors: LDM-503-13a

  • Due: 2020-06-01

  • Completed: 2021-03-31

The Data Release Production science payload is now capable of generating templates for use in image differencing, as described in LDM-151 [Swinbank et al., 2020] §§5.2.3 & 5.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-12: Background estimation over the full visit.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DAX-8

  • Successors: LSST-1580

  • Due: 2020-11-30

  • Completed: 2020-06-30

Warning

No description available

DM-DRP-13: PSF estimation over the full visit.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DAX-8

  • Successors: LSST-1580, DM-DRP-15

  • Due: 2020-11-30

  • Completed: 2023-02-28

The Data Release Production science payload now includes code for estimating the PSF over the full focal plane, as described in LDM-151 [Swinbank et al., 2020] §6.11.3. This does not require that the PSF model be “physically motivated” (ie, decomposed into atmospheric and optical components). Further, this represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-15: All varieties of coadd required for object detection and characterization are now produced during normal pipeline operation (although not necessarily at the ultimately required level of fidelity).#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DRP-13

  • Successors: LDM-503-11b

  • Due: 2020-11-30

  • Completed: 2018-11-30

Warning

No description available

DM-DRP-21: Integrated image characterization pipeline for Data Release Production.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-AP-9, DM-DAX-8

  • Successors: LSST-1580

  • Due: 2020-11-30

  • Completed: 2021-10-29

The Data Release Production science payload now includes an integrated image characterization pipeline, as described in LDM-151 [Swinbank et al., 2020] §5.1, including iterative refinement of the image characterization. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-30: Forced photometry is now performed on individual processed visit images during data releases.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F20F

  • Successors: LDM-503-13a

  • Due: 2020-11-30

  • Completed: 2021-02-01

The Data Release Production science payload now includes the capability of performing “forced” photometry at the positions of known Objects on Processed Visit Images, as described in LDM-151 [Swinbank et al., 2020] §5.5.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-34: Selection maps are generated during data releases.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F20F

  • Successors: LDM-503-13a

  • Due: 2020-11-30

  • Completed: 2021-05-24

The Data Release Production science payload now includes the capability of generating multi-scale maps that describe LSST’s depth and efficiency at detecting different classes of object as part of a data release, as described in LDM-151 [Swinbank et al., 2020] §5.6.3. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-23: Atmospheric characterization based on data from the Auxiliary Telescope now available.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F21S, DM-DRP-9, COMC-060200-10000

  • Successors: LSST-1580

  • Due: 2021-06-01

  • Completed: 2021-08-31

The Calibration Products Production science payload is now capable of producing an atmospheric transmission model based on observations from the Auxiliary Telescope, as described in LDM-151 [Swinbank et al., 2020] §4.6.1. This milestone does not presume that the Auxiliary Telescope itself is available for testing: it is satisfied by the availability of a pipeline that can process appropriate simulated or precursor data. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-26: Overlap resolution at tract & patch boundaries.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DRP-27

  • Successors: LDM-503-13a

  • Due: 2021-08-31

  • Completed: 2021-02-01

The Data Release Production science payload is now capable of finalizing the definition of Objects which fall within the overlap regions at the edges of tracts and patches, as described in LDM-151 [Swinbank et al., 2020] §5.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects.#

The Data Release Production science payload now includes the capability to build Objects by cross-matching the Source and DIASource tables, as described in LDM-151 [Swinbank et al., 2020] §5.3.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-28: Difference images are now a first-class data product during data release processing.#

The Data Release Production science payload now includes the capability to generate difference images, and to perform source detection and characterization on those images, as described in LDM-151 [Swinbank et al., 2020] §5.2.5. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-35: Simultaneous measurement across a suite of coadds representing different bandpasses, epocs, and flavors.#

  • WBS: 02C.04

  • Level: 3

  • Successors: DM-DRP-31

  • Due: 2021-08-31

  • Completed: 2021-10-29

The Data Release Production science payload now includes the capability to perform consistent measurement of all Objects in a patch across multiple coadds, as described in LDM-151 [Swinbank et al., 2020] §5.3.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-24: Physically motivated PSF model, including separate characterization of contributions from the atmosphere and the telescope system.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: LSST-1520

  • Successors: LSST-1580

  • Due: 2024-11-01

  • Completion pending

Per LDM-151 [Swinbank et al., 2020] §6.11.3, the PSF model used by the Data Release Production science payload is now decomposed into separate contributions from the telescope optics and the atmosphere.

DM-DRP-31: A photometric redshift is now provided for each object.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: LSST-1520, DM-DRP-35

  • Successors: DM-DRP-36

  • Due: 2024-11-01

  • Completion pending

The Data Release Production science payload now includes the capability of calculating photometric redshifts as part of a data release, as (briefly) described in LDM-151 [Swinbank et al., 2020] §5.6.5. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-36: Solar System Processing integrated to data release processing.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-AP-17, DM-DRP-31

  • Successors: LSST-1580

  • Due: 2024-11-01

  • Completion pending

The Data Release Production science payload now generates SSObjects, by delegating to the Solar System Processing (SSP) pipeline being delivered by the Alert Production team, as described in LDM-151 [Swinbank et al., 2020] §5.6.1. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

02C.05: Science User Interface and Tools#

_images/graph_02C.05.png

Fig. 5 Relationships between milestones in WBS 02C.05 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-578: Propose a workspace structure for discussion.#

  • WBS: 02C.05.05

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-574: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.01

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-575: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.02

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-576: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.03

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-577: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.04

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-519: Research the development framework for SUI#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2014-10-14

  • Completed: 2014-10-14

Warning

No description available

DLP-520: Access Qserv for catalog search in SUI prototype#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2014-12-02

  • Completed: 2014-12-02

Warning

No description available

DLP-521: Define the data access APIs with database team#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2015-03-11

  • Completed: 2015-03-11

Warning

No description available

DLP-518: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.05

  • Level: 3

  • Due: 2015-03-20

  • Completed: 2015-03-20

Warning

No description available

DLP-564: Able to use some data-access APIs developed by database team#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2015-07-28

  • Completed: 2015-11-17

Warning

No description available

DLP-256: First draft definition of workspace#

  • WBS: 02C.05.05

  • Level: 3

  • Due: 2015-07-31

  • Completed: 2015-11-17

Warning

No description available

DLP-218: Provide API for image display in Firefly#

  • WBS: 02C.05.02

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-11-17

Warning

No description available

DLP-216: Provide basic APIs to use Firefly visualization components#

  • WBS: 02C.05.02

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2016-11-02

Warning

No description available

DLP-872: Convert major portion of GWT in Firefly to pure JavaScript#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2016-02-24

  • Completed: 2016-04-08

Warning

No description available

DM-SUIT-3: Time series analysis tool for WISE data#

  • WBS: 02C.05

  • Level: 3

  • Successors: LDM-503-01

  • Due: 2016-09-30

  • Completed: 2017-11-30

Warning

No description available

DLP-871: Firefly server side code refactor#

  • WBS: 02C.05.01

  • Level: 3

  • Predecessors: DMTC-0000-0030

  • Successors: S17S

  • Due: 2016-11-30

  • Completed: 2017-07-31

Warning

No description available

DM-SUIT-2: Search WISE coaded data single exposure images in PDAC (the images are from IRSA at IPAC, not NCSA)#

  • WBS: 02C.05

  • Level: 3

  • Predecessors: DM-DAX-2

  • Successors: LDM-503-01

  • Due: 2017-07-31

  • Completed: 2017-11-30

Warning

No description available

DM-SUIT-1: Search and display WISE sources (objects) in PDAC#

  • WBS: 02C.05

  • Level: 3

  • Predecessors: DM-DAX-1

  • Successors: LDM-503-01

  • Due: 2017-11-30

  • Completed: 2017-11-30

Warning

No description available

DM-SUIT-4: Multiple data traces in chart space#

  • WBS: 02C.05

  • Level: 3

  • Successors: LDM-503-01

  • Due: 2017-11-30

  • Completed: 2017-11-30

Warning

No description available

DLP-207: Extra LSST query capabilities in web UI#

  • WBS: 02C.05.01

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-08-01

Warning

No description available

DLP-205: LSST web UI search external catalog#

  • WBS: 02C.05.01

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-08-01

Warning

No description available

DLP-222: Provide time-domain data view#

  • WBS: 02C.05.02

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-08-01

Warning

No description available

DM-SUIT-8: SUIT portal integrated with workspace#

  • WBS: 02C.05

  • Level: 3

  • Successors: DM-SUIT-10

  • Due: 2018-11-30

  • Completed: 2020-05-29

Portal is integrated with LSST-provided WebDAV or VOSpace service exposing the User File Workspace. Portal provides functions for saving image and catalog query results to the logged-in user’s workspace, with the user’s identity correctly applied. Portal provides functions for accessing data in the user’s workspace and displaying it using the Portal’s image and table visualization tools, as well as using data from the workspace in other Portal contexts where a file can be uploaded, e.g., to perform a multi-object query. (NB: Still achievable post-DM-10, though only with WebDAV.).

DM-SUIT-5: Search and display processed HSC data#

Portal provides access to a collection of public-release HSC data processed by LSST DM, including calibrated single-epoch images, coadds, and catalogs (including at least an Object-like catalog). Access is obtained via IVOA interfaces including TAP for catalogs and ObsTAP for image metadata. (NB: Still achievable post-DM-10).

DM-SUIT-7: Mapping between SUIT systems & NCSA auth system#

  • WBS: 02C.05

  • Level: 3

  • Predecessors: S19F

  • Successors: DM-SUIT-10

  • Due: 2019-05-31

  • Completed: 2019-05-31

Portal is integrated with NCSA-provided federated-identity authentication and authorization system: the Portal recognizes that a user is logged in and displays that state, and passes the user’s identity along to LSST data services that it accesses. (NB: Still achievable post-DM-10) Claimed in May, based on installation on lsst-lsp-int.

DM-SUIT-10: SUIT deployment procedure#

Portal software is adapted for deployment in Kubernetes. The set of container images required for deployment is documented in an LTD-style manner. Configuration files for Kubernetes are provided in Github and documented to a level that permits deployments to be performed by non-IPAC staff. (NB: Still achievable post-DM-10).

02C.06: Science Data Archive and Application Services#

_images/graph_02C.06.png

Fig. 6 Relationships between milestones in WBS 02C.06 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-836: New logging for DM#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2014-08-01

  • Completed: 2015-11-17

Warning

No description available

DLP-835: New xrootd client#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: DLP-834

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-834: Automated Testbed for Qserv#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: LSST-1440

  • Successors: DLP-835

  • Due: 2014-08-01

  • Completed: 2014-08-01

Warning

No description available

DLP-818: CSS#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2014-11-25

  • Completed: 2015-11-17

Warning

No description available

DLP-801: Qserv uses C++-based Geometry Code#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2014-11-25

  • Completed: 2015-11-17

Warning

No description available

DLP-817: Multi-node Multi-query Integration Testing Harness#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2015-05-29

  • Completed: 2015-11-17

Warning

No description available

DLP-800: Data Ingest into MySQL#

  • WBS: 02C.06.01

  • Level: 3

  • Due: 2015-05-29

  • Completed: 2015-11-17

Warning

No description available

DLP-25: Prototype Web Service for SUI#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-09-01

Warning

No description available

DLP-805: Distributed Database and Table Management#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2015-10-29

  • Completed: 2015-10-29

Warning

No description available

DLP-635: Qserv Query Cancellation#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2015-11-25

  • Completed: 2016-01-12

Warning

No description available

DLP-649: Shared Scans#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2016-02-24

  • Completed: 2017-05-26

Warning

No description available

DLP-802: Alert Production Database (APDB) design#

  • WBS: 02C.06.01

  • Level: 3

  • Due: 2016-11-30

  • Completed: 2021-05-31

Warning

No description available

DLP-472: Qserv Data Distribution#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: DMTC-0000-3000, S17F

  • Due: 2017-05-31

  • Completed: 2018-06-29

Warning

No description available

DM-DAX-2: Query service supporting IVOA TAP protocol, w/ support for asynchronous queries#

An IVOA TAP service supporting both synchronous and asynchronous query models, integrated with the LSP authentication architecture, and able to access catalog data hosted in a downstream in Qserv instance.

DM-DAX-1: WISE data ingest to PDAC#

Warning

No description available

DM-DAX-6: Prototype level 1 database#

Warning

No description available

DLP-473: Async Queries in Qserv#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: S17F, F17F

  • Successors: DLP-725

  • Due: 2017-11-30

  • Completed: 2017-11-01

Warning

No description available

DLP-725: Working Prototype of AP/L1 Database#

  • WBS: 02C.06.01

  • Level: 3

  • Predecessors: F17F, DLP-473

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-11-01

Warning

No description available

DLP-837: AP-ready Butler Framework#

  • WBS: 02C.06.02

  • Level: 3

  • Successors: F20F

  • Due: 2017-12-01

  • Completed: 2019-08-30

Warning

No description available

DM-DAX-11: Transformed EFD schema#

  • WBS: 02C.06

  • Level: 3

  • Successors: LSST-1580

  • Due: 2018-03-30

  • Completed: 2018-05-31

Warning

No description available

DM-DAX-10: Per-object gather support in Butler#

  • WBS: 02C.06

  • Level: 3

  • Predecessors: F18F

  • Successors: DM-DAX-5

  • Due: 2018-11-30

  • Completed: 2019-11-29

Demonstrate Butler configuration able to gather all data pertaining to a particular object, such as retrieving cutouts from all warps at a given box on a skymap.

DM-DAX-5: Database ingest in support of HSC reprocessing (ie, large catalog ingest)#

Existing Qserv ingest tooling is highly manual and poorly parallelized. Develop new tooling to address these issues and support ingest campaigns at the scale needed for DRP.

DM-DAX-7: Generation 3 Butler, running on a POSIX file-system provided by NCSA#

  • WBS: 02C.06

  • Level: 3

  • Predecessors: DM-DAX-12

  • Successors: DM-DAX-13

  • Due: 2020-11-30

  • Completed: 2020-06-30

Given a stood-up data backbone instance, demonstrate instantiation/configuration of a Butler Gen3 instance capable of retrieving images from that backbone instance. (As currently conceived, the data backbone is either a GPFS filesystem or object store, in combination with metadata that is a superset of that maintained in a Butler Gen3 registry. Depending on particular technologies and implementation strategies chosen for the data backbone, this may be either a “simple matter of configuration” of existing Butler connectors, or may require some additional custom connectors to be written.).

DM-DAX-8: Scatter-gather support in PipelineTask#

Demonstrate a functional PipelineTask with scatter/gather functionality, such as Jointcal, SkyCorrect, or FGCM.

DM-DAX-12: Generation 3 middleware at feature parity with Generation 2#

Generation 3 middleware, consisting of Butler updates and the PipelineTask framework, are now able to support existing science payloads with no loss of functionality over previous versions. That is, Gen 3 may not be feature complete, but Gen 2 can be retired with no functionality regressions across the subsystem.

DM-DAX-14: Provenance system review completed#

  • WBS: 02C.06

  • Level: 3

  • Predecessors: F20F

  • Successors: DM-DAX-9

  • Due: 2020-11-30

  • Completed: 2022-01-31

Warning

No description available

DM-DAX-13: Generation 2 middleware retired#

The Generation 2 middleware is deprecated, and may be removed from the codebase. No current science payload and no current execution service makes use of it.

DM-DAX-15: DAX Construction Complete#

  • WBS: 02C.06

  • Level: 3

  • Successors: LSST-1580

  • Due: 2024-01-11

  • Completion pending

Warning

No description available

DM-DAX-9: Butler Provenance System#

  • WBS: 02C.06

  • Level: 3

  • Predecessors: LSST-1520, DM-DAX-12, DM-DAX-14, COMC-060200-20930

  • Successors: LDM-503-17a

  • Due: 2024-11-01

  • Completion pending

Warning

No description available

02C.07: LSST Data Facility#

_images/graph_02C.07.png

Fig. 7 Relationships between milestones in WBS 02C.07 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-359: Improve fault tolerance in AP simulator#

  • WBS: 02C.07.01

  • Level: 3

  • Due: 2014-11-19

  • Completed: 2014-11-19

Warning

No description available

DLP-360: Prepare for OCS Middleware integration with AP DMCS#

  • WBS: 02C.07.01

  • Level: 3

  • Due: 2014-11-21

  • Completed: 2014-11-21

Warning

No description available

DLP-422: Evaluate OpenStack as a development environment#

  • WBS: 02C.07.03

  • Level: 3

  • Due: 2015-01-29

  • Completed: 2015-01-29

Warning

No description available

DLP-450: Support lsst-dev for developers#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2015-02-02

  • Completed: 2015-02-02

Warning

No description available

DLP-449: Upgrade storage server hardware#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2015-02-02

  • Completed: 2016-07-15

Warning

No description available

DLP-400: Refactor basic event services#

  • WBS: 02C.07.02

  • Level: 3

  • Successors: DLP-556

  • Due: 2015-05-28

  • Completed: 2015-01-26

Warning

No description available

DLP-441: Base Center requirements document#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2015-05-29

  • Completed: 2015-05-29

Warning

No description available

DLP-451: Deploy OpenStack capability#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2015-07-30

  • Completed: 2015-07-30

Warning

No description available

DLP-356: Set up Joint Coordination Council (JCC) for coordination with satellite processing center#

  • WBS: 02C.07.00

  • Level: 3

  • Due: 2015-08-26

  • Completed: 2015-08-26

Warning

No description available

DLP-361: Build prototype AP DMCS, including events#

  • WBS: 02C.07.01

  • Level: 3

  • Due: 2015-08-26

  • Completed: 2015-08-26

Warning

No description available

DLP-355: Establish MOA with CC-IN2P3 satellite processing center#

  • WBS: 02C.07.00

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-08-31

Warning

No description available

DLP-404: Identify candidate security packages#

  • WBS: 02C.07.02

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2016-06-07

Warning

No description available

DLP-410: Implement Puppet for system configuration management#

  • WBS: 02C.07.02

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-08-31

Warning

No description available

DLP-415: Evaluate iRODS, Datacat, and similar file system management technologies#

  • WBS: 02C.07.02

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-08-31

Warning

No description available

DLP-423: DevOps assessment#

  • WBS: 02C.07.03

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-08-31

Warning

No description available

DLP-430: Initial refinement of facility integration plan#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2015-08-31

  • Completed: 2015-08-31

Warning

No description available

DLP-357: Initial refinement of facility integration plan#

  • WBS: 02C.07.01

  • Level: 3

  • Successors: DLP-358

  • Due: 2015-09-29

  • Completed: 2016-06-15

Warning

No description available

DLP-389: Implement advanced logging functionality, including event-based logging#

  • WBS: 02C.07.01

  • Level: 3

  • Successors: DLP-566

  • Due: 2015-09-29

  • Completed: 2015-09-29

Warning

No description available

DLP-452: Support datasets on basic file system infrastructure with apropos policies#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2016-01-22

  • Completed: 2016-07-15

Warning

No description available

DLP-411: Monitor OpenStack#

  • WBS: 02C.07.02

  • Level: 3

  • Successors: DLP-412

  • Due: 2016-02-05

  • Completed: 2016-06-07

Warning

No description available

DLP-850: Prototype authentication/authorization system complete#

  • WBS: 02C.07.02

  • Level: 3

  • Successors: S17S

  • Due: 2016-02-22

  • Completed: 2016-06-07

Warning

No description available

DLP-405: Investigate CAS integration with Internet2 AAA tools#

  • WBS: 02C.07.02

  • Level: 3

  • Successors: S17S

  • Due: 2016-02-22

  • Completed: 2016-07-15

Warning

No description available

DLP-469: Coordination with NCSA/CC-IN2P3 JCC#

  • WBS: 02C.07.00

  • Level: 3

  • Successors: DLP-864

  • Due: 2016-02-24

  • Completed: 2016-06-07

Warning

No description available

DLP-358: Refinement of facility integration plan and traceability of science and operational requirements#

  • WBS: 02C.07.01

  • Level: 3

  • Predecessors: DLP-357

  • Due: 2016-02-24

  • Completed: 2016-04-08

Warning

No description available

DLP-364: Build prototype production sequence for the DMCS#

  • WBS: 02C.07.01

  • Level: 3

  • Successors: DLP-566

  • Due: 2016-02-24

  • Completed: 2016-06-07

Warning

No description available

DLP-471: Refresh sizing model#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2016-02-24

  • Completed: 2016-06-07

Warning

No description available

DLP-431: Refinement of facility integration plan and traceability of science and operational requirements#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2016-02-24

  • Completed: 2016-07-15

Warning

No description available

DLP-424: Integration of DevOps to OpenStack#

  • WBS: 02C.07.03

  • Level: 3

  • Due: 2016-02-29

  • Completed: 2016-07-15

Warning

No description available

DLP-847: Meet camera DAQ/OCS milestones for FY16#

  • WBS: 02C.07.01

  • Level: 3

  • Predecessors: DMTC-0000-0030

  • Successors: F17F

  • Due: 2016-11-30

  • Completed: 2017-09-01

Warning

No description available

DLP-366: Mid-scale processing of eligible camera data#

  • WBS: 02C.07.01

  • Level: 3

  • Predecessors: DMTC-0000-0030

  • Successors: F17F

  • Due: 2016-11-30

  • Completed: 2017-11-01

Warning

No description available

DLP-412: Monitor development cluster#

Warning

No description available

DM-NCSA-1: Provide regular reprocessing service for HSC data#

Warning

No description available

DM-NCSA-2: Provide access to results of regular reprocessing (NB the form this takes depends upon available DAX functionality)#

Warning

No description available

DM-NCSA-28: IIP control code#

Warning

No description available

DLP-864: Coordination with NCSA/CC-IN2P3 JCC#

  • WBS: 02C.07.00

  • Level: 3

  • Predecessors: DLP-469

  • Successors: S18F

  • Due: 2017-11-30

  • Completed: 2017-11-01

Warning

No description available

DLP-368: Initial AP DMCS#

  • WBS: 02C.07.01

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-11-01

Warning

No description available

DLP-459: Qserv capability to hold data reduced at NCSA#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-07-25

Warning

No description available

DLP-458: Ingest of bulk data from real cameras to ComCam scale#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-10-18

Warning

No description available

DLP-457: Support runs on real cameras data and ImSim data#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-11-01

Warning

No description available

DLP-442: Procure infrastructure for Base Center to meet FY18 goals#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Due: 2017-11-30

  • Completed: 2018-04-30

Warning

No description available

DLP-433: Procure infrastructure for Archive Center to FY18 scale#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Due: 2017-11-30

  • Completed: 2018-04-30

Warning

No description available

DMTC-4500-0010: Start NCSA S18#

  • WBS: 02C.07

  • Level: 3

  • Due: 2017-12-01

  • Completed: 2017-12-01

Time-based milestone marking the start of NCSA’s Spring 2018 development cycle.

DM-NCSA-27: Deliver header service code#

Warning

No description available

DM-NCSA-29: Deliver IIP image client code#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: DM-NCSA-28

  • Successors: LSST-1200

  • Due: 2017-12-29

  • Completed: 2018-03-31

Warning

No description available

DM-NCSA-6: Ability to transfer files originating from Tucson to NCSA and ingest files at NCSA, including metadata and provenance#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-04b

  • Due: 2018-03-05

  • Completed: 2018-10-31

Warning

No description available

DM-NCSA-5: Level 1 archiving system able to acquire pixel data from the Aux Tel DAQ, header metadata via OCS, assemble FITS image,#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-04b

  • Due: 2018-03-30

  • Completed: 2018-05-31

Warning

No description available

DM-NCSA-7: Capability to paint displays for Tucson and NCSA#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-04b

  • Due: 2018-03-30

  • Completed: 2018-06-29

Warning

No description available

DM-NCSA-9: Test instance of alert distribution hosting service and L1 database in Development & Integration Enclave#

Hosting an instance of the alert distribution service delivered from University of Washington on our test containerized application management service platform, running feeds from the pipeline to the distribution services. Note: This was informally tied to LDM-503-05, which was de-scoped to a small demonstration of running an instance of Kafka at NCSA, so no L1 database was needed. We believe this was completed in August 2018.

DM-NCSA-10: Sustained archiving service that is OCS commandable#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-06

  • Due: 2018-09-25

  • Completed: 2019-05-31

The capability to use the existing OCS/CCS to command data acquisition and hand-off to a test Observatory Operations Data Service (OODS).

DM-NCSA-3: Provide database for metadata, provenance, location and demonstrate ingest at small scale#

Warning

No description available

DM-NCSA-4: Minimal support for the small operational schema including file metadata and provenance for every file, and record of in#

Warning

No description available

DM-NCSA-16: Perform ISR processing on ComCam-scale data.#

Process ComCam-scale data through the ISR pipeline to generate QC parameters used by the observatory.

A5090: Expected Arrival of ComCam Systems to NCSA#

  • WBS: 02C.07.09

  • Level: 4

  • Predecessors: T&SC-6500-8100

  • Due: 2019-05-08

  • Completed: 2019-04-30

Arrival at NCSA of hardware to support ComCam testing and integration.

DM-NCSA-13: Header Writing Service for Spectrograph use case#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-08

  • Due: 2019-05-14

  • Completed: 2019-05-31

The Header builder integrated into OCS and demonstrated. Header builder has identified sources of information for all modes of observing with the Spectrograph.

A4640: Ship ComCam Hardware to Tucson#

  • WBS: 02C.07.09

  • Level: 4

  • Due: 2019-06-20

  • Completed: 2019-06-27

Date for ComCam support systems to ship from NCSA to Tucson.

A2330: Ship AA System to Chile#

  • WBS: 02C.07.09

  • Level: 4

  • Due: 2019-06-28

  • Completed: 2019-06-28

Date for Authentication and Authorization hardware to ship from NCSA to Chile.

A4369.9: Ship Hardware to BDC#

  • WBS: 02C.07.09

  • Level: 4

  • Successors: A4670

  • Due: 2019-07-15

  • Completed: 2019-07-17

Date for Base Data Center hardware to ship from NCSA to Chile.

DM-NCSA-11: Verified acquisition of raw and crosstalk-corrected exposures at raft scale, incl. correct metadata#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: F18F, T&SC-5900-0400

  • Successors: DLP-552, LDM-503-06

  • Due: 2019-07-29

  • Completed: 2020-06-30

Exercise the handoff between the L1 archiving and data backbone endpoint, including a monitoring service that tracks files that have been transferred and received.

A4860: ComCam Systems in Tucson ready for NCSA#

  • WBS: 02C.07.09

  • Level: 4

  • Successors: LDM-503-09, A5860

  • Due: 2019-08-14

  • Completed: 2019-08-01

ComCam Test Stand ready for access by NCSA staff.

A4680: June 2019 Travel to Chile - Travel End#

  • WBS: 02C.07.09

  • Level: 4

  • Successors: LDM-503-09

  • Due: 2019-08-16

  • Completed: 2019-08-24

End of travel to Chile to install hardware and systems at the Base Data Center.

A4670: June 2019 Travel to Chile - Travel Start#

  • WBS: 02C.07.09

  • Level: 4

  • Predecessors: A4369.9

  • Due: 2019-08-17

  • Completed: 2019-08-17

Start of travel to Chile to install hardware and systems at the Base Data Center.

A4750: Summit AA Systems Installed at the Summit#

  • WBS: 02C.07.09

  • Level: 4

  • Due: 2019-08-29

  • Completed: 2019-08-29

Authorization and Authentication systems installed at the Summit.

DM-NCSA-20: ComCam Archiving Service#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: T&SC-5900-0400

  • Successors: LDM-503-11a

  • Due: 2019-09-06

  • Completed: 2020-06-30

Implement a Service that acquires raw data from a camera at raft scale and sends to NCSA for ingestion into the Data Backbone. This is a preliminary milestone to demonstrate initial capabilities.

A1300: Start Milestone: October 1st, 2019#

  • WBS: 02C.07.05

  • Level: 4

  • Due: 2019-10-01

  • Completed: 2019-10-01

Warning

No description available

A6150: Start Milestone: January 1st, 2020#

  • WBS: 02C.07

  • Level: 3

  • Due: 2020-01-03

  • Completed: 2020-01-03

Warning

No description available

DM-NCSA-14: Data Backbone endpoints in Chile for ingestion and access, distribution over WAN, ingest at NCSA into custodial file sto#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-08

  • Due: 2020-06-22

  • Completed: 2021-11-17

Provide a data Backbone endpoint in Chile, allowing distribution over a wide-area network and ingestion at NCSA.

DM-NCSA-15: Batch Processing Service for offline spectrograph data processing#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: DM-DAX-12

  • Successors: LDM-503-08

  • Due: 2020-11-30

  • Completed: 2020-05-29

Demonstrate the ability to process offline Spectrograph data using a science pipeline in the Batch Processing Service. Output(s) will be available in the Data Backbone.

DM-NCSA-22: Information in consolidated database available to QA portal#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: DM-DAX-12

  • Successors: LDM-503-11a

  • Due: 2020-11-30

  • Completed: 2020-08-31

Scientific and operational data products stored in the consolidated database during ComCam data processing are available for QA assessment of ComCam data through normal analysis access environments.

DM-NCSA-21: L1 Offline Processing Service, Raft Scale, ComCam#

  • WBS: 02C.07

  • Level: 3

  • Successors: LSST-1580

  • Due: 2024-11-01

  • Completed: 2023-05-31

Demonstrated running of processing campaigns in offline mode needed for the ComCam phase of commissioning to debug the instrument. Likely includes calibration, ISR, and other pipelines.

02C.08: International Communications and Base Site#

_images/graph_02C.08.png

Fig. 8 Relationships between milestones in WBS 02C.08 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DM-NET-1: Rubin Network Re-engineering#

  • WBS: 02C.08.02

  • Level: 2

  • Predecessors: LSST-1440

  • Due: 2015-06-11

  • Completed: 2015-08-05

Warning

No description available

DLP-42: La Serena to Santiago Path Design Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2015-09-28

  • Completed: 2015-04-30

Warning

No description available

DMTC-8100-2010: First Data passed on 100G ring without errors - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2015-10-01

  • Completed: 2015-10-01

Warning

No description available

DMTC-7400-0815: La Serena-Gatehouse path design Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2015-12-15

  • Completed: 2015-12-25

Warning

No description available

DLP-43: Fiber Delivery Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8000-0760

  • Due: 2015-12-16

  • Completed: 2017-02-28

Warning

No description available

DMTC-7400-2010: Gatehouse to La Serena path design Ready - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-41

  • Due: 2015-12-16

  • Completed: 2015-12-16

Warning

No description available

DLP-41: Mountain - Base Fiber Path Ready#

Warning

No description available

DMTC-7400-2170: Gatehouse to Summit contract signed; Design Ready - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-41

  • Due: 2016-01-15

  • Completed: 2016-01-15

Warning

No description available

DMTC-7400-2050: Fiber delivery - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2016-03-15

  • Completed: 2016-03-31

Warning

No description available

DLP-63: Vendor selection Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2290

  • Due: 2016-06-01

  • Completed: 2016-03-31

Warning

No description available

DMTC-7400-2290: Final Selection of equipment vendor - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-63

  • Due: 2016-06-01

  • Completed: 2016-10-01

Warning

No description available

DMTC-8100-2050: 100G ring operating at 95.0% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2016-09-28

  • Completed: 2016-10-01

Warning

No description available

DMTC-8100-2250: FIU-CIARA Management Contract Delivery #2 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2016-09-30

  • Completed: 2016-09-30

Warning

No description available

DMTC-8000-0806: 3T Fiber Installation Certification and Acceptance Test - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DLP-45

  • Due: 2017-01-02

  • Completed: 2017-03-15

Warning

No description available

DLP-37: Summits-Gatehouse Fiber Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2210

  • Due: 2017-02-28

  • Completed: 2017-05-19

Warning

No description available

DMTC-7400-2210: Fiber delivery by Telefonica - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-37

  • Due: 2017-02-28

  • Completed: 2017-02-28

Warning

No description available

DMTC-8200-0480: VLAN design finalized Miami-NCSA - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2017-04-24

  • Completed: 2017-03-23

Warning

No description available

DMTC-7400-2250: End of commissioning and Acceptance - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-40

  • Due: 2017-06-01

  • Completed: 2017-06-30

Warning

No description available

DMTC-8100-2474: Agreement with Monet Cable Partner signed Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2017-06-30

  • Completed: 2017-04-28

Warning

No description available

DMTC-7400-2420: REUNA La Serena - Santiago DWDM equipment - Complete#

Warning

No description available

DMTC-7400-2090: Report on functional fiber connections, including AURA equipment - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2017-08-18

  • Completed: 2017-12-01

Warning

No description available

DMTC-7400-2330: LSST and AURA summit to base DWDM equipment - Complete#

Warning

No description available

DMTC-8100-2290: FIU-CIARA Management Contract Delivery #3 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2017-09-29

  • Completed: 2017-09-01

Warning

No description available

DMTC-7400-2480: AURA DWDM equipment in operation - Complete#

Warning

No description available

DMTC-8000-0818: Acceptance Test with Equipment Installed - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2480

  • Due: 2017-11-30

  • Completed: 2017-12-01

Warning

No description available

DMTC-7400-2510: REUNA La Serena - Santiago Optical Channel in operation - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2480

  • Due: 2017-11-30

  • Completed: 2017-11-01

Warning

No description available

DMTC-7400-2130: End of commissioning. Acceptance of fiber connection - Complete#

Warning

No description available

DLP-44: La Serena to Santiago with test traffic AURA DWDM LSST Fiber#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8000-0826

  • Due: 2018-03-02

  • Completed: 2017-12-31

Warning

No description available

DMTC-8000-0826: Intermediate traffic testing and report - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-44

  • Due: 2018-03-02

  • Completed: 2018-03-02

Warning

No description available

DLP-40: Summits-Gatehouse Initial Capability Ready#

Warning

No description available

DLP-65: Mountain - Base LSST and AURA DWM End Node Installation and Test Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2330

  • Due: 2018-03-14

  • Completed: 2018-03-31

Warning

No description available

DM-NET-2: Mountain - Base Network Functional 2 x 100 Gbps#

  • WBS: 02C.08.01

  • Level: 1

  • Predecessors: DLP-559

  • Successors: LDM-503-08, COMC-0620-2050

  • Due: 2018-03-27

  • Completed: 2018-04-02

Warning

No description available

DMTC-8100-2490: First light test report - Complete  Contract#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2018-04-02

  • Completed: 2018-06-01

Warning

No description available

DMTC-8000-0836: La Serena to Santiago with live traffic and uptime over LSST Lambda Ready - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DLP-61

  • Due: 2018-05-11

  • Completed: 2019-07-23

Warning

No description available

DMTC-8100-2090: 100G ring operating at 97.0% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-36

  • Due: 2018-06-29

  • Completed: 2018-06-29

Warning

No description available

DMTC-8200-0510: ESNet Request - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-70, DLP-69

  • Due: 2018-09-26

  • Completed: 2018-09-26

Warning

No description available

DM-NET-3: Initial Network Ready (Summit)#

  • WBS: 02C.08.03

  • Level: 2

  • Predecessors: T&SC-2600-1545

  • Successors: LDM-503-08

  • Due: 2018-09-28

  • Completed: 2018-03-05

Warning

No description available

DM-NET-6: Summit LAN installed#

  • WBS: 02C.08.00

  • Level: 3

  • Predecessors: T&SC-2600-1545

  • Successors: LDM-503-08

  • Due: 2018-09-28

  • Completed: 2018-04-02

Warning

No description available

DMTC-8100-2330: FIU-CIARA Management Contract Delivery #4 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2018-10-01

  • Completed: 2018-10-31

Warning

No description available

DMTC-8100-2530: Data passing test report - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-33

  • Due: 2018-10-26

  • Completed: 2019-05-31

Contract Deliverable Report #4 Spectrum Complete. Test of first data passing over Santiago - Boca Raton Spectrum link.

IT-716-M: Base Center Beneficial Occupancy#

  • WBS: 02C.08.01

  • Level: 4

  • Predecessors: T&SC-0400-1600

  • Due: 2018-11-26

  • Completed: 2019-03-18

Warning

No description available

IT-718-M: Base Center Full Occupancy#

  • WBS: 02C.08.01

  • Level: 4

  • Due: 2019-02-12

  • Completed: 2019-06-30

Externally linked to TS milestone for full occupancy of Base Data Center, including all construction, utilities, access and dust control.

DMTC-8000-0870: Report on functional fiber, including uptime, 1st year functional milestone - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2019-03-22

  • Completed: 2019-07-23

Contract Deliverable Report #8 Report on functional fiber, including uptime, 1st year functional milestone - Complete. Covers first year of operational service of La Serena - Santiago primary fiber, including availability and outage information.

DLP-36: Santiago - Miami Early Integration Ready#

Warning

No description available

DMTC-8100-2112: Miami - Boca Raton path diverse fiber - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-36

  • Due: 2019-04-22

  • Completed: 2019-02-01

Warning

No description available

DMTC-8100-2130: 100G ring operating at 98.0% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2019-05-08

  • Completed: 2019-02-13

Warning

No description available

IT-716-M2: Furniture installed in Offices, purchased cabling received#

  • WBS: 02C.08.01

  • Level: 4

  • Due: 2019-05-31

  • Completed: 2019-10-31

Externally linked to TS milestone for installation of furniture of Base Office Building and Base Data Center. All cabling to connect offices to Base LAN has been received.

DMTC-8000-0950: Technical Implementation Plan for 40Gbps diverse path - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2019-06-05

  • Completed: 2019-03-25

Contract Deliverable Report #10 Technical Implementation Plan for 40Gbps diverse path - Complete. Covers the fiber and equipment, ownership/service levels for La Serena - Santiago secondary path.

DM-NET-4: Base LAN installed#

  • WBS: 02C.08.03

  • Level: 2

  • Successors: LDM-503-11

  • Due: 2019-07-15

  • Completed: 2020-01-31

Installed the Base LAN equipment in the Base Date Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for ATS observing. (Level 2 equivalent of Level 3 milestone IT-725-M).

IT-725-M: Base Data Center Network Complete#

  • WBS: 02C.08.01

  • Level: 4

  • Successors: LDM-503-10

  • Due: 2019-07-29

  • Completed: 2021-08-20

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for ATS observing. (Level 3 equivalent of Level 2 milestone DM-NET-4).

DMTC-8100-2370: FIU-CIARA Management Contract Delivery #5 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2019-09-30

  • Completed: 2019-12-31

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DLP-70: Miami - Chicago Early Integration Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8200-0510

  • Due: 2019-10-01

  • Completed: 2019-11-01

Miami - Atlanta, and ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DLP-69: Boca Raton - Chicago Early Integration Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8200-0510

  • Due: 2019-10-01

  • Completed: 2019-11-14

Boca Raton - Atlanta and ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DMTC-8200-0600: ESNet Implementation Phase 1- Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2019-10-01

  • Completed: 2019-11-14

ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DMTC-8100-2170: 100G ring operating at 99.0% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2020-07-02

  • Completed: 2020-02-19

Contract Deliverable Report #6 100G managed ring Contract Delivery #6 Complete. Covers availability and outages.

DLP-33: Santiago - Boca Raton Full Integration Ready#

Spectrum link operational at 100 Gbps from Santiago - Boca Raton, including terrestrial segment from Santiago - Sao Paolo, and Monet submarine cable from Sao Paolo - Boca Raton.

DMTC-6800-1325: Network Verification Pre-Tests#

  • WBS: 02C.08.00

  • Level: 4

  • Successors: DLP-33

  • Due: 2020-09-14

  • Completed: 2020-02-28

Warning

No description available

DLP-46: Full Capacity over Diverse path Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8000-0990, S19F

  • Due: 2020-09-30

  • Completed: 2021-08-31

Both primary (100 Gbps) and secondary (40 - 100 Gbps) circuits from La Serena - Santiago are operational and tested with loads to full capacity.

DMTC-8100-2410: FIU-CIARA Management Contract Delivery #6 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2020-09-30

  • Completed: 2020-09-30

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DLP-465: Install networking for normal Base Center operations for full-camera scale#

  • WBS: 02C.08.01

  • Level: 3

  • Predecessors: F20F

  • Successors: LSST-1580, CAMM8350

  • Due: 2020-11-30

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for full camera observing.

DLP-468: Provide networking for formal Chilean DAC production at full-camera scale#

  • WBS: 02C.08.02

  • Level: 3

  • Predecessors: DLP-36, F20F

  • Successors: LDM-503-11

  • Due: 2020-11-30

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with full camera observing.

DLP-467: Install networking for full-camera-scale Chilean DAC serving on order 80 people#

  • WBS: 02C.08.02

  • Level: 3

  • Predecessors: F20F

  • Successors: LDM-503-11

  • Due: 2020-11-30

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with commissioning camera observing.

DM-NET-5: Base - Archive Network Functional 100 Gbps#

  • WBS: 02C.08

  • Level: 2

  • Predecessors: DLP-464, COMC-0021

  • Successors: CAMM8330, CAMM8340, COMC-0620-2240

  • Due: 2020-12-21

  • Completed: 2021-02-01

All segments of the Base - Archive Network, including La - Serena - Santiago - Sao Paolo - Miami/Boca Raton - Chicago - Champaign are operationally available at 100 Gbps line rate (effective rate is typically lower, ~ 80%).

DLP-464: Install networking for normal Base Center operations for ComCam scale#

  • WBS: 02C.08.01

  • Level: 3

  • Predecessors: F18F

  • Successors: CAMM8340, DM-NET-5

  • Due: 2020-12-21

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for commissioning camera observing.

DLP-466: Install networking for ComCam-scale Chilean DAC serving on order 20 people#

  • WBS: 02C.08.02

  • Level: 3

  • Predecessors: F18F

  • Successors: LSST-1580, CAMM8330, CAMM8340

  • Due: 2020-12-21

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with ATS observing.

DMTC-8100-2210: 100G ring operating at 99.99% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2021-06-30

  • Completed: 2021-07-30

Contract Deliverable Report #7 100G managed ring Contract Delivery #7 Complete. Covers availability and outages.

DMTC-8100-2450: FIU-CIARA Management Contract Delivery #7 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2021-09-01

  • Completed: 2021-09-30

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DMTC-6800-1350: Network End-to-End Verification Test#

  • WBS: 02C.08.00

  • Level: 4

  • Predecessors: SITCOM-130

  • Successors: LSST-1580

  • Due: 2025-03-06

  • Completion pending

Warning

No description available

02C.10: Science Quality and Reliability Engineering#

_images/graph_02C.10.png

Fig. 9 Relationships between milestones in WBS 02C.10 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-340: Operating system testing for public stack releases increased#

  • WBS: 02C.10.02

  • Level: 3

  • Successors: DLP-342, DLP-341

  • Due: 2015-08-31

  • Completed: 2016-04-08

Warning

No description available

DLP-334: SQuaRE release improved or changed CI system#

  • WBS: 02C.10.02

  • Level: 3

  • Successors: DLP-337

  • Due: 2015-08-31

  • Completed: 2016-04-08

Warning

No description available

DLP-342: Publish updated introductory software guide to developing with the stack#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: DLP-340

  • Due: 2015-12-29

  • Completed: 2016-04-08

Warning

No description available

DLP-341: Deploy stack user support and discussion forum#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: DLP-340

  • Successors: DLP-337

  • Due: 2015-12-29

  • Completed: 2016-04-08

Warning

No description available

DLP-337: Release new stack documentation infrastructure#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: DLP-341, DLP-334

  • Due: 2016-02-24

  • Completed: 2016-05-31

Warning

No description available

DLP-345: Prototype metrics collection system deployed#

  • WBS: 02C.10.02

  • Level: 3

  • Due: 2016-02-29

  • Completed: 2016-05-31

Warning

No description available

DLP-351: Intermediate Performance Metrics routinelly calculated and published#

  • WBS: 02C.10.02

  • Level: 3

  • Due: 2016-02-29

  • Completed: 2016-05-31

Warning

No description available

DLP-346: SQuaSH Prototype using existing architectural components deployed#

  • WBS: 02C.10.02

  • Level: 3

  • Due: 2016-08-31

  • Completed: 2017-07-25

Warning

No description available

DM-SQRE-1: Project internal Jupyter notebook service#

  • WBS: 02C.10.02

  • Level: 3

  • Successors: LDM-503-01, DM-SQRE-2

  • Due: 2017-08-31

  • Completed: 2017-11-01

Warning

No description available

DLP-344: Release SQuaSH-in-a-box automated deployment platform#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: F17F

  • Due: 2017-11-30

  • Completed: 2017-11-01

Warning

No description available

DLP-352: Metrics of Interest integrated in running stack or post-tests#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2017-11-30

  • Completed: 2017-11-01

Warning

No description available

DM-SQRE-2: Notebook service capabilities are suitable for the commissioning team to develop notebooks for its needs#

Members of the integration and commissioning team are able to develop notebooks in nublado that are capable of the interactive data exploration that is suitable to their needs to support ComCam commissioning. Data is available via a Butler to a mounted filesystem. They are able to work effectively, though in some cases their user experience can be improved. They are primarily trained in person by bootcamps and may require extensive support via Slack. The service may be intermittently unavailable due to engineering work.

DM-SQRE-3: Notebook service stable for commissioning and other internal project users#

Members of the integration and commissioning team as well as science pipelines developers use nublado to support their interactive data exploration needs relating to ComCam commissioning. They are able to work effectively though in some cases their user experience can be improved. Access to EFD data is easily available from notebooks. Data is available via Butler to a mounted filesystem. There is a curated library of notebooks modeling common tasks and a way to automatically verify that they are in working condition. Users can refer to documentation for answers to common problems. The service is stable and interruptions to the service are primarily notified well in advance and co-ordinated with the commissioning team to minimize disruption to the extent that the underlying infrastructure permits it.

DM-SQRE-6: IVOA Server-side Operations for Data Access API#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: F20F

  • Successors: LSST-1580, DM-SQRE-7

  • Due: 2020-11-30

  • Completed: 2022-02-28

Warning

No description available

DM-SQRE-7: IVOA-compliant image metadata services#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: F20F, DM-SQRE-6

  • Successors: LSST-1580

  • Due: 2020-11-30

  • Completed: 2022-06-30

Warning

No description available

DM-SQRE-4: Notebook service ready for verification & validation#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: DM-SQRE-3

  • Successors: LDM-503-14

  • Due: 2023-02-28

  • Completion pending

Science pipeline developers and members of the verification and validation science team use nublado to support their interactive data exploration needs relating to ComCam commissioning. They are able to work effectively though in some cases their user experience can be improved. This includes access to EFD, catalogue and pixel data. There are curated libraries of continuously integrated notebooks modeling common tasks and they is a reasonable workflow to allow Github-based collaboration on those notebooks. Users can refer to documentation for how to get started and for answers to common problems. The service is stable with minimal interruptions, but may not always scale well to large number of users.

DM-SQRE-5: Notebook service ready for general science users#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: LSST-1560

  • Successors: LSST-1580

  • Due: 2025-03-06

  • Completion pending

Members of the science collaborations or other non-project staff with access to data use nublado to support their interactive scientific data exploration. They are able to share data with each other using LSP sharing mechanisms and can access data without a direct filesystem mount (eg VO services, remote Butler etc). The service scales well to large number of users provided the infrastructure is sized for them. Users can primarily rely on documentation for support, and use a helpdesk system for questions. There is robust functionality for performing large distributed catalog operations and for visually exploring LSSTCam sized images.

02C.11: Security#

_images/graph_02C.11.png

Fig. 10 Relationships between milestones in WBS 02C.11 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

A9440: START: Data Security#

  • WBS: 02C.11

  • Level: Undefined

  • Predecessors: LSST-1440

  • Due: 2022-10-03

  • Completed: 2022-10-03

Warning

No description available

DMTC-021103-1085: COMP: Delayed Data Store Installed#

  • WBS: 02C.11.03

  • Level: 4

  • Due: 2023-11-28

  • Completed: 2024-08-30

Warning

No description available

DMTC-021102-1175: COMP: Firewalls and Physical Security#

  • WBS: 02C.11.02

  • Level: 4

  • Successors: LSST-1580

  • Due: 2023-12-06

  • Completion pending

Warning

No description available

DMTC-021101-1155: COMP: Encryption#

  • WBS: 02C.11.01

  • Level: 4

  • Predecessors: SUMMIT-2976

  • Successors: LSST-1580

  • Due: 2023-12-07

  • Completion pending

Warning

No description available

Bibliography#

[1]

Yusra AlSayyad. LDM-503-13a: Science Pipelines Fall 2020 Release Test Plan and Report. June 2021. Vera C. Rubin Observatory Data Management Test Report DMTR-302. URL: https://dmtr-302.lsst.io/.

[2] (1,2)

Eric C. Bellm and John D. Swinbank. LSST Level 1 System Test Specification. January 2019. Vera C. Rubin Observatory Data Management Controlled Document LDM-533. URL: https://ldm-533.lsst.io/.

[3]

Jim Bosch, Hsin-Fang Chiang, Michelle Gower, and John D. Swinbank. LSST Level 2 System Test Specification. January 2021. Vera C. Rubin Observatory Data Management Controlled Document LDM-534. URL: https://ldm-534.lsst.io/.

[4]

Michelle Butler. LDM-503-8 Spectrograph Data Acquisition Test Plan and Report. December 2019. Vera C. Rubin Observatory Data Management Test Report DMTR-121. URL: https://dmtr-121.lsst.io/.

[5]

Michelle Butler. LDM-503-8b (Small Scale CCOB Data Access) Test Plan and Report. July 2019. Vera C. Rubin Observatory Data Management Test Report DMTR-102. URL: https://dmtr-102.lsst.io/.

[6]

Michelle Butler. LDM-503-10: DAQ Validation Test Plan and Report. September 2020. Vera C. Rubin Observatory Data Management Test Report DMTR-181. URL: https://dmtr-181.lsst.io/.

[7]

Michelle Butler. LDM-503-10b: Large Scale CCOB Data Access Test Plan and Report. October 2020. Vera C. Rubin Observatory Data Management Test Report DMTR-182. URL: https://dmtr-182.lsst.io/.

[8]

Michelle Butler. LDM-503-6: ComCam Interface Verification Readiness Test Plan and Report. October 2020. Vera C. Rubin Observatory Data Management Test Report DMTR-171. URL: https://dmtr-171.lsst.io/.

[9] (1,2)

Michelle Butler, Jim Parsons, and Michelle Gower. LSST DM Raw Image Archiving Service Test Specification. January 2021. Vera C. Rubin Observatory Data Management Controlled Document LDM-538. URL: https://ldm-538.lsst.io/.

[10]

Jeffrey Carlin. LDM-GEN3: Gen 3 Butler Acceptance Testing Test Plan and Report. July 2022. Vera C. Rubin Observatory Data Management Test Report DMTR-271. URL: https://dmtr-271.lsst.io/.

[11]

G. P. Dubois-Felsmann, L.P. Guy, J. Carlin, K.S. Krughoff, C. Slater, and M. Wood-Vasey. LSST Science Platform Test Specification. August 2020. Vera C. Rubin Observatory Data Management Controlled Document LDM-540. URL: https://ldm-540.lsst.io/.

[12]

Gregory Dubois-Felsmann. LDM-503-10a: LSP with Authentication and TAP Test Plan and Report. January 2020. Vera C. Rubin Observatory Data Management Test Report DMTR-161. URL: https://dmtr-161.lsst.io/.

[13]

Gregory Dubois-Felsmann. LDM-503-14a: RSP redeployed on the Interim Data Facility (IDF), ready for DP0.1 Test Plan and Report. October 2021. Vera C. Rubin Observatory Data Management Test Report DMTR-301. URL: https://dmtr-301.lsst.io/.

[14]

Gregory Dubois-Felsmann. LDM-503-RSPa: RSP on the Interim Data Facility (IDF) is ready for DP0.2 Test Plan and Report. January 2023. Vera C. Rubin Observatory Data Management Test Report DMTR-381. URL: https://dmtr-381.lsst.io/.

[15]

Gregory Dubois-Felsmann, David Ciardi, Fritz Mueller, and Frossie Economou. Data Management LSST Science Platform Requirements. April 2019. Vera C. Rubin Observatory Data Management Controlled Document LDM-554. URL: https://ldm-554.lsst.io/.

[16]

Gregory Dubois-Felsmann, Frossie Economou, Kian-Tat Lim, Fritz Mueller, Stephen R. Pietrowicz, and Xiuqin Wu. Science Platform Design. January 2019. Vera C. Rubin Observatory Data Management Controlled Document LDM-542. URL: https://ldm-542.lsst.io/.

[17]

Robert Gruendl. LDM-503-11a: ComCam OPS Readiness Test Plan and Report. November 2020. Vera C. Rubin Observatory Data Management Test Report DMTR-231. URL: https://dmtr-231.lsst.io/.

[18] (1,2)

Margaret Johnson and Robert A. Gruendl. Proposed DM OPS Rehearsals. August 2020. Vera C. Rubin Observatory Data Management Controlled Document LDM-643. URL: https://ldm-643.lsst.io/.

[19]

M. Jurić, T. Axelrod, A.C. Becker, J. Becla, E. Bellm, J.F. Bosch, D. Ciardi, A.J. Connolly, G.P. Dubois-Felsmann, F. Economou, M. Freemon, M. Gelman, R. Gill, M. Graham, L.P. Guy, Ž. Ivezić, T. Jenness, J. Kantor, K.S. Krughoff, K-T Lim, R.H. Lupton, F. Mueller, D. Nidever, W. O'Mullane, M. Patterson, D. Petravick, D. Shaw, C. Slater, M. Strauss, J. Swinbank, J.A. Tyson, M. Wood-Vasey, and X. Wu. Data Products Definition Document. July 2023. Vera C. Rubin Observatory LSE-163. URL: https://lse-163.lsst.io/.

[20]

M. Jurić, D. Ciardi, G.P. Dubois-Felsmann, and L.P. Guy. LSST Science Platform Vision Document. July 2019. Vera C. Rubin Observatory LSE-319. URL: https://lse-319.lsst.io/.

[21]

Jeff Kantor. Vera C. Rubin Observatory Network Verification Document. May 2020. Vera C. Rubin Observatory Data Management Controlled Document LDM-732. URL: https://ldm-732.lsst.io/.

[22]

Simon Krughoff. DM-503-EFDa: EFD on Summit for M1/M3 Test Plan and Report. September 2021. Vera C. Rubin Observatory Data Management Test Report DMTR-291. URL: https://dmtr-291.lsst.io/.

[23]

K-T Lim. Header Service Interface. 2019. Vera C. Rubin Observatory LSE-400. URL: https://lse-400.lsst.io.

[24] (1,2)

K.-T. Lim, J. Bosch, G. Dubois-Felsmann, T. Jenness, J. Kantor, W. O'Mullane, D. Petravick, G. Comoretto, and the DM Leadership Team. Data Management System Design. August 2020. Vera C. Rubin Observatory Data Management Controlled Document LDM-148. URL: https://ldm-148.lsst.io/.

[25]

William O'Mullane. Data Preview 0: Definition and planning. February 2024. Vera C. Rubin Observatory Technical Note RTN-001. URL: https://rtn-001.lsst.io/.

[26] (1,2,3,4,5,6)

William O'Mullane, Frossie Economou, Tim Jenness, Andrew Loftus, and John D. Swinbank. Data Management Releases for Verification/Integration. August 2023. Vera C. Rubin Observatory Data Management Controlled Document LDM-564. URL: https://ldm-564.lsst.io/.

[27] (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31)

J.D. Swinbank, T. Axelrod, A.C. Becker, J. Becla, E. Bellm, J.F. Bosch, H. Chiang, D.R. Ciardi, A.J. Connolly, G.P. Dubois-Felsmann, F. Economou, M. Fisher-Levine, M. Graham, Ž. Ivezić, M. Jurić, T. Jenness, R.L. Jones, J. Kantor, S. Krughoff, K-T. Lim, R.H. Lupton, F. Mueller, D. Petravick, P.A. Price, D.J. Reiss, D. Shaw, C. Slater, M. Wood-Vasey, X. Wu, P. Yoachim, and _for the LSST Data Management_. Data Management Science Pipelines Design. November 2020. Vera C. Rubin Observatory Data Management Controlled Document LDM-151. URL: https://ldm-151.lsst.io/.

[28]

John Swinbank. LDM-503-07 (Camera Data Processing) Test Plan and Report. January 2019. Vera C. Rubin Observatory Data Management Test Report DMTR-112. URL: https://dmtr-112.lsst.io/.

[29]

John Swinbank. LDM-503-11b: Science Pipelines Fall 2019 Release Test Plan and Report. September 2020. Vera C. Rubin Observatory Data Management Test Report DMTR-192. URL: https://dmtr-192.lsst.io/.

[30]

John D. Swinbank. LDM-503-09a (Science Pipelines Fall 2018 Release) Test Plan and Report. April 2019. Vera C. Rubin Observatory Data Management Test Report DMTR-111. URL: https://dmtr-111.lsst.io/.