DMTN-158: DM Milestone Summary

  • John Swinbank, William O'Mullane

Latest Revision: 2022-08-13

Provenance

This document was generated based on the contents of the lsst-dm/milestones repository, version 891695a5, dated 2022-08-13. This corresponds to the status recorded in the project controls system for July 2022.

Notation

Throughout this document, the identifiers of completed milestones are set in italics; those of milestones which are still pending, in roman.

Summary

The DM Subsystem is currently tracking 377 milestones: 7 at Level 1, 83 at Level 2, 272 at Level 3, and 14 at Level 4. 1 have no level defined. Of these, 344 have been completed. Of the incomplete milestones, 6 are late relative to the baseline schedule, while the remainder are scheduled for the future.

_images/burndown.png

Figure 1 Milestone completion as a function of date.

Currently overdue milestones

  • DM-AP-10: Advanced single frame measurement pipeline for Alert Production. [Due 2020-06-01]
  • DM-AP-12: Difference imaging is now agnostic to the PSF of the template image. [Due 2021-08-31]
  • DM-DRP-13: PSF estimation over the full visit. [Due 2020-11-30]
  • DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects. [Due 2021-08-31]
  • DM-DRP-29: Moving point source model fitting now available. [Due 2018-11-30]
  • DM-DRP-37: Artifact rejection and background matching during coadd construction. [Due 2018-11-30]

Milestones by due date

Due in January 2024

  • LDM-503-17a: Test report: Final Pipelines Delivery
  • DM-AP-14: Alert filtering system available.
  • DM-AP-16: Full integration of the Alert Production system within the operational environment.
  • DM-DAX-15: DAX Construction Complete
  • DM-SQRE-5: Notebook service ready for general science users
  • DMTC-6800-1350: Network End-to-End Verification Test

Due in December 2023

  • No milestones due.

Due in November 2023

  • No milestones due.

Due in October 2023

Due in September 2023

  • DLP-526: Archive Center Integration Complete at SLAC
  • LDM-503-16: Ops rehearsal for data release processing #3
  • LDM-503-12a: LSSTcam On Sky Readiness
  • LDM-503-EFDc: EFD data is queriable through TAP in the Science Platform
  • DM-AP-15: Alert distribution system fully integrated.
  • DM-AP-17: Solar System Processing pipeline available.
  • DM-DRP-24: Physically motivated PSF model, including separate characterization of contributions from the atmosphere and the telescope system.
  • DM-DRP-31: A photometric redshift is now provided for each object.
  • DM-DRP-36: Solar System Processing integrated to data release processing.
  • DM-DAX-9: Butler Provenance System

Due in August 2023

  • No milestones due.

Due in July 2023

  • No milestones due.

Due in June 2023

  • No milestones due.

Due in May 2023

  • LDM-503-14: Test report: DM Readiness for Science Verification Surveys
  • LDM-503-15: Ops rehearsal for data release processing #2
  • LDM-503-16a: Test report: Pipelines Release Spring 2023

Due in April 2023

  • No milestones due.

Due in March 2023

  • No milestones due.

Due in February 2023

  • DM-NCSA-21: L1 Offline Processing Service, Raft Scale, ComCam
  • LDM-503-12: Ops rehearsal for commissioning #3
  • LDM-503-13: Ops rehearsal for data release processing #1 (ComCam data)
  • LDM-503-EFDa: Engineering Facility Database at the summit capturing and enabling access to all telemetry with M1/M3 active
  • LDM-503-EFDb: EFD/telemetry data replicated at the LDF, stored in InfluxDB. aggregated into parquet files
  • DM-DRP-20: Refined set of LSST calibration products.
  • DM-SQRE-4: Notebook service ready for verification & validation

Due in January 2023

  • No milestones due.

Due in December 2022

  • No milestones due.

Due in November 2022

  • No milestones due.

Due in October 2022

  • No milestones due.

Due in September 2022

  • LDM-503-RSPa: Science Platform ready for DP0.2 with Image Services

Due in August 2022

  • No milestones due.

Due in July 2022

  • No milestones due.

Due in June 2022

  • No milestones due.

Due in May 2022

  • S22F: S22 Complete
  • LDM-503-15a: Test report: Pipelines Release Spring 2022

Due in April 2022

  • No milestones due.

Due in March 2022

  • No milestones due.

Due in February 2022

  • No milestones due.

Due in January 2022

  • No milestones due.

Due in December 2021

  • S22S: S22 Start
  • DM-PORTAL: Resume development of LSP portal aspect
  • DM-CLOUD: Decide on final data facility design for DR1

Due in November 2021

  • F21F: F21 Complete

Due in October 2021

  • No milestones due.

Due in September 2021

Due in August 2021

  • LDM-503-13a: Test report: Pipelines Release Fall 2020
  • DM-AP-12: Difference imaging is now agnostic to the PSF of the template image.
  • DM-AP-13: Trailed source measurement on difference images.
  • DM-DRP-26: Overlap resolution at tract & patch boundaries.
  • DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects.
  • DM-DRP-28: Difference images are now a first-class data product during data release processing.
  • DM-DRP-35: Simultaneous measurement across a suite of coadds representing different bandpasses, epocs, and flavors.

Due in July 2021

  • No milestones due.

Due in June 2021

  • DMTC-8100-2210: 100G ring operating at 99.99% availability - Complete
  • F21S: F21 Start
  • DM-DRP-23: Atmospheric characterization based on data from the Auxiliary Telescope now available.

Due in May 2021

Due in April 2021

Due in March 2021

  • No milestones due.

Due in February 2021

  • No milestones due.

Due in January 2021

Due in December 2020

  • DM-NET-5: Base - Archive Network Functional 100 Gbps
  • DLP-464: Install networking for normal Base Center operations for ComCam scale
  • DLP-466: Install networking for ComCam-scale Chilean DAC serving on order 20 people
  • LDM-503-09: Test report: Ops rehearsal for commissioning #1
  • LDM-503-11b: Test report: Pipelines Release Fall 2019
  • LDM-503-14a: Science Platform ready for DP0

Due in November 2020

  • DM-NCSA-15: Batch Processing Service for offline spectrograph data processing
  • DM-NCSA-22: Information in consolidated database available to QA portal
  • DLP-465: Install networking for normal Base Center operations for full-camera scale
  • DLP-468: Provide networking for formal Chilean DAC production at full-camera scale
  • DLP-467: Install networking for full-camera-scale Chilean DAC serving on order 80 people
  • F20F: F20 Complete
  • LDM-503-08: Test report: Spectrograph data acquisition
  • LDM-503-10: Test report: DAQ validation
  • LDM-503-11a: ComCam Ops Readiness
  • DM-DRP-12: Background estimation over the full visit.
  • DM-DRP-13: PSF estimation over the full visit.
  • DM-DRP-15: All varieties of coadd required for object detection and characterization are now produced during normal pipeline operation (although not necessarily at the ultimately required level of fidelity).
  • DM-DRP-21: Integrated image characterization pipeline for Data Release Production.
  • DM-DRP-30: Forced photometry is now performed on individual processed visit images during data releases.
  • DM-DRP-34: Selection maps are generated during data releases.
  • DM-DAX-7: Generation 3 Butler, running on a POSIX file-system provided by NCSA
  • DM-DAX-8: Scatter-gather support in PipelineTask
  • DM-DAX-12: Generation 3 middleware at feature parity with Generation 2
  • DM-DAX-14: Provenance system review completed
  • DM-SQRE-6: IVOA Server-side Operations for Data Access API
  • DM-SQRE-7: IVOA-compliant image metadata services

Due in October 2020

  • No milestones due.

Due in September 2020

  • DLP-46: Full Capacity over Diverse path Ready
  • DMTC-8000-0990: Full capacity La Serena to Santiago over Primary link AND Secondary link - Complete
  • DMTC-8100-2410: FIU-CIARA Management Contract Delivery #6 Complete
  • DLP-33: Santiago - Boca Raton Full Integration Ready
  • DMTC-6800-1325: Network Verification Pre-Tests

Due in August 2020

  • No milestones due.

Due in July 2020

  • DMTC-8100-2170: 100G ring operating at 99.0% availability - Complete

Due in June 2020

  • DM-NCSA-14: Data Backbone endpoints in Chile for ingestion and access, distribution over WAN, ingest at NCSA into custodial file sto
  • F20S: F20 Start
  • LDM-503-06: Test report: DM ComCam interface verification readiness
  • DM-AP-10: Advanced single frame measurement pipeline for Alert Production.
  • DM-DRP-22: Template generation integrated with Data Release Production pipelines.

Due in May 2020

  • DLP-533: Base Center Integration Complete
  • S20F: S20 Complete
  • LDM-503-05: Test report: Alert distribution validation
  • LDM-503-10b: Large Scale CCOB Data Access
  • LDM-503-09a: Test report: Pipelines Release Fall 2018
  • DM-AP-6: Alert format defined & queue system available.
  • DM-AP-18: Alert packet generation

Due in April 2020

  • No milestones due.

Due in March 2020

  • No milestones due.

Due in February 2020

  • DM-SQRE-3: Notebook service stable for commissioning and other internal project users

Due in January 2020

  • A6150: Start Milestone: January 1st, 2020
  • DM-DRP-8: Calibration product generation for the Auxiliary Telescope.

Due in December 2019

Due in November 2019

  • F19F: F19 Complete
  • DM-STAFF: Staffing Checkpoint
  • LDM-503-08b: Small Scale CCOB Data Access
  • DM-AP-11: Difference imaging includes noise decorrelation and correction for differential chromatic refraction.

Due in October 2019

  • A1300: Start Milestone: October 1st, 2019
  • DLP-70: Miami - Chicago Early Integration Ready
  • DLP-69: Boca Raton - Chicago Early Integration Ready
  • DMTC-8200-0600: ESNet Implementation Phase 1- Complete

Due in September 2019

Due in August 2019

  • A4670: June 2019 Travel to Chile - Travel Start
  • A4680: June 2019 Travel to Chile - Travel End
  • A4750: Summit AA Systems Installed at the Summit
  • A4860: ComCam Systems in Tucson ready for NCSA
  • DLP-552: LSST Software Release 9.1 Complete, Ready for Full Camera

Due in July 2019

  • DM-NCSA-11: Verified acquisition of raw and crosstalk-corrected exposures at raft scale, incl. correct metadata
  • A4369.9: Ship Hardware to BDC
  • IT-725-M: Base Data Center Network Complete
  • DM-NET-4: Base LAN installed
  • LDM-503-10a: Science Platform: TAP service with federated SSO A&A

Due in June 2019

  • A2330: Ship AA System to Chile
  • A4640: Ship ComCam Hardware to Tucson
  • DMTC-8000-0950: Technical Implementation Plan for 40Gbps diverse path - Complete
  • F19S: F19 Start

Due in May 2019

  • DM-NCSA-13: Header Writing Service for Spectrograph use case
  • A5090: Expected Arrival of ComCam Systems to NCSA
  • IT-716-M2: Furniture installed in Offices, purchased cabling received
  • DMTC-8100-2130: 100G ring operating at 98.0% availability - Complete
  • S19F: S19 Complete
  • DM-SUIT-7: Mapping between SUIT systems & NCSA auth system
  • DM-SUIT-10: SUIT deployment procedure

Due in April 2019

  • DLP-36: Santiago - Miami Early Integration Ready
  • DMTC-8100-2112: Miami - Boca Raton path diverse fiber - Complete

Due in March 2019

  • DM-NCSA-16: Perform ISR processing on ComCam-scale data.
  • DMTC-8000-0870: Report on functional fiber, including uptime, 1st year functional milestone - Complete

Due in February 2019

  • DM-NCSA-3: Provide database for metadata, provenance, location and demonstrate ingest at small scale
  • DM-NCSA-4: Minimal support for the small operational schema including file metadata and provenance for every file, and record of in
  • IT-718-M: Base Center Full Occupancy
  • LDM-503-02: Test report: HSC reprocessing
  • LDM-503-04: Test report: Aux Tel DAQ integration functionality test
  • LDM-503-07: Test report: Camera data processing
  • LDM-503-04b: Test report: Aux Tel DAQ interface Integration Verification and Spectrograph Operations Rehearsal
  • DM-SUIT-5: Search and display processed HSC data
  • DM-DRP-2: Basic visualization and quality assessment tools operational on HSC-scale data volumes.
  • DM-DRP-11: Pipelines code provides supports for database ingestion of results.
  • DM-DAX-5: Database ingest in support of HSC reprocessing (ie, large catalog ingest)

Due in January 2019

  • DM-AP-8: Advanced ISR, including ghost and linear feature masking, correction for the Brighter-Fatter effect and compensation for pixel response non-uniformity.

Due in December 2018

Due in November 2018

  • IT-716-M: Base Center Beneficial Occupancy
  • F18F: F18 Complete
  • DM-SUIT-8: SUIT portal integrated with workspace
  • DM-DRP-19: QA metrics are generated during pipeline execution.
  • DM-DRP-37: Artifact rejection and background matching during coadd construction.
  • DM-DRP-29: Moving point source model fitting now available.
  • DM-DAX-10: Per-object gather support in Butler
  • DM-SQRE-2: Notebook service capabilities are suitable for the commissioning team to develop notebooks for its needs

Due in October 2018

  • DMTC-8100-2330: FIU-CIARA Management Contract Delivery #4 Complete
  • DMTC-8100-2530: Data passing test report - Complete
  • DM-DRP-9: Data reduction pipeline for the Auxiliary Telescope.

Due in September 2018

Due in August 2018

  • DM-NCSA-9: Test instance of alert distribution hosting service and L1 database in Development & Integration Enclave
  • DLP-551: LSST Software Release 8.1 Complete, Ready for Commissioning Camera

Due in July 2018

  • DM-AP-9: JOINTCAL1: Jointcal at a functional level
  • DM-DRP-7: Coordinate transformation tool provided for use with the Collimated Beam Projector.
  • DM-DRP-17: Simultaneous photometric and astrometric fitting to multiple exposures.

Due in June 2018

  • DMTC-8100-2090: 100G ring operating at 97.0% availability - Complete
  • F18S: F18 Start
  • DM-AP-7: Basic instrument signature removal (ISR) capability.
  • DM-DRP-3: PSF-homogenized coadd construction.
  • DM-DRP-5: Camera package supporting the LSST Camera.
  • DM-DRP-38: Camera package supporting the Commissioning Camera.

Due in May 2018

  • DMTC-8000-0836: La Serena to Santiago with live traffic and uptime over LSST Lambda Ready - Complete
  • S18F: S18 Complete

Due in April 2018

Due in March 2018

  • DM-NCSA-5: Level 1 archiving system able to acquire pixel data from the Aux Tel DAQ, header metadata via OCS, assemble FITS image,
  • DM-NCSA-6: Ability to transfer files originating from Tucson to NCSA and ingest files at NCSA, including metadata and provenance
  • DM-NCSA-7: Capability to paint displays for Tucson and NCSA
  • DM-NET-2: Mountain - Base Network Functional 2 x 100 Gbps
  • DLP-61: La Serena – Santiago link Ready
  • DLP-44: La Serena to Santiago with test traffic AURA DWDM LSST Fiber
  • DMTC-8000-0826: Intermediate traffic testing and report - Complete
  • DLP-40: Summits-Gatehouse Initial Capability Ready
  • DLP-65: Mountain - Base LSST and AURA DWM End Node Installation and Test Ready
  • DM-DRP-32: Object classification system available.
  • DM-DAX-11: Transformed EFD schema

Due in February 2018

  • No milestones due.

Due in January 2018

  • DM-DRP-16: Global photometric fitting (e.g. Burke et al. Forward Global Calibration Method).

Due in December 2017

Due in November 2017

  • DLP-864: Coordination with NCSA/CC-IN2P3 JCC
  • DLP-368: Initial AP DMCS
  • DLP-459: Qserv capability to hold data reduced at NCSA
  • DLP-458: Ingest of bulk data from real cameras to ComCam scale
  • DLP-457: Support runs on real cameras data and ImSim data
  • DLP-442: Procure infrastructure for Base Center to meet FY18 goals
  • DLP-433: Procure infrastructure for Archive Center to FY18 scale
  • DMTC-8000-0818: Acceptance Test with Equipment Installed - Complete
  • DMTC-7400-2510: REUNA La Serena - Santiago Optical Channel in operation - Complete
  • DMTC-7400-2130: End of commissioning. Acceptance of fiber connection - Complete
  • F17F: F17 Complete
  • LDM-503-01: Test report: Science Platform with WISE data in PDAC
  • LDM-503-03: Test report: Alert generation validation
  • DM-SUIT-1: Search and display WISE sources (objects) in PDAC
  • DM-SUIT-4: Multiple data traces in chart space
  • DLP-207: Extra LSST query capabilities in web UI
  • DLP-205: LSST web UI search external catalog
  • DLP-222: Provide time-domain data view
  • DM-DRP-14: Insertion of simulated sources into the data stream to check pipeline performance.
  • DM-DRP-18: Initial multi-band deblending algorithm available.
  • DM-DAX-1: WISE data ingest to PDAC
  • DM-DAX-6: Prototype level 1 database
  • DLP-473: Async Queries in Qserv
  • DLP-725: Working Prototype of AP/L1 Database
  • DLP-344: Release SQuaSH-in-a-box automated deployment platform
  • DLP-352: Metrics of Interest integrated in running stack or post-tests

Due in October 2017

Due in September 2017

Due in August 2017

  • DMTC-7400-2090: Report on functional fiber connections, including AURA equipment - Complete
  • DLP-558: LSST Software Release 7.1 Complete
  • DM-AP-1: Basic single frame measurement pipeline.
  • DM-AP-2: Alard & Lupton-style image differencing.
  • DM-AP-3: Point source & dipole measurement on difference images.
  • DM-AP-4: DIASource association
  • DM-AP-5: DIAObject generation
  • DM-DRP-6: Camera package supporting the Auxiliary Telescope.
  • DM-SQRE-1: Project internal Jupyter notebook service

Due in July 2017

  • DMTC-7400-2420: REUNA La Serena - Santiago DWDM equipment - Complete
  • DM-SUIT-2: Search WISE coaded data single exposure images in PDAC (the images are from IRSA at IPAC, not NCSA)
  • DM-DAX-2: Query service supporting IVOA TAP protocol, w/ support for asynchronous queries

Due in June 2017

Due in May 2017

  • DM-NCSA-1: Provide regular reprocessing service for HSC data
  • DM-NCSA-2: Provide access to results of regular reprocessing (NB the form this takes depends upon available DAX functionality)
  • DLP-538: LSE-130 Data items Exchanged between Cam & DM - Phase 3
  • DMTC-0000-3000: LSST Software Release S17 Complete
  • S17F: S17 Complete
  • DM-DRP-1: HSC merger complete: all functionality deployed for the most recent HSC data release processing is now available within the LSST stack.
  • DM-DRP-4: Calibration product generation in support of basic ISR.
  • DLP-736: Performance and capability enhancements in support of catalog use cases
  • DLP-152: Initial PSF-matched image coaddition available
  • DLP-472: Qserv Data Distribution

Due in April 2017

Due in March 2017

  • No milestones due.

Due in February 2017

  • DLP-37: Summits-Gatehouse Fiber Ready
  • DMTC-7400-2210: Fiber delivery by Telefonica - Complete
  • DLP-567: LSST Software Release 7.0 Complete

Due in January 2017

  • DMTC-8000-0806: 3T Fiber Installation Certification and Acceptance Test - Complete

Due in December 2016

Due in November 2016

  • DLP-847: Meet camera DAQ/OCS milestones for FY16
  • DLP-366: Mid-scale processing of eligible camera data
  • DLP-412: Monitor development cluster
  • DMTC-0000-0030: LSST Software Release F16 Complete
  • DLP-871: Firefly server side code refactor
  • DLP-802: Alert Production Database (APDB) design

Due in October 2016

  • No milestones due.

Due in September 2016

  • DMTC-8100-2050: 100G ring operating at 95.0% availability - Complete
  • DMTC-8100-2250: FIU-CIARA Management Contract Delivery #2 Complete
  • DLP-544: LSE-72 OCS-DM master interface - Phase 3
  • DM-SUIT-3: Time series analysis tool for WISE data

Due in August 2016

  • DLP-557: LSST Software Release 6.1 Complete
  • DLP-579: Usability and developer efficiency
  • DLP-346: SQuaSH Prototype using existing architectural components deployed

Due in July 2016

  • DLP-541: LSE-68 Camera-DM Image data interfaces - Phase 3

Due in June 2016

Due in May 2016

  • DMTC-0000-0020: LSST Software Release X16 Complete
  • DLP-694: Prototype Brighter-Fatter correction available

Due in April 2016

  • DLP-45: Delivery of “functional Pre-operations Link” (at least 4Gbps), Ready
  • DMTC-8000-0910: Delivery of 4Gbs link from La Serena to Santiago Ready - Complete

Due in March 2016

Due in February 2016

  • DLP-469: Coordination with NCSA/CC-IN2P3 JCC
  • DLP-358: Refinement of facility integration plan and traceability of science and operational requirements
  • DLP-364: Build prototype production sequence for the DMCS
  • DLP-850: Prototype authentication/authorization system complete
  • DLP-405: Investigate CAS integration with Internet2 AAA tools
  • DLP-411: Monitor OpenStack
  • DLP-424: Integration of DevOps to OpenStack
  • DLP-471: Refresh sizing model
  • DLP-431: Refinement of facility integration plan and traceability of science and operational requirements
  • DLP-566: LSST Software Release 6.0 Complete
  • DLP-872: Convert major portion of GWT in Firefly to pure JavaScript
  • DLP-486: Prototype merge & deduplication in tract & patch overlaps
  • DLP-103: Prototype multi-coadd peak association available
  • DLP-100: Consistent multi-band deblending available
  • DLP-649: Shared Scans
  • DLP-345: Prototype metrics collection system deployed
  • DLP-351: Intermediate Performance Metrics routinelly calculated and published
  • DLP-337: Release new stack documentation infrastructure

Due in January 2016

  • DLP-452: Support datasets on basic file system infrastructure with apropos policies
  • DLP-41: Mountain - Base Fiber Path Ready
  • DMTC-7400-2170: Gatehouse to Summit contract signed; Design Ready - Complete
  • DLP-276: Dipole measurement suite

Due in December 2015

  • DLP-43: Fiber Delivery Ready
  • DMTC-8000-0760: Technical Implementation plan of fiber connection (primary link) - Complete
  • DMTC-7400-0815: La Serena-Gatehouse path design Ready
  • DMTC-7400-2010: Gatehouse to La Serena path design Ready - Complete
  • DLP-288: HSC merger complete
  • DLP-700: Galaxy shear fitting parameters available
  • DLP-342: Publish updated introductory software guide to developing with the stack
  • DLP-341: Deploy stack user support and discussion forum

Due in November 2015

  • DLP-101: Prototype coadd detection available
  • DLP-73: Footprints redesigned and refactored
  • DLP-635: Qserv Query Cancellation

Due in October 2015

  • DMTC-8100-2010: First Data passed on 100G ring without errors - Complete
  • DLP-528: Archive Site Ready for Equipment Configuration
  • DLP-275: Deliver document on effects of DCR
  • DLP-805: Distributed Database and Table Management

Due in September 2015

  • DLP-357: Initial refinement of facility integration plan
  • DLP-389: Implement advanced logging functionality, including event-based logging
  • DLP-42: La Serena to Santiago Path Design Ready

Due in August 2015

  • DLP-356: Set up Joint Coordination Council (JCC) for coordination with satellite processing center
  • DLP-355: Establish MOA with CC-IN2P3 satellite processing center
  • DLP-361: Build prototype AP DMCS, including events
  • DLP-404: Identify candidate security packages
  • DLP-410: Implement Puppet for system configuration management
  • DLP-415: Evaluate iRODS, Datacat, and similar file system management technologies
  • DLP-423: DevOps assessment
  • DLP-430: Initial refinement of facility integration plan
  • DLP-556: LSST Software Release 5.1 Complete
  • DLP-218: Provide API for image display in Firefly
  • DLP-216: Provide basic APIs to use Firefly visualization components
  • DLP-503: Prototype direct coaddition
  • DLP-502: Prototype image artifact identification available
  • DLP-25: Prototype Web Service for SUI
  • DLP-340: Operating system testing for public stack releases increased
  • DLP-334: SQuaRE release improved or changed CI system

Due in July 2015

  • DLP-451: Deploy OpenStack capability
  • DLP-256: First draft definition of workspace
  • DLP-564: Able to use some data-access APIs developed by database team

Due in June 2015

  • DM-NET-1: Rubin Network Re-engineering
  • DLP-559: Mountain - Base Network Functional 1 Gbps

Due in May 2015

  • DLP-400: Refactor basic event services
  • DLP-441: Base Center requirements document
  • DLP-817: Multi-node Multi-query Integration Testing Harness
  • DLP-800: Data Ingest into MySQL

Due in April 2015

  • No milestones due.

Due in March 2015

  • DLP-518: Evaluate best-of-breed tools. Technology exploration.
  • DLP-521: Define the data access APIs with database team

Due in February 2015

  • DLP-450: Support lsst-dev for developers
  • DLP-449: Upgrade storage server hardware
  • DLP-555: LSST Software Release 5.0 Complete
  • DLP-644: Generalized Single Frame Astrometry Pipeline

Due in January 2015

  • DLP-422: Evaluate OpenStack as a development environment

Due in December 2014

  • DLP-520: Access Qserv for catalog search in SUI prototype
  • DLP-642: Camera Team using DM code

Due in November 2014

  • DLP-360: Prepare for OCS Middleware integration with AP DMCS
  • DLP-359: Improve fault tolerance in AP simulator
  • DLP-818: CSS
  • DLP-801: Qserv uses C++-based Geometry Code

Due in October 2014

  • DLP-519: Research the development framework for SUI

Due in September 2014

  • DLP-554: LSST Software Release 4.0 Complete
  • DLP-724: Forced per-epoch photometry
  • DLP-716: Fundamental flux measurements

Due in August 2014

  • DLP-578: Propose a workspace structure for discussion.
  • DLP-574: Evaluate best-of-breed tools. Technology exploration.
  • DLP-575: Evaluate best-of-breed tools. Technology exploration.
  • DLP-576: Evaluate best-of-breed tools. Technology exploration.
  • DLP-577: Evaluate best-of-breed tools. Technology exploration.
  • DLP-639: Re-design camera geometry classes
  • DLP-638: Refactor AFW R&D Code
  • DLP-588: AFW R&D code refactored
  • DLP-715: Fundamental centroid and shape measurements
  • DLP-697: Single epoch CModel minimization
  • DLP-696: Single epoch Sersic model fitting
  • DLP-836: New logging for DM
  • DLP-835: New xrootd client
  • DLP-834: Automated Testbed for Qserv

Milestones by WBS

02C.00: Data Management Level 2 Milestones

_images/graph_02C.00.png

Figure 2 Relationships between milestones in WBS 02C.00 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.

DLP-554: LSST Software Release 4.0 Complete

  • WBS: 02C.00
  • Level: 2
  • Successors: DLP-556
  • Due: 2014-09-02
  • Completed: 2014-09-30

Warning

No description available

DLP-555: LSST Software Release 5.0 Complete

  • WBS: 02C.00
  • Level: 2
  • Due: 2015-02-26
  • Completed: 2015-03-06

Warning

No description available

DLP-559: Mountain - Base Network Functional 1 Gbps

  • WBS: 02C.00
  • Level: 1
  • Successors: DM-NET-2
  • Due: 2015-06-02
  • Completed: 2015-06-01

Warning

No description available

DLP-556: LSST Software Release 5.1 Complete

Warning

No description available

DLP-528: Archive Site Ready for Equipment Configuration

  • WBS: 02C.00
  • Level: 1
  • Due: 2015-10-01
  • Completed: 2015-10-01

This is complete in. data/pmcs/201904-ME.xls.

DLP-566: LSST Software Release 6.0 Complete

Warning

No description available

DMTC-0000-0020: LSST Software Release X16 Complete

  • WBS: 02C.00
  • Level: 2
  • Predecessors: DLP-694
  • Due: 2016-05-31
  • Completed: 2016-07-12

Warning

No description available

DLP-541: LSE-68 Camera-DM Image data interfaces - Phase 3

  • WBS: 02C.00
  • Level: 2
  • Successors: S17S
  • Due: 2016-07-05
  • Completed: 2016-10-31

Complete with LCR-1788 June 2019.

DLP-557: LSST Software Release 6.1 Complete

Warning

No description available

DLP-579: Usability and developer efficiency

  • WBS: 02C.00
  • Level: 3
  • Successors: DLP-557
  • Due: 2016-08-31
  • Completed: 2017-12-01

Warning

No description available

DLP-544: LSE-72 OCS-DM master interface - Phase 3

  • WBS: 02C.00
  • Level: 2
  • Successors: DLP-558
  • Due: 2016-09-30
  • Completed: 2016-10-31

Warning

No description available

DMTC-0000-0030: LSST Software Release F16 Complete

Warning

No description available

S17S: S17 Start

Warning

No description available

DLP-567: LSST Software Release 7.0 Complete

  • WBS: 02C.00
  • Level: 2
  • Due: 2017-02-28
  • Completed: 2017-02-28

Warning

No description available

DLP-538: LSE-130 Data items Exchanged between Cam & DM - Phase 3

approval of LCR-1494 in June 2019 allows this to be closed, LSE-400 [21] covers the items.

DMTC-0000-3000: LSST Software Release S17 Complete

Warning

No description available

S17F: S17 Complete

Warning

No description available

F17S: F17 Start

  • WBS: 02C.00
  • Level: 2
  • Predecessors: DLP-736
  • Successors: F17F
  • Due: 2017-06-01
  • Completed: 2017-06-01

Warning

No description available

DLP-558: LSST Software Release 7.1 Complete

Warning

No description available

F17F: F17 Complete

Warning

No description available

LDM-503-01: Test report: Science Platform with WISE data in PDAC

This test demonstrates the deployment of the Portal and API Aspects of the LSST Science Platform (LSE-319 [19], LDM-542 [15], LDM-554 [14]) to serve all-sky precursor data, including catalogs totaling of order 50–100 billion rows, in a prototype LSST Data Access Center hosted in the LSST Data Facility at NCSA.

LDM-503-03: Test report: Alert generation validation

This test demonstrates the successful execution of a prototype Alert Generation science payload (LDM-148 [22], LDM-151 [25]), processing data from precursor surveys at a relatively small scale using compute resources at the LSST Data Facility at NCSA.

S18S: S18 Start

Warning

No description available

LSST-1160: Header Service Integration

  • WBS: 02C.00
  • Level: 2
  • Predecessors: DM-NCSA-27
  • Successors: LSST-1190
  • Due: 2017-12-29
  • Completed: 2018-03-31

Warning

No description available

LSST-1190: Control Interfaces Verification: Single Visit

Warning

No description available

LSST-1200: Interface Verification: Single Visit

Warning

No description available

LSST-1220: EFD ETL Under DM Control

  • WBS: 02C.00
  • Level: 2
  • Predecessors: LSST-1190
  • Successors: LSST-1580
  • Due: 2017-12-29
  • Completed: 2019-03-29

Warning

No description available

S18F: S18 Complete

Warning

No description available

F18S: F18 Start

Warning

No description available

DLP-551: LSST Software Release 8.1 Complete, Ready for Commissioning Camera

  • WBS: 02C.00
  • Level: 1
  • Predecessors: DLP-558
  • Successors: DLP-552, COMC-0100
  • Due: 2018-08-31
  • Completed: 2018-07-12

Warning

No description available

F18F: F18 Complete

Warning

No description available

S19S: S19 Start

  • WBS: 02C.00
  • Level: 2
  • Predecessors: S18S
  • Successors: S20S
  • Due: 2018-12-01
  • Completed: 2018-12-01

Warning

No description available

LDM-503-02: Test report: HSC reprocessing

This test demonstrates the successful execution of a prototype Data Release Production science payload (LDM-148 [22], LDM-151 [25]), processing data from precursor surveys at a relatively small scale using compute resources at the LSST Data Facility at NCSA.

LDM-503-04: Test report: Aux Tel DAQ integration functionality test

  • WBS: 02C.00
  • Level: 2
  • Test specification: LDM-538 [9] v1.0 S4.1 (RAS-00-00)
  • Predecessors: DM-NCSA-4
  • Successors: LDM-503-10, LDM-503-04b
  • Due: 2019-02-28
  • Completed: 2018-06-29

This test demonstrates that LSST data acquisition system can capture raw data from instrumentation and construct a well-formed image with proper headers.

LDM-503-07: Test report: Camera data processing

This test demonstrates that the LSST Science Pipelines can be successfully used to load and perform basic processing of data from the LSST Camera test systems. In the process, it establishes that data is properly made available at the LSST Data Facility at NCSA, and is accessible through the LSST Science Platform.

LDM-503-04b: Test report: Aux Tel DAQ interface Integration Verification and Spectrograph Operations Rehearsal

This test demonstrates that the well-formed image generated in LDM-503-04 can be written to the permanent record of the survey.

S19F: S19 Complete

Time-based milestone marking the completion of the Spring 2019 development cycle.

F19S: F19 Start

  • WBS: 02C.00
  • Level: 2
  • Predecessors: F18S
  • Successors: F19F
  • Due: 2019-06-01
  • Completed: 2019-06-01

Warning

No description available

LDM-503-10a: Science Platform: TAP service with federated SSO A&A

This test demonstrates the successful integration of a single-sign-on federated authentication system, and a basic authorization system, with the three Aspects of the LSST Science Platform (Portal, Notebook, and API), with the API Aspect containing at least a TAP service.

DLP-552: LSST Software Release 9.1 Complete, Ready for Full Camera

This pipeline software release will precede LDM-503-12a. This will have all functionality required to process data from LSSTCam. It will not have the full functionality for data release processing.

LDM-503-08b: Small Scale CCOB Data Access

  • WBS: 02C.00
  • Level: 2
  • Test specification: DMTR-102 [5] [DM-16073]
  • Predecessors: A5860
  • Successors: LDM-503-09
  • Due: 2019-11-08
  • Completed: 2019-07-15 [DM-17118]

This test demonstrates the integration of the CCOB with the permanent record of the survey.

F19F: F19 Complete

  • WBS: 02C.00
  • Level: 2
  • Predecessors: F19S
  • Successors: F20S, S20S, DM-STAFF
  • Due: 2019-11-27
  • Completed: 2019-11-29

Time-based milestone marking the completion of the Fall 2019 development cycle.

DM-STAFF: Staffing Checkpoint

  • WBS: 02C.00
  • Level: 2
  • Predecessors: F19F
  • Successors: LDM-503-13
  • Due: 2019-11-27
  • Completed: 2019-11-29

DM Management evaluate current DM staffing against project schedule and budget. If there is a shortfall through mid-2022 (the “ops gap”), this is the point at which we need to fill it.

S20S: S20 Start

  • WBS: 02C.00
  • Level: 2
  • Predecessors: S19S, S19F, F19F
  • Successors: S21S, S20F
  • Due: 2019-12-02
  • Completed: 2019-12-02

Warning

No description available

DLP-533: Base Center Integration Complete

  • WBS: 02C.00
  • Level: 1
  • Successors: COMC-1564, COMC-0100, COMC-0620-2090
  • Due: 2020-05-13
  • Completed: 2020-02-28

This milestone represents the availability of the base center in Las Serena. This includes DM staging machines, the commissioning cluster. It also includes the Chilean Data Access center machines. The later will only be deployed after the US DAC is up.

S20F: S20 Complete

Time-based milestone marking the completion of the Spring 2020 development cycle.

LDM-503-05: Test report: Alert distribution validation

  • WBS: 02C.00
  • Level: 2
  • Test specification: LDM-533 [2] v2.0 SS 4.7–4.9 (test cases LVV-T216, LVV-T217, LVV-T218)
  • Predecessors: DM-NCSA-9, F18S, LDM-503-03, DM-AP-18
  • Successors: LDM-503-09
  • Due: 2020-05-29
  • Completed: 2018-07-17

This test demonstrates that a full-LSST-scale (10,000 messages every 39 seconds) alert stream can be distributed to end users. It further shows that simple filters based on alert contents can be applied to that stream.

LDM-503-10b: Large Scale CCOB Data Access

  • WBS: 02C.00
  • Level: 2
  • Test specification: DMTR-182 [7] [DM-16194]
  • Predecessors: CAMM8040, CAMM8300, S20F, LDM-503-10a
  • Successors: LSST-1580
  • Due: 2020-05-29
  • Completed: 2020-04-22 [DM-17123]

Demonstrate the ability to transfer data from the CCOB with 21 rafts from SLAC and ingested at NCSA and make available through an instance of the LSP.

LDM-503-09a: Test report: Pipelines Release Fall 2018

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2018 and early 2019. The contents of the release are described in LDM-564 [24]. The release will be accompanied by a characterization report.

F20S: F20 Start

Time-based milestone marking the start of the Fall 2020 development cycle.

LDM-503-06: Test report: DM ComCam interface verification readiness

This milestone records successful transfer of an image equivalent to one raft from the DAQ at the summit to reliable storage in the LSST Data Facility at NCSA, from where it will be made available for scientific evaluation through the LSST Science Platform.

F20F: F20 Complete

Time-based milestone marking the completion of the Fall 2020 development cycle.

LDM-503-08: Test report: Spectrograph data acquisition

This test demonstrates data acquisition from the Auxiliary Telescope spectrograph, transmission of all image files with associated metadata to NCSA for screening, and then further processing if needed.

LDM-503-10: Test report: DAQ validation

Test of the extended data acquisition (DAQ) network in Chile from the Summit to the Base Data Center.

LDM-503-11a: ComCam Ops Readiness

Demonstrate that all DM functionality required to support ComCam operations has been delivered.

LDM-503-09: Test report: Ops rehearsal for commissioning #1

This milestone represents the completion of a multi-day exercise in which (simulated) observations are performed and data is processed in as close to an operational mode as is possible at this stage in construction. In particular, it will demonstrate transfer, archiving and ingestion of raw data; offline processing of calibration and science data; and curation of the resultant data products.

LDM-503-11b: Test report: Pipelines Release Fall 2019

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2019 and early 2020. The contents of the release are described in LDM-564 [24]. The release will be accompanied by a characterization report.

LDM-503-14a: Science Platform ready for DP0

The Rubin Science Platform is ready to support Data Preview 0.1 as deployed on the IDF. The RSP capabilities are largely those demonstrated under previous milestones, but ported to the IDF and with deployment and authentication/authorization infrastructure improvements that support the scale of DP0.1 usage. The Notebook Aspect provides access to the Rubin Science Pipelines stack and to data exposed in a cloud-deployed Gen3 Butler repository; the API Aspect provides TAP queries against a Qserv deployment of catalog tables, but not image metadata; and the Portal Aspect provides a UI for visual and ADQL construction of queries against those tables. Image and image metadata access services in the API and Portal Aspects are not included.

LDM-503-11: Ops rehearsal for commissioning #2

Template generation, including code required to build templates in both Data Release and Alert Production science payloads, can now product templates which are corrected for the effects of Differential Chromatic Refraction (DCR), as described in LDM-151 [25] §6.18. Further, noise in image differences can be “decorrelated”, as described in LDM-151 [25] §6.19.1. Both of these represents initial capabilities, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

S21S: S21 Start

Warning

No description available

S21F: S21 Complete

Time-based milestone marking the completion of the Spring 2021 development cycle.

DM-IT-FIB: Camera Fibers Ready

  • WBS: 02C.00
  • Level: 2
  • Predecessors: S21F
  • Successors: COMC-060200-20790
  • Due: 2021-05-31
  • Completed: 2021-10-29

Camera fibers ready deployed from the Computer Room to the top of the TMA.

F21S: F21 Start

Warning

No description available

LDM-503-13a: Test report: Pipelines Release Fall 2020

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2020 and early 2021. The contents of the release are described in LDM-564 [24]. The release will be accompanied by a characterization report.

LDM-GEN3: Gen3 parity demonstrated

Show parity of Butler Gen3 with Butler Gen2 systems. We should be able to run processing pipelines with Gen3 which previously ran on Gen2. Demonstrate the ability to write new pipelines using the framework. Framework should be ready to support Preops processing, specifically DP0.2 (RTN-001 [23]).

F21F: F21 Complete

Time-based milestone marking the completion of the Fall 2021 development cycle.

S22S: S22 Start

Warning

No description available

DM-PORTAL: Resume development of LSP portal aspect

  • WBS: 02C.00
  • Level: 2
  • Predecessors: S22S
  • Successors: LSST-1580
  • Due: 2021-12-01
  • Completed: 2022-01-31

Warning

No description available

DM-CLOUD: Decide on final data facility design for DR1

  • WBS: 02C.00
  • Level: 2
  • Predecessors: S22S
  • Successors: LSST-1580
  • Due: 2021-12-01
  • Completed: 2022-01-31

Warning

No description available

S22F: S22 Complete

  • WBS: 02C.00
  • Level: 2
  • Successors: LSST-1590, DM11, LDM-503-15a
  • Due: 2022-05-05
  • Completed: 2022-05-31

Time-based milestone marking the completion of the Spring 2022 development cycle.

LDM-503-15a: Test report: Pipelines Release Spring 2022

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2021 and early 2022. The contents of the release are described in LDM-564 [24]. The release will be accompanied by a characterization report.

LDM-503-RSPa: Science Platform ready for DP0.2 with Image Services

The Rubin Science Platform is ready to support Data Preview 0.2 as deployed on the IDF. The RSP serves an end-to-end Rubin-Gen3-processed dataset with a substantial set of image and SDM-ified catalog data products. Beyond the capabilities provided under LDM-503-14a, this deployment provides: initial IVOA-oriented image metadata and image services in the API Aspect, specifically an ObsTAP service and a basic SODA-compatible image cutout service; Portal Aspect support for accessing these services and browsing images from the released dataset; and access to a User File Workspace from all three Aspects.

LDM-503-12: Ops rehearsal for commissioning #3

This milestone builds upon LDM-503-11, and serves to demonstrate that all necessary procedures and workflows are in place to support commissioning activities.

LDM-503-13: Ops rehearsal for data release processing #1 (ComCam data)

This milestone builds upon LDM-503-12, but with its scope extended beyond procedures which are necessary in commissioning to cover the operational era.

LDM-503-EFDa: Engineering Facility Database at the summit capturing and enabling access to all telemetry with M1/M3 active

An instance of the Engineering and Facility Database at the summit capturing and enabling access to all telemetry, including from an active M1M3. This may be distinct from the Auxiliary Telescope EFD instance. Demonstrate the ability to run for five days, allowing data retrieval and plotting via notebooks.

LDM-503-EFDb: EFD/telemetry data replicated at the LDF, stored in InfluxDB. aggregated into parquet files

  • WBS: 02C.00
  • Level: 2
  • Test specification: Undefined [DM-26344]
  • Predecessors: LDM-503-EFDa, COMC-0030, COMC-0100
  • Successors: LSST-1520
  • Due: 2023-02-14
  • Completion pending [DM-26347]

Demonstrate that EFD data from the Summit appears at a Data Facility. Show that the Parquet files generated contain all low frequency EFD values and summaries of high frequency values.

LDM-503-14: Test report: DM Readiness for Science Verification Surveys

This milestone represents the point at which all code and/or services which are supplied by DM in support of science verification will be tested and demonstrated to meet their requirements.

LDM-503-15: Ops rehearsal for data release processing #2

  • WBS: 02C.00
  • Level: 2
  • Test specification: Undefined [DM-17134]
  • Predecessors: LSST-1510, LSST-1516, LDM-503-13, COMC-0100
  • Successors: LDM-503-16
  • Due: 2023-05-31
  • Completion pending [DM-17135]

This milestone builds upon LDM-503-13, and will demonstrate the production of a data release under simulated operational conditions using data from science verification activities.

LDM-503-16a: Test report: Pipelines Release Spring 2023

  • WBS: 02C.00
  • Level: 2
  • Test specification: Undefined [DM-32560]
  • Predecessors: LSST-1516, COMC-0240
  • Successors: LDM-503-17a
  • Due: 2023-05-31
  • Completion pending [DM-32561]

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2023 and early 2023. The contents of the release are described in LDM-564 [24]. The release will be accompanied by a characterization report.

DLP-526: Archive Center Integration Complete at SLAC

  • WBS: 02C.00
  • Level: 1
  • Predecessors: LSST-1520
  • Successors: COMC-1664
  • Due: 2023-09-01
  • Completion pending

This milestone represents the availability of an archive center to host LSST DR1 data. This includes an science platform interface with the implied services for Data Access, a portal and Notebooks.

LDM-503-16: Ops rehearsal for data release processing #3

  • WBS: 02C.00
  • Level: 2
  • Test specification: Undefined [DM-17138]
  • Predecessors: LSST-1520, LSST-1510, LDM-503-15, COMC-0100
  • Successors: LDM-503-17, SITCOM-119
  • Due: 2023-09-01
  • Completion pending [DM-17139]

This milestone builds upon LDM-503-15 to demonstrate the production of a complete data release under simulated operational conditions.

LDM-503-12a: LSSTcam On Sky Readiness

Demonstrate that all DM functionality required to support operations with the full LSST Camera has been delivered.

LDM-503-EFDc: EFD data is queriable through TAP in the Science Platform

Demonstrate that engineering data for a specific observation time can be retrieved via TAP on the Science Platform or directly to a TAP tool like Topcat. Test DMS-REQ-0102 (Priority 1a) to permanently archive metadata from OCS and make available to rights holders. Also test DMS-REQ-0358 (Priority 1a), demonstrating support of simultaneous queries.

LDM-503-17: Final operations rehearsal

  • WBS: 02C.00
  • Level: 2
  • Test specification: Undefined [DM-17140]
  • Predecessors: DM15, DM2, DM7, DM11, LDM-503-16, COMC-0100, COMC-0620-9000
  • Successors: LSST-1580
  • Due: 2023-10-10
  • Completion pending [DM-17141]

This milestone builds upon LDM-503-16 to demonstrate successful operation of all aspects of the DM subsystem under simulated operational conditions.

LDM-503-RSPb: Science Platform ready for Science Verification

The Rubin Science Platform is ready to support Science Verification activities in the final phase of Rubin Observatory commissioning. In additional to the capabilities demonstrated in Milestone-2 (LDM-503-RSPa), the RSP will be scalable to the user load required to support Science Verification; provide access to a regularly updating stream of newly-acquired LSSTCam data; provide access to next-to-data computing and user batch computing to support intensive data analysis; provide access to a User Database Workspace; provide a means for integrating with a TBD focal-plane-scale visualization tool; provide capabilities in all three Aspects for following links between related datasets and exploiting provenance information; provide access to the EFD for recently acquired data, extending the capability demonstrated for LDM-503-EFDc; and provide substantial user-facing documentation, with links between the data and documentation.

LDM-503-17a: Test report: Final Pipelines Delivery

This milestone describes the final Science Pipelines release from the DM Construction Project. The contents of the release are described in LDM-564 [24]. The release will be accompanied by a characterization report.

02C.03: Alert Production

_images/graph_02C.03.png

Figure 3 Relationships between milestones in WBS 02C.03 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.

DLP-639: Re-design camera geometry classes

  • WBS: 02C.03.01
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-13

Warning

No description available

DLP-638: Refactor AFW R&D Code

  • WBS: 02C.03.05
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-13

Warning

No description available

DLP-642: Camera Team using DM code

  • WBS: 02C.03.01
  • Level: 3
  • Due: 2014-12-10
  • Completed: 2014-12-10

Warning

No description available

DLP-644: Generalized Single Frame Astrometry Pipeline

  • WBS: 02C.03.08
  • Level: 3
  • Due: 2015-02-24
  • Completed: 2015-02-24

Warning

No description available

DLP-275: Deliver document on effects of DCR

  • WBS: 02C.03.04
  • Level: 3
  • Successors: DLP-276
  • Due: 2015-10-29
  • Completed: 2016-04-08

Warning

No description available

DLP-276: Dipole measurement suite

  • WBS: 02C.03.04
  • Level: 3
  • Predecessors: DLP-275
  • Due: 2016-01-22
  • Completed: 2016-07-12

Warning

No description available

DM-AP-1: Basic single frame measurement pipeline.

Warning

No description available

DM-AP-2: Alard & Lupton-style image differencing.

Warning

No description available

DM-AP-3: Point source & dipole measurement on difference images.

Warning

No description available

DM-AP-4: DIASource association

  • WBS: 02C.03
  • Level: 3
  • Successors: LDM-503-03
  • Due: 2017-08-31
  • Completed: 2017-11-01

Warning

No description available

DM-AP-5: DIAObject generation

  • WBS: 02C.03
  • Level: 3
  • Successors: LDM-503-03
  • Due: 2017-08-31
  • Completed: 2017-11-01

Warning

No description available

DM-AP-7: Basic instrument signature removal (ISR) capability.

Warning

No description available

DM-AP-9: JOINTCAL1: Jointcal at a functional level

Warning

No description available

DM-AP-8: Advanced ISR, including ghost and linear feature masking, correction for the Brighter-Fatter effect and compensation for pixel response non-uniformity.

  • WBS: 02C.03
  • Level: 3
  • Predecessors: F18F
  • Successors: LDM-503-11b
  • Due: 2019-01-04
  • Completed: 2019-01-17

Warning

No description available

DM-AP-11: Difference imaging includes noise decorrelation and correction for differential chromatic refraction.

Template generation, including code required to build templates in both Data Release and Alert Production science payloads, can now product templates which are corrected for the effects of Differential Chromatic Refraction (DCR), as described in LDM-151 [25] §6.18. Further, noise in image differences can be “decorrelated”, as described in LDM-151 [25] §6.19.1. Both of these represents initial capabilities, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-AP-6: Alert format defined & queue system available.

Warning

No description available

DM-AP-18: Alert packet generation

  • WBS: 02C.03
  • Level: 3
  • Predecessors: S20F
  • Successors: LDM-503-05, DM-AP-6
  • Due: 2020-05-29
  • Completed: 2020-05-31

The AP Pipeline is now capable of generating per-DIASource alert packets, which comply with the current version of the LSST alert schema. This milestone only covers packet generation; success involves only those serialized alerts being e.g. written to disk, and does not imply that an alert transport system is available.

DM-AP-10: Advanced single frame measurement pipeline for Alert Production.

  • WBS: 02C.03
  • Level: 3
  • Predecessors: F20S, DM-AP-1
  • Successors: LSST-1580
  • Due: 2020-06-01
  • Completion pending

The Single Frame Processing system is now substantially complete and provides all functionality described in LDM-151 [25] §3.1. It supports all expected interfaces, data-flows, and products, but the details of the algorithms are expected to undergo continued refinement.

DM-AP-12: Difference imaging is now agnostic to the PSF of the template image.

  • WBS: 02C.03
  • Level: 3
  • Predecessors: DM-AP-11
  • Successors: LDM-503-15a
  • Due: 2021-08-31
  • Completion pending

The Alert Generation science payload is now agnostic to the PSF of the template, as described in LDM-151 [25] §3.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-AP-13: Trailed source measurement on difference images.

  • WBS: 02C.03
  • Level: 3
  • Predecessors: DM-AP-3
  • Successors: LDM-503-13a
  • Due: 2021-08-31
  • Completed: 2021-10-29

The Alert Generation science payload now provides the capability to fit a trailed point source model to detections made on difference images, as described in LDM-151 [25] §3.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-AP-15: Alert distribution system fully integrated.

  • WBS: 02C.03
  • Level: 3
  • Predecessors: LDM-503-12a
  • Successors: DM16, DM-AP-16
  • Due: 2023-09-01
  • Completion pending

The Alert Distribution system (LDM-151 [25] §3.3) is now available for deployment at the LSST Data Facility. The system is substantially complete, and supports all expected interfaces, data-flows, and products, but the details of the algorithms may undergo continued refinement.

DM-AP-17: Solar System Processing pipeline available.

  • WBS: 02C.03
  • Level: 3
  • Predecessors: LDM-503-12a
  • Successors: DM-DRP-36
  • Due: 2023-09-01
  • Completion pending

All of the algorithmic components which comprise the Solar System Processing (SSP) pipeline, as described in LDM-151 [25] §§6.25–6.28, are available for integration into science payloads. These components must provide realistic interfaces, data-flows and products, but it is expected that the details of the algorithms will undergo continued refinement.

DM-AP-14: Alert filtering system available.

  • WBS: 02C.03
  • Level: 3
  • Predecessors: LDM-503-17a, DM-AP-6
  • Successors: LSST-1580
  • Due: 2024-01-05
  • Completion pending

The Alert Filtering Service (LDM-151 [25] §3.3.4; LSE-163 [18] §3.5.2) can now be deployed as part of a prototype Alert Distribution system (LDM-151 [25] §3.3). This service is substantially complete, and supports all expected interfaces, data-flows, and products, but may not yet provide the complete set of filtering capabilities and/or may not perform at the level required to distribute the full LSST alert stream.

DM-AP-16: Full integration of the Alert Production system within the operational environment.

  • WBS: 02C.03
  • Level: 3
  • Predecessors: LDM-503-17a, DM-AP-15, COMC-0240
  • Successors: LSST-1580
  • Due: 2024-01-05
  • Completion pending

The Alert Production system (comprising Single Frame Processing, LDM-151 [25] §3.1; Alert Generation, LDM-151 [25] §3.3; and Precovery and Forced Photometry, LDM-151 [25] §3.4) is now available for deployment at the LSST Data Facility. The system is substantially complete, and supports all expected interfaces, data-flows and products, but the details of the algorithms may undergo continued refinement.

02C.04: Data Release Production

_images/graph_02C.04.png

Figure 4 Relationships between milestones in WBS 02C.04 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.

DLP-588: AFW R&D code refactored

  • WBS: 02C.04.01
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-715: Fundamental centroid and shape measurements

  • WBS: 02C.04.06
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2015-12-04

Warning

No description available

DLP-697: Single epoch CModel minimization

  • WBS: 02C.04.06
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-696: Single epoch Sersic model fitting

  • WBS: 02C.04.06
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-724: Forced per-epoch photometry

  • WBS: 02C.04.06
  • Level: 3
  • Due: 2014-09-12
  • Completed: 2014-09-12

Warning

No description available

DLP-716: Fundamental flux measurements

  • WBS: 02C.04.06
  • Level: 3
  • Due: 2014-09-12
  • Completed: 2014-09-12

Warning

No description available

DLP-503: Prototype direct coaddition

  • WBS: 02C.04.04
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-08-31

Warning

No description available

DLP-502: Prototype image artifact identification available

  • WBS: 02C.04.04
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-08-31

Warning

No description available

DLP-101: Prototype coadd detection available

  • WBS: 02C.04.05
  • Level: 3
  • Due: 2015-11-24
  • Completed: 2016-04-11

Warning

No description available

DLP-73: Footprints redesigned and refactored

  • WBS: 02C.04.01
  • Level: 3
  • Due: 2015-11-24
  • Completed: 2017-05-09

Warning

No description available

DLP-700: Galaxy shear fitting parameters available

  • WBS: 02C.04.06
  • Level: 3
  • Due: 2015-12-04
  • Completed: 2016-05-31

Warning

No description available

DLP-288: HSC merger complete

  • WBS: 02C.04.01
  • Level: 3
  • Due: 2015-12-29
  • Completed: 2016-12-07

Warning

No description available

DLP-486: Prototype merge & deduplication in tract & patch overlaps

  • WBS: 02C.04.05
  • Level: 3
  • Successors: DLP-566
  • Due: 2016-02-29
  • Completed: 2016-04-11

Warning

No description available

DLP-103: Prototype multi-coadd peak association available

  • WBS: 02C.04.05
  • Level: 3
  • Successors: DLP-566
  • Due: 2016-02-29
  • Completed: 2016-04-11

Warning

No description available

DLP-100: Consistent multi-band deblending available

  • WBS: 02C.04.05
  • Level: 3
  • Successors: DLP-566
  • Due: 2016-02-29
  • Completed: 2016-04-11

Warning

No description available

DLP-694: Prototype Brighter-Fatter correction available

  • WBS: 02C.04.02
  • Level: 3
  • Successors: DMTC-0000-0020
  • Due: 2016-05-31
  • Completed: 2016-05-31

Warning

No description available

DM-DRP-1: HSC merger complete: all functionality deployed for the most recent HSC data release processing is now available within the LSST stack.

Warning

No description available

DM-DRP-4: Calibration product generation in support of basic ISR.

Warning

No description available

DLP-736: Performance and capability enhancements in support of catalog use cases

  • WBS: 02C.04.01
  • Level: 3
  • Predecessors: DMTC-0000-3000, S17F
  • Successors: F17S
  • Due: 2017-05-31
  • Completed: 2016-12-19

Warning

No description available

DLP-152: Initial PSF-matched image coaddition available

  • WBS: 02C.04.04
  • Level: 3
  • Predecessors: S17F
  • Due: 2017-05-31
  • Completed: 2016-12-19

Warning

No description available

DM-DRP-6: Camera package supporting the Auxiliary Telescope.

  • WBS: 02C.04
  • Level: 3
  • Successors: LDM-503-08
  • Due: 2017-08-31
  • Completed: 2018-01-31

Warning

No description available

DM-DRP-14: Insertion of simulated sources into the data stream to check pipeline performance.

  • WBS: 02C.04
  • Level: 3
  • Successors: LDM-503-11b
  • Due: 2017-11-30
  • Completed: 2017-12-01

Warning

No description available

DM-DRP-18: Initial multi-band deblending algorithm available.

  • WBS: 02C.04
  • Level: 3
  • Successors: LDM-503-11b
  • Due: 2017-11-30
  • Completed: 2017-12-01

Warning

No description available

DM-DRP-16: Global photometric fitting (e.g. Burke et al. Forward Global Calibration Method).

  • WBS: 02C.04
  • Level: 3
  • Successors: LDM-503-09a
  • Due: 2018-01-31
  • Completed: 2018-05-31

Warning

No description available

DM-DRP-32: Object classification system available.

  • WBS: 02C.04
  • Level: 3
  • Successors: LDM-503-09a
  • Due: 2018-03-30
  • Completed: 2018-10-31

Warning

No description available

DM-DRP-3: PSF-homogenized coadd construction.

Warning

No description available

DM-DRP-5: Camera package supporting the LSST Camera.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: S18F
  • Successors: LDM-503-09a
  • Due: 2018-06-29
  • Completed: 2018-06-29

Warning

No description available

DM-DRP-38: Camera package supporting the Commissioning Camera.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: S18F
  • Successors: LSST-1513, LDM-503-09a
  • Due: 2018-06-29
  • Completed: 2018-06-29

Warning

No description available

DM-DRP-7: Coordinate transformation tool provided for use with the Collimated Beam Projector.

  • WBS: 02C.04
  • Level: 3
  • Successors: LDM-503-09a, T&SC-2100-2252
  • Due: 2018-07-12
  • Completed: 2018-07-05

Warning

No description available

DM-DRP-17: Simultaneous photometric and astrometric fitting to multiple exposures.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: S18F, DM-AP-9
  • Successors: LDM-503-09a
  • Due: 2018-07-20
  • Completed: 2018-11-29

Warning

No description available

DM-DRP-9: Data reduction pipeline for the Auxiliary Telescope.

Warning

No description available

DM-DRP-19: QA metrics are generated during pipeline execution.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: S18F
  • Successors: LDM-503-11b
  • Due: 2018-11-29
  • Completed: 2018-12-13

Warning

No description available

DM-DRP-37: Artifact rejection and background matching during coadd construction.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: F18F, DM-DRP-3
  • Successors: LSST-1580
  • Due: 2018-11-30
  • Completion pending

The Data Release Production science payload now includes code for generating final estimates of the sky background and artifact masks, as described in LDM-151 [25] §5.2.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-29: Moving point source model fitting now available.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: F18F
  • Successors: LSST-1580
  • Due: 2018-11-30
  • Completion pending

The Data Release Production science payload now includes the capability of fitting moving a moving point source model that includes parallax and proper motion as free parameters to all objects, as described in LDM-151 [25] §6.7.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-2: Basic visualization and quality assessment tools operational on HSC-scale data volumes.

Warning

No description available

DM-DRP-11: Pipelines code provides supports for database ingestion of results.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: S18F
  • Successors: F20F, DM-DAX-5
  • Due: 2019-02-28
  • Completed: 2019-12-31

The Data Release Production science payload now generates results which are compliant with the LSST Science Data Model, and which are suitable for database ingest.

DM-DRP-8: Calibration product generation for the Auxiliary Telescope.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: S18F, COMC-060200-10000
  • Successors: LSST-1580
  • Due: 2020-01-21
  • Completed: 2021-07-31

The Calibration Products Production science payload is now capable of generating those products which are required to calibrate the Auxiliary Telescope. This represent an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-22: Template generation integrated with Data Release Production pipelines.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: F20S
  • Successors: LDM-503-13a
  • Due: 2020-06-01
  • Completed: 2021-03-31

The Data Release Production science payload is now capable of generating templates for use in image differencing, as described in LDM-151 [25] §§5.2.3 & 5.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-12: Background estimation over the full visit.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: DM-DAX-8
  • Successors: LSST-1580
  • Due: 2020-11-30
  • Completed: 2020-06-30

Warning

No description available

DM-DRP-13: PSF estimation over the full visit.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: DM-DAX-8
  • Successors: LSST-1580, DM-DRP-15
  • Due: 2020-11-30
  • Completion pending

The Data Release Production science payload now includes code for estimating the PSF over the full focal plane, as described in LDM-151 [25] §6.11.3. This does not require that the PSF model be “physically motivated” (ie, decomposed into atmospheric and optical components). Further, this represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-15: All varieties of coadd required for object detection and characterization are now produced during normal pipeline operation (although not necessarily at the ultimately required level of fidelity).

  • WBS: 02C.04
  • Level: 3
  • Predecessors: DM-DRP-13
  • Successors: LDM-503-11b
  • Due: 2020-11-30
  • Completed: 2018-11-30

Warning

No description available

DM-DRP-21: Integrated image characterization pipeline for Data Release Production.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: DM-AP-9, DM-DAX-8
  • Successors: LSST-1580
  • Due: 2020-11-30
  • Completed: 2021-10-29

The Data Release Production science payload now includes an integrated image characterization pipeline, as described in LDM-151 [25] §5.1, including iterative refinement of the image characterization. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-30: Forced photometry is now performed on individual processed visit images during data releases.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: F20F
  • Successors: LDM-503-13a
  • Due: 2020-11-30
  • Completed: 2021-02-01

The Data Release Production science payload now includes the capability of performing “forced” photometry at the positions of known Objects on Processed Visit Images, as described in LDM-151 [25] §5.5.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-34: Selection maps are generated during data releases.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: F20F
  • Successors: LDM-503-13a
  • Due: 2020-11-30
  • Completed: 2021-05-24

The Data Release Production science payload now includes the capability of generating multi-scale maps that describe LSST’s depth and efficiency at detecting different classes of object as part of a data release, as described in LDM-151 [25] §5.6.3. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-23: Atmospheric characterization based on data from the Auxiliary Telescope now available.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: F21S, DM-DRP-9, COMC-060200-10000
  • Successors: LSST-1580
  • Due: 2021-06-01
  • Completed: 2021-08-31

The Calibration Products Production science payload is now capable of producing an atmospheric transmission model based on observations from the Auxiliary Telescope, as described in LDM-151 [25] §4.6.1. This milestone does not presume that the Auxiliary Telescope itself is available for testing: it is satisfied by the availability of a pipeline that can process appropriate simulated or precursor data. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-26: Overlap resolution at tract & patch boundaries.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: DM-DRP-27
  • Successors: LDM-503-13a
  • Due: 2021-08-31
  • Completed: 2021-02-01

The Data Release Production science payload is now capable of finalizing the definition of Objects which fall within the overlap regions at the edges of tracts and patches, as described in LDM-151 [25] §5.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects.

The Data Release Production science payload now includes the capability to build Objects by cross-matching the Source and DIASource tables, as described in LDM-151 [25] §5.3.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-28: Difference images are now a first-class data product during data release processing.

The Data Release Production science payload now includes the capability to generate difference images, and to perform source detection and characterization on those images, as described in LDM-151 [25] §5.2.5. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-35: Simultaneous measurement across a suite of coadds representing different bandpasses, epocs, and flavors.

  • WBS: 02C.04
  • Level: 3
  • Successors: DM-DRP-31
  • Due: 2021-08-31
  • Completed: 2021-10-29

The Data Release Production science payload now includes the capability to perform consistent measurement of all Objects in a patch across multiple coadds, as described in LDM-151 [25] §5.3.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-20: Refined set of LSST calibration products.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: DM-DRP-4, COMC-0100
  • Successors: LSST-1580, SITCOM-120
  • Due: 2023-02-14
  • Completed: 2021-07-29

The Calibration Products Production science payload is now capable of generating all calibration products which are required to enter commissioning.

DM-DRP-24: Physically motivated PSF model, including separate characterization of contributions from the atmosphere and the telescope system.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: LSST-1520
  • Successors: LSST-1580
  • Due: 2023-09-01
  • Completion pending

Per LDM-151 [25] §6.11.3, the PSF model used by the Data Release Production science payload is now decomposed into separate contributions from the telescope optics and the atmosphere.

DM-DRP-31: A photometric redshift is now provided for each object.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: LSST-1520, DM-DRP-35
  • Successors: DM-DRP-36
  • Due: 2023-09-01
  • Completion pending

The Data Release Production science payload now includes the capability of calculating photometric redshifts as part of a data release, as (briefly) described in LDM-151 [25] §5.6.5. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-36: Solar System Processing integrated to data release processing.

  • WBS: 02C.04
  • Level: 3
  • Predecessors: DM-AP-17, DM-DRP-31
  • Successors: LSST-1580, DM7
  • Due: 2023-09-01
  • Completion pending

The Data Release Production science payload now generates SSObjects, by delegating to the Solar System Processing (SSP) pipeline being delivered by the Alert Production team, as described in LDM-151 [25] §5.6.1. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

02C.05: Science User Interface and Tools

_images/graph_02C.05.png

Figure 5 Relationships between milestones in WBS 02C.05 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.

DLP-578: Propose a workspace structure for discussion.

  • WBS: 02C.05.05
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-574: Evaluate best-of-breed tools. Technology exploration.

  • WBS: 02C.05.01
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-575: Evaluate best-of-breed tools. Technology exploration.

  • WBS: 02C.05.02
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-576: Evaluate best-of-breed tools. Technology exploration.

  • WBS: 02C.05.03
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-577: Evaluate best-of-breed tools. Technology exploration.

  • WBS: 02C.05.04
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-519: Research the development framework for SUI

  • WBS: 02C.05.01
  • Level: 3
  • Due: 2014-10-14
  • Completed: 2014-10-14

Warning

No description available

DLP-520: Access Qserv for catalog search in SUI prototype

  • WBS: 02C.05.01
  • Level: 3
  • Due: 2014-12-02
  • Completed: 2014-12-02

Warning

No description available

DLP-521: Define the data access APIs with database team

  • WBS: 02C.05.01
  • Level: 3
  • Due: 2015-03-11
  • Completed: 2015-03-11

Warning

No description available

DLP-518: Evaluate best-of-breed tools. Technology exploration.

  • WBS: 02C.05.05
  • Level: 3
  • Due: 2015-03-20
  • Completed: 2015-03-20

Warning

No description available

DLP-564: Able to use some data-access APIs developed by database team

  • WBS: 02C.05.01
  • Level: 3
  • Due: 2015-07-28
  • Completed: 2015-11-17

Warning

No description available

DLP-256: First draft definition of workspace

  • WBS: 02C.05.05
  • Level: 3
  • Due: 2015-07-31
  • Completed: 2015-11-17

Warning

No description available

DLP-218: Provide API for image display in Firefly

  • WBS: 02C.05.02
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-11-17

Warning

No description available

DLP-216: Provide basic APIs to use Firefly visualization components

  • WBS: 02C.05.02
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2016-11-02

Warning

No description available

DLP-872: Convert major portion of GWT in Firefly to pure JavaScript

  • WBS: 02C.05.01
  • Level: 3
  • Due: 2016-02-24
  • Completed: 2016-04-08

Warning

No description available

DM-SUIT-3: Time series analysis tool for WISE data

  • WBS: 02C.05
  • Level: 3
  • Successors: LDM-503-01
  • Due: 2016-09-30
  • Completed: 2017-11-30

Warning

No description available

DLP-871: Firefly server side code refactor

  • WBS: 02C.05.01
  • Level: 3
  • Predecessors: DMTC-0000-0030
  • Successors: S17S
  • Due: 2016-11-30
  • Completed: 2017-07-31

Warning

No description available

DM-SUIT-2: Search WISE coaded data single exposure images in PDAC (the images are from IRSA at IPAC, not NCSA)

  • WBS: 02C.05
  • Level: 3
  • Predecessors: DM-DAX-2
  • Successors: LDM-503-01
  • Due: 2017-07-31
  • Completed: 2017-11-30

Warning

No description available

DM-SUIT-1: Search and display WISE sources (objects) in PDAC

  • WBS: 02C.05
  • Level: 3
  • Predecessors: DM-DAX-1
  • Successors: LDM-503-01
  • Due: 2017-11-30
  • Completed: 2017-11-30

Warning

No description available

DM-SUIT-4: Multiple data traces in chart space

  • WBS: 02C.05
  • Level: 3
  • Successors: LDM-503-01
  • Due: 2017-11-30
  • Completed: 2017-11-30

Warning

No description available

DLP-207: Extra LSST query capabilities in web UI

  • WBS: 02C.05.01
  • Level: 3
  • Predecessors: F17F
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-08-01

Warning

No description available

DLP-205: LSST web UI search external catalog

  • WBS: 02C.05.01
  • Level: 3
  • Predecessors: F17F
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-08-01

Warning

No description available

DLP-222: Provide time-domain data view

  • WBS: 02C.05.02
  • Level: 3
  • Predecessors: F17F
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-08-01

Warning

No description available

DM-SUIT-8: SUIT portal integrated with workspace

  • WBS: 02C.05
  • Level: 3
  • Successors: DM-SUIT-10
  • Due: 2018-11-30
  • Completed: 2020-05-29

Portal is integrated with LSST-provided WebDAV or VOSpace service exposing the User File Workspace. Portal provides functions for saving image and catalog query results to the logged-in user’s workspace, with the user’s identity correctly applied. Portal provides functions for accessing data in the user’s workspace and displaying it using the Portal’s image and table visualization tools, as well as using data from the workspace in other Portal contexts where a file can be uploaded, e.g., to perform a multi-object query. (NB: Still achievable post-DM-10, though only with WebDAV.).

DM-SUIT-5: Search and display processed HSC data

Portal provides access to a collection of public-release HSC data processed by LSST DM, including calibrated single-epoch images, coadds, and catalogs (including at least an Object-like catalog). Access is obtained via IVOA interfaces including TAP for catalogs and ObsTAP for image metadata. (NB: Still achievable post-DM-10).

DM-SUIT-7: Mapping between SUIT systems & NCSA auth system

  • WBS: 02C.05
  • Level: 3
  • Predecessors: S19F
  • Successors: DM-SUIT-10
  • Due: 2019-05-31
  • Completed: 2019-05-31

Portal is integrated with NCSA-provided federated-identity authentication and authorization system: the Portal recognizes that a user is logged in and displays that state, and passes the user’s identity along to LSST data services that it accesses. (NB: Still achievable post-DM-10) Claimed in May, based on installation on lsst-lsp-int.

DM-SUIT-10: SUIT deployment procedure

Portal software is adapted for deployment in Kubernetes. The set of container images required for deployment is documented in an LTD-style manner. Configuration files for Kubernetes are provided in Github and documented to a level that permits deployments to be performed by non-IPAC staff. (NB: Still achievable post-DM-10).

02C.06: Science Data Archive and Application Services

_images/graph_02C.06.png

Figure 6 Relationships between milestones in WBS 02C.06 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.

DLP-836: New logging for DM

  • WBS: 02C.06.02
  • Level: 3
  • Predecessors: LSST-1440
  • Due: 2014-08-01
  • Completed: 2015-11-17

Warning

No description available

DLP-835: New xrootd client

  • WBS: 02C.06.02
  • Level: 3
  • Predecessors: DLP-834
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-834: Automated Testbed for Qserv

  • WBS: 02C.06.02
  • Level: 3
  • Predecessors: LSST-1440
  • Successors: DLP-835
  • Due: 2014-08-01
  • Completed: 2014-08-01

Warning

No description available

DLP-818: CSS

  • WBS: 02C.06.02
  • Level: 3
  • Due: 2014-11-25
  • Completed: 2015-11-17

Warning

No description available

DLP-801: Qserv uses C++-based Geometry Code

  • WBS: 02C.06.02
  • Level: 3
  • Due: 2014-11-25
  • Completed: 2015-11-17

Warning

No description available

DLP-817: Multi-node Multi-query Integration Testing Harness

  • WBS: 02C.06.02
  • Level: 3
  • Due: 2015-05-29
  • Completed: 2015-11-17

Warning

No description available

DLP-800: Data Ingest into MySQL

  • WBS: 02C.06.01
  • Level: 3
  • Due: 2015-05-29
  • Completed: 2015-11-17

Warning

No description available

DLP-25: Prototype Web Service for SUI

  • WBS: 02C.06.02
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-09-01

Warning

No description available

DLP-805: Distributed Database and Table Management

  • WBS: 02C.06.02
  • Level: 3
  • Due: 2015-10-29
  • Completed: 2015-10-29

Warning

No description available

DLP-635: Qserv Query Cancellation

  • WBS: 02C.06.02
  • Level: 3
  • Due: 2015-11-25
  • Completed: 2016-01-12

Warning

No description available

DLP-649: Shared Scans

  • WBS: 02C.06.02
  • Level: 3
  • Due: 2016-02-24
  • Completed: 2017-05-26

Warning

No description available

DLP-802: Alert Production Database (APDB) design

  • WBS: 02C.06.01
  • Level: 3
  • Due: 2016-11-30
  • Completed: 2021-05-31

Warning

No description available

DLP-472: Qserv Data Distribution

  • WBS: 02C.06.02
  • Level: 3
  • Predecessors: DMTC-0000-3000, S17F
  • Due: 2017-05-31
  • Completed: 2018-06-29

Warning

No description available

DM-DAX-2: Query service supporting IVOA TAP protocol, w/ support for asynchronous queries

An IVOA TAP service supporting both synchronous and asynchronous query models, integrated with the LSP authentication architecture, and able to access catalog data hosted in a downstream in Qserv instance.

DM-DAX-1: WISE data ingest to PDAC

Warning

No description available

DM-DAX-6: Prototype level 1 database

Warning

No description available

DLP-473: Async Queries in Qserv

  • WBS: 02C.06.02
  • Level: 3
  • Predecessors: S17F, F17F
  • Successors: DLP-725
  • Due: 2017-11-30
  • Completed: 2017-11-01

Warning

No description available

DLP-725: Working Prototype of AP/L1 Database

  • WBS: 02C.06.01
  • Level: 3
  • Predecessors: F17F, DLP-473
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-11-01

Warning

No description available

DLP-837: AP-ready Butler Framework

  • WBS: 02C.06.02
  • Level: 3
  • Successors: F20F
  • Due: 2017-12-01
  • Completed: 2019-08-30

Warning

No description available

DM-DAX-11: Transformed EFD schema

  • WBS: 02C.06
  • Level: 3
  • Successors: LSST-1580
  • Due: 2018-03-30
  • Completed: 2018-05-31

Warning

No description available

DM-DAX-10: Per-object gather support in Butler

  • WBS: 02C.06
  • Level: 3
  • Predecessors: F18F
  • Successors: DM-DAX-5
  • Due: 2018-11-30
  • Completed: 2019-11-29

Demonstrate Butler configuration able to gather all data pertaining to a particular object, such as retrieving cutouts from all warps at a given box on a skymap.

DM-DAX-5: Database ingest in support of HSC reprocessing (ie, large catalog ingest)

Existing Qserv ingest tooling is highly manual and poorly parallelized. Develop new tooling to address these issues and support ingest campaigns at the scale needed for DRP.

DM-DAX-7: Generation 3 Butler, running on a POSIX file-system provided by NCSA

  • WBS: 02C.06
  • Level: 3
  • Predecessors: DM-DAX-12
  • Successors: DM-DAX-13
  • Due: 2020-11-30
  • Completed: 2020-06-30

Given a stood-up data backbone instance, demonstrate instantiation/configuration of a Butler Gen3 instance capable of retrieving images from that backbone instance. (As currently conceived, the data backbone is either a GPFS filesystem or object store, in combination with metadata that is a superset of that maintained in a Butler Gen3 registry. Depending on particular technologies and implementation strategies chosen for the data backbone, this may be either a “simple matter of configuration” of existing Butler connectors, or may require some additional custom connectors to be written.).

DM-DAX-8: Scatter-gather support in PipelineTask

Demonstrate a functional PipelineTask with scatter/gather functionality, such as Jointcal, SkyCorrect, or FGCM.

DM-DAX-12: Generation 3 middleware at feature parity with Generation 2

Generation 3 middleware, consisting of Butler updates and the PipelineTask framework, are now able to support existing science payloads with no loss of functionality over previous versions. That is, Gen 3 may not be feature complete, but Gen 2 can be retired with no functionality regressions across the subsystem.

DM-DAX-14: Provenance system review completed

  • WBS: 02C.06
  • Level: 3
  • Predecessors: F20F
  • Successors: DM-DAX-9
  • Due: 2020-11-30
  • Completed: 2022-01-31

Warning

No description available

DM-DAX-13: Generation 2 middleware retired

The Generation 2 middleware is deprecated, and may be removed from the codebase. No current science payload and no current execution service makes use of it.

DM-DAX-9: Butler Provenance System

Warning

No description available

DM-DAX-15: DAX Construction Complete

  • WBS: 02C.06
  • Level: 3
  • Successors: LSST-1580
  • Due: 2024-01-11
  • Completion pending

Warning

No description available

02C.07: LSST Data Facility

_images/graph_02C.07.png

Figure 7 Relationships between milestones in WBS 02C.07 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.

DLP-359: Improve fault tolerance in AP simulator

  • WBS: 02C.07.01
  • Level: 3
  • Due: 2014-11-19
  • Completed: 2014-11-19

Warning

No description available

DLP-360: Prepare for OCS Middleware integration with AP DMCS

  • WBS: 02C.07.01
  • Level: 3
  • Due: 2014-11-21
  • Completed: 2014-11-21

Warning

No description available

DLP-422: Evaluate OpenStack as a development environment

  • WBS: 02C.07.03
  • Level: 3
  • Due: 2015-01-29
  • Completed: 2015-01-29

Warning

No description available

DLP-450: Support lsst-dev for developers

  • WBS: 02C.07.04
  • Level: 3
  • Due: 2015-02-02
  • Completed: 2015-02-02

Warning

No description available

DLP-449: Upgrade storage server hardware

  • WBS: 02C.07.04
  • Level: 3
  • Due: 2015-02-02
  • Completed: 2016-07-15

Warning

No description available

DLP-400: Refactor basic event services

  • WBS: 02C.07.02
  • Level: 3
  • Successors: DLP-556
  • Due: 2015-05-28
  • Completed: 2015-01-26

Warning

No description available

DLP-441: Base Center requirements document

  • WBS: 02C.07.04
  • Level: 3
  • Due: 2015-05-29
  • Completed: 2015-05-29

Warning

No description available

DLP-451: Deploy OpenStack capability

  • WBS: 02C.07.04
  • Level: 3
  • Due: 2015-07-30
  • Completed: 2015-07-30

Warning

No description available

DLP-356: Set up Joint Coordination Council (JCC) for coordination with satellite processing center

  • WBS: 02C.07.00
  • Level: 3
  • Due: 2015-08-26
  • Completed: 2015-08-26

Warning

No description available

DLP-361: Build prototype AP DMCS, including events

  • WBS: 02C.07.01
  • Level: 3
  • Due: 2015-08-26
  • Completed: 2015-08-26

Warning

No description available

DLP-355: Establish MOA with CC-IN2P3 satellite processing center

  • WBS: 02C.07.00
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-08-31

Warning

No description available

DLP-404: Identify candidate security packages

  • WBS: 02C.07.02
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2016-06-07

Warning

No description available

DLP-410: Implement Puppet for system configuration management

  • WBS: 02C.07.02
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-08-31

Warning

No description available

DLP-415: Evaluate iRODS, Datacat, and similar file system management technologies

  • WBS: 02C.07.02
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-08-31

Warning

No description available

DLP-423: DevOps assessment

  • WBS: 02C.07.03
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-08-31

Warning

No description available

DLP-430: Initial refinement of facility integration plan

  • WBS: 02C.07.04
  • Level: 3
  • Due: 2015-08-31
  • Completed: 2015-08-31

Warning

No description available

DLP-357: Initial refinement of facility integration plan

  • WBS: 02C.07.01
  • Level: 3
  • Successors: DLP-358
  • Due: 2015-09-29
  • Completed: 2016-06-15

Warning

No description available

DLP-389: Implement advanced logging functionality, including event-based logging

  • WBS: 02C.07.01
  • Level: 3
  • Successors: DLP-566
  • Due: 2015-09-29
  • Completed: 2015-09-29

Warning

No description available

DLP-452: Support datasets on basic file system infrastructure with apropos policies

  • WBS: 02C.07.04
  • Level: 3
  • Due: 2016-01-22
  • Completed: 2016-07-15

Warning

No description available

DLP-411: Monitor OpenStack

  • WBS: 02C.07.02
  • Level: 3
  • Successors: DLP-412
  • Due: 2016-02-05
  • Completed: 2016-06-07

Warning

No description available

DLP-850: Prototype authentication/authorization system complete

  • WBS: 02C.07.02
  • Level: 3
  • Successors: S17S
  • Due: 2016-02-22
  • Completed: 2016-06-07

Warning

No description available

DLP-405: Investigate CAS integration with Internet2 AAA tools

  • WBS: 02C.07.02
  • Level: 3
  • Successors: S17S
  • Due: 2016-02-22
  • Completed: 2016-07-15

Warning

No description available

DLP-469: Coordination with NCSA/CC-IN2P3 JCC

  • WBS: 02C.07.00
  • Level: 3
  • Successors: DLP-864
  • Due: 2016-02-24
  • Completed: 2016-06-07

Warning

No description available

DLP-358: Refinement of facility integration plan and traceability of science and operational requirements

  • WBS: 02C.07.01
  • Level: 3
  • Predecessors: DLP-357
  • Due: 2016-02-24
  • Completed: 2016-04-08

Warning

No description available

DLP-364: Build prototype production sequence for the DMCS

  • WBS: 02C.07.01
  • Level: 3
  • Successors: DLP-566
  • Due: 2016-02-24
  • Completed: 2016-06-07

Warning

No description available

DLP-471: Refresh sizing model

  • WBS: 02C.07.04
  • Level: 3
  • Due: 2016-02-24
  • Completed: 2016-06-07

Warning

No description available

DLP-431: Refinement of facility integration plan and traceability of science and operational requirements

  • WBS: 02C.07.04
  • Level: 3
  • Due: 2016-02-24
  • Completed: 2016-07-15

Warning

No description available

DLP-424: Integration of DevOps to OpenStack

  • WBS: 02C.07.03
  • Level: 3
  • Due: 2016-02-29
  • Completed: 2016-07-15

Warning

No description available

DLP-847: Meet camera DAQ/OCS milestones for FY16

  • WBS: 02C.07.01
  • Level: 3
  • Predecessors: DMTC-0000-0030
  • Successors: F17F
  • Due: 2016-11-30
  • Completed: 2017-09-01

Warning

No description available

DLP-366: Mid-scale processing of eligible camera data

  • WBS: 02C.07.01
  • Level: 3
  • Predecessors: DMTC-0000-0030
  • Successors: F17F
  • Due: 2016-11-30
  • Completed: 2017-11-01

Warning

No description available

DLP-412: Monitor development cluster

Warning

No description available

DM-NCSA-1: Provide regular reprocessing service for HSC data

Warning

No description available

DM-NCSA-2: Provide access to results of regular reprocessing (NB the form this takes depends upon available DAX functionality)

Warning

No description available

DM-NCSA-28: IIP control code

Warning

No description available

DLP-864: Coordination with NCSA/CC-IN2P3 JCC

  • WBS: 02C.07.00
  • Level: 3
  • Predecessors: DLP-469
  • Successors: S18F
  • Due: 2017-11-30
  • Completed: 2017-11-01

Warning

No description available

DLP-368: Initial AP DMCS

  • WBS: 02C.07.01
  • Level: 3
  • Predecessors: F17F
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-11-01

Warning

No description available

DLP-459: Qserv capability to hold data reduced at NCSA

  • WBS: 02C.07.04
  • Level: 3
  • Predecessors: F17F
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-07-25

Warning

No description available

DLP-458: Ingest of bulk data from real cameras to ComCam scale

  • WBS: 02C.07.04
  • Level: 3
  • Predecessors: F17F
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-10-18

Warning

No description available

DLP-457: Support runs on real cameras data and ImSim data

  • WBS: 02C.07.04
  • Level: 3
  • Predecessors: F17F
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-11-01

Warning

No description available

DLP-442: Procure infrastructure for Base Center to meet FY18 goals

  • WBS: 02C.07.04
  • Level: 3
  • Predecessors: F17F
  • Due: 2017-11-30
  • Completed: 2018-04-30

Warning

No description available

DLP-433: Procure infrastructure for Archive Center to FY18 scale

  • WBS: 02C.07.04
  • Level: 3
  • Predecessors: F17F
  • Due: 2017-11-30
  • Completed: 2018-04-30

Warning

No description available

DMTC-4500-0010: Start NCSA S18

  • WBS: 02C.07
  • Level: 3
  • Due: 2017-12-01
  • Completed: 2017-12-01

Time-based milestone marking the start of NCSA’s Spring 2018 development cycle.

DM-NCSA-27: Deliver header service code

Warning

No description available

DM-NCSA-29: Deliver IIP image client code

  • WBS: 02C.07
  • Level: 3
  • Predecessors: DM-NCSA-28
  • Successors: LSST-1200
  • Due: 2017-12-29
  • Completed: 2018-03-31

Warning

No description available

DM-NCSA-6: Ability to transfer files originating from Tucson to NCSA and ingest files at NCSA, including metadata and provenance

  • WBS: 02C.07
  • Level: 3
  • Successors: LDM-503-04b
  • Due: 2018-03-05
  • Completed: 2018-10-31

Warning

No description available

DM-NCSA-5: Level 1 archiving system able to acquire pixel data from the Aux Tel DAQ, header metadata via OCS, assemble FITS image,

  • WBS: 02C.07
  • Level: 3
  • Successors: LDM-503-04b
  • Due: 2018-03-30
  • Completed: 2018-05-31

Warning

No description available

DM-NCSA-7: Capability to paint displays for Tucson and NCSA

  • WBS: 02C.07
  • Level: 3
  • Successors: LDM-503-04b
  • Due: 2018-03-30
  • Completed: 2018-06-29

Warning

No description available

DM-NCSA-9: Test instance of alert distribution hosting service and L1 database in Development & Integration Enclave

Hosting an instance of the alert distribution service delivered from University of Washington on our test containerized application management service platform, running feeds from the pipeline to the distribution services. Note: This was informally tied to LDM-503-05, which was de-scoped to a small demonstration of running an instance of Kafka at NCSA, so no L1 database was needed. We believe this was completed in August 2018.

DM-NCSA-10: Sustained archiving service that is OCS commandable

  • WBS: 02C.07
  • Level: 3
  • Successors: LDM-503-06
  • Due: 2018-09-25
  • Completed: 2019-05-31

The capability to use the existing OCS/CCS to command data acquisition and hand-off to a test Observatory Operations Data Service (OODS).

DM-NCSA-3: Provide database for metadata, provenance, location and demonstrate ingest at small scale

Warning

No description available

DM-NCSA-4: Minimal support for the small operational schema including file metadata and provenance for every file, and record of in

Warning

No description available

DM-NCSA-16: Perform ISR processing on ComCam-scale data.

Process ComCam-scale data through the ISR pipeline to generate QC parameters used by the observatory.

A5090: Expected Arrival of ComCam Systems to NCSA

  • WBS: 02C.07.09
  • Level: 4
  • Predecessors: T&SC-6500-8100
  • Due: 2019-05-08
  • Completed: 2019-04-30

Arrival at NCSA of hardware to support ComCam testing and integration.

DM-NCSA-13: Header Writing Service for Spectrograph use case

  • WBS: 02C.07
  • Level: 3
  • Successors: LDM-503-08
  • Due: 2019-05-14
  • Completed: 2019-05-31

The Header builder integrated into OCS and demonstrated. Header builder has identified sources of information for all modes of observing with the Spectrograph.

A4640: Ship ComCam Hardware to Tucson

  • WBS: 02C.07.09
  • Level: 4
  • Due: 2019-06-20
  • Completed: 2019-06-27

Date for ComCam support systems to ship from NCSA to Tucson.

A2330: Ship AA System to Chile

  • WBS: 02C.07.09
  • Level: 4
  • Due: 2019-06-28
  • Completed: 2019-06-28

Date for Authentication and Authorization hardware to ship from NCSA to Chile.

A4369.9: Ship Hardware to BDC

  • WBS: 02C.07.09
  • Level: 4
  • Successors: A4670
  • Due: 2019-07-15
  • Completed: 2019-07-17

Date for Base Data Center hardware to ship from NCSA to Chile.

DM-NCSA-11: Verified acquisition of raw and crosstalk-corrected exposures at raft scale, incl. correct metadata

  • WBS: 02C.07
  • Level: 3
  • Predecessors: F18F, T&SC-5900-0400
  • Successors: DLP-552, LDM-503-06
  • Due: 2019-07-29
  • Completed: 2020-06-30

Exercise the handoff between the L1 archiving and data backbone endpoint, including a monitoring service that tracks files that have been transferred and received.

A4860: ComCam Systems in Tucson ready for NCSA

  • WBS: 02C.07.09
  • Level: 4
  • Successors: LDM-503-09, A5860
  • Due: 2019-08-14
  • Completed: 2019-08-01

ComCam Test Stand ready for access by NCSA staff.

A4680: June 2019 Travel to Chile - Travel End

  • WBS: 02C.07.09
  • Level: 4
  • Successors: LDM-503-09
  • Due: 2019-08-16
  • Completed: 2019-08-24

End of travel to Chile to install hardware and systems at the Base Data Center.

A4670: June 2019 Travel to Chile - Travel Start

  • WBS: 02C.07.09
  • Level: 4
  • Predecessors: A4369.9
  • Due: 2019-08-17
  • Completed: 2019-08-17

Start of travel to Chile to install hardware and systems at the Base Data Center.

A4750: Summit AA Systems Installed at the Summit

  • WBS: 02C.07.09
  • Level: 4
  • Due: 2019-08-29
  • Completed: 2019-08-29

Authorization and Authentication systems installed at the Summit.

DM-NCSA-20: ComCam Archiving Service

  • WBS: 02C.07
  • Level: 3
  • Predecessors: T&SC-5900-0400
  • Successors: LDM-503-11a
  • Due: 2019-09-06
  • Completed: 2020-06-30

Implement a Service that acquires raw data from a camera at raft scale and sends to NCSA for ingestion into the Data Backbone. This is a preliminary milestone to demonstrate initial capabilities.

A1300: Start Milestone: October 1st, 2019

  • WBS: 02C.07.05
  • Level: 4
  • Due: 2019-10-01
  • Completed: 2019-10-01

Warning

No description available

A6150: Start Milestone: January 1st, 2020

  • WBS: 02C.07
  • Level: 3
  • Due: 2020-01-03
  • Completed: 2020-01-03

Warning

No description available

DM-NCSA-14: Data Backbone endpoints in Chile for ingestion and access, distribution over WAN, ingest at NCSA into custodial file sto

  • WBS: 02C.07
  • Level: 3
  • Successors: LDM-503-08
  • Due: 2020-06-22
  • Completed: 2021-11-17

Provide a data Backbone endpoint in Chile, allowing distribution over a wide-area network and ingestion at NCSA.

DM-NCSA-15: Batch Processing Service for offline spectrograph data processing

  • WBS: 02C.07
  • Level: 3
  • Predecessors: DM-DAX-12
  • Successors: LDM-503-08
  • Due: 2020-11-30
  • Completed: 2020-05-29

Demonstrate the ability to process offline Spectrograph data using a science pipeline in the Batch Processing Service. Output(s) will be available in the Data Backbone.

DM-NCSA-22: Information in consolidated database available to QA portal

  • WBS: 02C.07
  • Level: 3
  • Predecessors: DM-DAX-12
  • Successors: LDM-503-11a
  • Due: 2020-11-30
  • Completed: 2020-08-31

Scientific and operational data products stored in the consolidated database during ComCam data processing are available for QA assessment of ComCam data through normal analysis access environments.

DM-NCSA-21: L1 Offline Processing Service, Raft Scale, ComCam

  • WBS: 02C.07
  • Level: 3
  • Predecessors: COMC-0100
  • Successors: LSST-1580
  • Due: 2023-02-14
  • Completion pending

Demonstrated running of processing campaigns in offline mode needed for the ComCam phase of commissioning to debug the instrument. Likely includes calibration, ISR, and other pipelines.

02C.08: International Communications and Base Site

_images/graph_02C.08.png

Figure 8 Relationships between milestones in WBS 02C.08 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.

DM-NET-1: Rubin Network Re-engineering

  • WBS: 02C.08.02
  • Level: 2
  • Predecessors: LSST-1440
  • Due: 2015-06-11
  • Completed: 2015-08-05

Warning

No description available

DLP-42: La Serena to Santiago Path Design Ready

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2015-09-28
  • Completed: 2015-04-30

Warning

No description available

DMTC-8100-2010: First Data passed on 100G ring without errors - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2015-10-01
  • Completed: 2015-10-01

Warning

No description available

DMTC-7400-0815: La Serena-Gatehouse path design Ready

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2015-12-15
  • Completed: 2015-12-25

Warning

No description available

DLP-43: Fiber Delivery Ready

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-8000-0760
  • Due: 2015-12-16
  • Completed: 2017-02-28

Warning

No description available

DMTC-7400-2010: Gatehouse to La Serena path design Ready - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-41
  • Due: 2015-12-16
  • Completed: 2015-12-16

Warning

No description available

DLP-41: Mountain - Base Fiber Path Ready

Warning

No description available

DMTC-7400-2170: Gatehouse to Summit contract signed; Design Ready - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-41
  • Due: 2016-01-15
  • Completed: 2016-01-15

Warning

No description available

DMTC-7400-2050: Fiber delivery - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2016-03-15
  • Completed: 2016-03-31

Warning

No description available

DLP-63: Vendor selection Ready

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-7400-2290
  • Due: 2016-06-01
  • Completed: 2016-03-31

Warning

No description available

DMTC-7400-2290: Final Selection of equipment vendor - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-63
  • Due: 2016-06-01
  • Completed: 2016-10-01

Warning

No description available

DMTC-8100-2050: 100G ring operating at 95.0% availability - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2016-09-28
  • Completed: 2016-10-01

Warning

No description available

DMTC-8100-2250: FIU-CIARA Management Contract Delivery #2 Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2016-09-30
  • Completed: 2016-09-30

Warning

No description available

DMTC-8000-0806: 3T Fiber Installation Certification and Acceptance Test - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DLP-45
  • Due: 2017-01-02
  • Completed: 2017-03-15

Warning

No description available

DLP-37: Summits-Gatehouse Fiber Ready

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-7400-2210
  • Due: 2017-02-28
  • Completed: 2017-05-19

Warning

No description available

DMTC-7400-2210: Fiber delivery by Telefonica - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-37
  • Due: 2017-02-28
  • Completed: 2017-02-28

Warning

No description available

DMTC-8200-0480: VLAN design finalized Miami-NCSA - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2017-04-24
  • Completed: 2017-03-23

Warning

No description available

DMTC-7400-2250: End of commissioning and Acceptance - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-40
  • Due: 2017-06-01
  • Completed: 2017-06-30

Warning

No description available

DMTC-8100-2474: Agreement with Monet Cable Partner signed Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2017-06-30
  • Completed: 2017-04-28

Warning

No description available

DMTC-7400-2420: REUNA La Serena - Santiago DWDM equipment - Complete

Warning

No description available

DMTC-7400-2090: Report on functional fiber connections, including AURA equipment - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2017-08-18
  • Completed: 2017-12-01

Warning

No description available

DMTC-7400-2330: LSST and AURA summit to base DWDM equipment - Complete

Warning

No description available

DMTC-8100-2290: FIU-CIARA Management Contract Delivery #3 Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2017-09-29
  • Completed: 2017-09-01

Warning

No description available

DMTC-7400-2480: AURA DWDM equipment in operation - Complete

Warning

No description available

DMTC-8000-0818: Acceptance Test with Equipment Installed - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-7400-2480
  • Due: 2017-11-30
  • Completed: 2017-12-01

Warning

No description available

DMTC-7400-2510: REUNA La Serena - Santiago Optical Channel in operation - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-7400-2480
  • Due: 2017-11-30
  • Completed: 2017-11-01

Warning

No description available

DMTC-7400-2130: End of commissioning. Acceptance of fiber connection - Complete

Warning

No description available

DLP-44: La Serena to Santiago with test traffic AURA DWDM LSST Fiber

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-8000-0826
  • Due: 2018-03-02
  • Completed: 2017-12-31

Warning

No description available

DMTC-8000-0826: Intermediate traffic testing and report - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-44
  • Due: 2018-03-02
  • Completed: 2018-03-02

Warning

No description available

DLP-40: Summits-Gatehouse Initial Capability Ready

Warning

No description available

DLP-65: Mountain - Base LSST and AURA DWM End Node Installation and Test Ready

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-7400-2330
  • Due: 2018-03-14
  • Completed: 2018-03-31

Warning

No description available

DM-NET-2: Mountain - Base Network Functional 2 x 100 Gbps

  • WBS: 02C.08.01
  • Level: 1
  • Predecessors: DLP-559
  • Successors: LDM-503-08, COMC-0620-2050
  • Due: 2018-03-27
  • Completed: 2018-04-02

Warning

No description available

DMTC-8100-2490: First light test report - Complete  Contract

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2018-04-02
  • Completed: 2018-06-01

Warning

No description available

DMTC-8000-0836: La Serena to Santiago with live traffic and uptime over LSST Lambda Ready - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DLP-61
  • Due: 2018-05-11
  • Completed: 2019-07-23

Warning

No description available

DMTC-8100-2090: 100G ring operating at 97.0% availability - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-36
  • Due: 2018-06-29
  • Completed: 2018-06-29

Warning

No description available

DMTC-8200-0510: ESNet Request - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-70, DLP-69
  • Due: 2018-09-26
  • Completed: 2018-09-26

Warning

No description available

DM-NET-3: Initial Network Ready (Summit)

  • WBS: 02C.08.03
  • Level: 2
  • Predecessors: T&SC-2600-1545
  • Successors: LDM-503-08
  • Due: 2018-09-28
  • Completed: 2018-03-05

Warning

No description available

DM-NET-6: Summit LAN installed

  • WBS: 02C.08.00
  • Level: 3
  • Predecessors: T&SC-2600-1545
  • Successors: LDM-503-08
  • Due: 2018-09-28
  • Completed: 2018-04-02

Warning

No description available

DMTC-8100-2330: FIU-CIARA Management Contract Delivery #4 Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2018-10-01
  • Completed: 2018-10-31

Warning

No description available

DMTC-8100-2530: Data passing test report - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-33
  • Due: 2018-10-26
  • Completed: 2019-05-31

Contract Deliverable Report #4 Spectrum Complete. Test of first data passing over Santiago - Boca Raton Spectrum link.

IT-716-M: Base Center Beneficial Occupancy

  • WBS: 02C.08.01
  • Level: 4
  • Predecessors: T&SC-0400-1600
  • Due: 2018-11-26
  • Completed: 2019-03-18

Warning

No description available

IT-718-M: Base Center Full Occupancy

  • WBS: 02C.08.01
  • Level: 4
  • Due: 2019-02-12
  • Completed: 2019-06-30

Externally linked to TS milestone for full occupancy of Base Data Center, including all construction, utilities, access and dust control.

DMTC-8000-0870: Report on functional fiber, including uptime, 1st year functional milestone - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2019-03-22
  • Completed: 2019-07-23

Contract Deliverable Report #8 Report on functional fiber, including uptime, 1st year functional milestone - Complete. Covers first year of operational service of La Serena - Santiago primary fiber, including availability and outage information.

DLP-36: Santiago - Miami Early Integration Ready

Warning

No description available

DMTC-8100-2112: Miami - Boca Raton path diverse fiber - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Successors: DLP-36
  • Due: 2019-04-22
  • Completed: 2019-02-01

Warning

No description available

DMTC-8100-2130: 100G ring operating at 98.0% availability - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2019-05-08
  • Completed: 2019-02-13

Warning

No description available

IT-716-M2: Furniture installed in Offices, purchased cabling received

  • WBS: 02C.08.01
  • Level: 4
  • Due: 2019-05-31
  • Completed: 2019-10-31

Externally linked to TS milestone for installation of furniture of Base Office Building and Base Data Center. All cabling to connect offices to Base LAN has been received.

DMTC-8000-0950: Technical Implementation Plan for 40Gbps diverse path - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2019-06-05
  • Completed: 2019-03-25

Contract Deliverable Report #10 Technical Implementation Plan for 40Gbps diverse path - Complete. Covers the fiber and equipment, ownership/service levels for La Serena - Santiago secondary path.

DM-NET-4: Base LAN installed

  • WBS: 02C.08.03
  • Level: 2
  • Successors: LDM-503-11
  • Due: 2019-07-15
  • Completed: 2020-01-31

Installed the Base LAN equipment in the Base Date Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for ATS observing. (Level 2 equivalent of Level 3 milestone IT-725-M).

IT-725-M: Base Data Center Network Complete

  • WBS: 02C.08.01
  • Level: 4
  • Successors: LDM-503-10
  • Due: 2019-07-29
  • Completed: 2021-08-20

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for ATS observing. (Level 3 equivalent of Level 2 milestone DM-NET-4).

DMTC-8100-2370: FIU-CIARA Management Contract Delivery #5 Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2019-09-30
  • Completed: 2019-12-31

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DLP-70: Miami - Chicago Early Integration Ready

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-8200-0510
  • Due: 2019-10-01
  • Completed: 2019-11-01

Miami - Atlanta, and ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DLP-69: Boca Raton - Chicago Early Integration Ready

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-8200-0510
  • Due: 2019-10-01
  • Completed: 2019-11-14

Boca Raton - Atlanta and ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DMTC-8200-0600: ESNet Implementation Phase 1- Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2019-10-01
  • Completed: 2019-11-14

ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DMTC-8100-2170: 100G ring operating at 99.0% availability - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2020-07-02
  • Completed: 2020-02-19

Contract Deliverable Report #6 100G managed ring Contract Delivery #6 Complete. Covers availability and outages.

DLP-33: Santiago - Boca Raton Full Integration Ready

Spectrum link operational at 100 Gbps from Santiago - Boca Raton, including terrestrial segment from Santiago - Sao Paolo, and Monet submarine cable from Sao Paolo - Boca Raton.

DMTC-6800-1325: Network Verification Pre-Tests

  • WBS: 02C.08.00
  • Level: 4
  • Successors: DLP-33
  • Due: 2020-09-14
  • Completed: 2020-02-28

Warning

No description available

DLP-46: Full Capacity over Diverse path Ready

  • WBS: 02C.08.03
  • Level: 3
  • Predecessors: DMTC-8000-0990, S19F
  • Due: 2020-09-30
  • Completed: 2021-08-31

Both primary (100 Gbps) and secondary (40 - 100 Gbps) circuits from La Serena - Santiago are operational and tested with loads to full capacity.

DMTC-8100-2410: FIU-CIARA Management Contract Delivery #6 Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2020-09-30
  • Completed: 2020-09-30

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DLP-465: Install networking for normal Base Center operations for full-camera scale

  • WBS: 02C.08.01
  • Level: 3
  • Predecessors: F20F
  • Successors: LSST-1580, CAMM8350
  • Due: 2020-11-30
  • Completed: 2020-11-30

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for full camera observing.

DLP-468: Provide networking for formal Chilean DAC production at full-camera scale

  • WBS: 02C.08.02
  • Level: 3
  • Predecessors: DLP-36, F20F
  • Successors: LDM-503-11
  • Due: 2020-11-30
  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with full camera observing.

DLP-467: Install networking for full-camera-scale Chilean DAC serving on order 80 people

  • WBS: 02C.08.02
  • Level: 3
  • Predecessors: F20F
  • Successors: LDM-503-11
  • Due: 2020-11-30
  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with commissioning camera observing.

DM-NET-5: Base - Archive Network Functional 100 Gbps

  • WBS: 02C.08
  • Level: 2
  • Predecessors: DLP-464, COMC-0021
  • Successors: CAMM8330, CAMM8340, COMC-0620-2240
  • Due: 2020-12-21
  • Completed: 2021-02-01

All segments of the Base - Archive Network, including La - Serena - Santiago - Sao Paolo - Miami/Boca Raton - Chicago - Champaign are operationally available at 100 Gbps line rate (effective rate is typically lower, ~ 80%).

DLP-464: Install networking for normal Base Center operations for ComCam scale

  • WBS: 02C.08.01
  • Level: 3
  • Predecessors: TS13, F18F
  • Successors: CAMM8340, DM-NET-5, COMC-060200-20790
  • Due: 2020-12-21
  • Completed: 2020-11-30

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for commissioning camera observing.

DLP-466: Install networking for ComCam-scale Chilean DAC serving on order 20 people

  • WBS: 02C.08.02
  • Level: 3
  • Predecessors: TS13, F18F
  • Successors: LSST-1580, CAMM8330, CAMM8340, COMC-060200-20790
  • Due: 2020-12-21
  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with ATS observing.

DMTC-8100-2210: 100G ring operating at 99.99% availability - Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2021-06-30
  • Completed: 2021-07-30

Contract Deliverable Report #7 100G managed ring Contract Delivery #7 Complete. Covers availability and outages.

DMTC-8100-2450: FIU-CIARA Management Contract Delivery #7 Complete

  • WBS: 02C.08.03
  • Level: 3
  • Due: 2021-09-01
  • Completed: 2021-09-30

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DMTC-6800-1350: Network End-to-End Verification Test

  • WBS: 02C.08.00
  • Level: Undefined
  • Predecessors: SITCOM-130
  • Successors: LSST-1580
  • Due: 2024-01-12
  • Completion pending

Warning

No description available

02C.10: Science Quality and Reliability Engineering

_images/graph_02C.10.png

Figure 9 Relationships between milestones in WBS 02C.10 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.

DLP-340: Operating system testing for public stack releases increased

  • WBS: 02C.10.02
  • Level: 3
  • Successors: DLP-342, DLP-341
  • Due: 2015-08-31
  • Completed: 2016-04-08

Warning

No description available

DLP-334: SQuaRE release improved or changed CI system

  • WBS: 02C.10.02
  • Level: 3
  • Successors: DLP-337
  • Due: 2015-08-31
  • Completed: 2016-04-08

Warning

No description available

DLP-342: Publish updated introductory software guide to developing with the stack

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: DLP-340
  • Due: 2015-12-29
  • Completed: 2016-04-08

Warning

No description available

DLP-341: Deploy stack user support and discussion forum

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: DLP-340
  • Successors: DLP-337
  • Due: 2015-12-29
  • Completed: 2016-04-08

Warning

No description available

DLP-337: Release new stack documentation infrastructure

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: DLP-341, DLP-334
  • Due: 2016-02-24
  • Completed: 2016-05-31

Warning

No description available

DLP-345: Prototype metrics collection system deployed

  • WBS: 02C.10.02
  • Level: 3
  • Due: 2016-02-29
  • Completed: 2016-05-31

Warning

No description available

DLP-351: Intermediate Performance Metrics routinelly calculated and published

  • WBS: 02C.10.02
  • Level: 3
  • Due: 2016-02-29
  • Completed: 2016-05-31

Warning

No description available

DLP-346: SQuaSH Prototype using existing architectural components deployed

  • WBS: 02C.10.02
  • Level: 3
  • Due: 2016-08-31
  • Completed: 2017-07-25

Warning

No description available

DM-SQRE-1: Project internal Jupyter notebook service

  • WBS: 02C.10.02
  • Level: 3
  • Successors: LDM-503-01, DM-SQRE-2
  • Due: 2017-08-31
  • Completed: 2017-11-01

Warning

No description available

DLP-344: Release SQuaSH-in-a-box automated deployment platform

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: F17F
  • Due: 2017-11-30
  • Completed: 2017-11-01

Warning

No description available

DLP-352: Metrics of Interest integrated in running stack or post-tests

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: F17F
  • Successors: S18S
  • Due: 2017-11-30
  • Completed: 2017-11-01

Warning

No description available

DM-SQRE-2: Notebook service capabilities are suitable for the commissioning team to develop notebooks for its needs

Members of the integration and commissioning team are able to develop notebooks in nublado that are capable of the interactive data exploration that is suitable to their needs to support ComCam commissioning. Data is available via a Butler to a mounted filesystem. They are able to work effectively, though in some cases their user experience can be improved. They are primarily trained in person by bootcamps and may require extensive support via Slack. The service may be intermittently unavailable due to engineering work.

DM-SQRE-3: Notebook service stable for commissioning and other internal project users

Members of the integration and commissioning team as well as science pipelines developers use nublado to support their interactive data exploration needs relating to ComCam commissioning. They are able to work effectively though in some cases their user experience can be improved. Access to EFD data is easily available from notebooks. Data is available via Butler to a mounted filesystem. There is a curated library of notebooks modeling common tasks and a way to automatically verify that they are in working condition. Users can refer to documentation for answers to common problems. The service is stable and interruptions to the service are primarily notified well in advance and co-ordinated with the commissioning team to minimize disruption to the extent that the underlying infrastructure permits it.

DM-SQRE-6: IVOA Server-side Operations for Data Access API

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: F20F
  • Successors: LSST-1580, DM-SQRE-7
  • Due: 2020-11-30
  • Completed: 2022-02-28

Warning

No description available

DM-SQRE-7: IVOA-compliant image metadata services

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: F20F, DM-SQRE-6
  • Successors: LSST-1580
  • Due: 2020-11-30
  • Completed: 2022-06-30

Warning

No description available

DM-SQRE-4: Notebook service ready for verification & validation

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: DM-SQRE-3, COMC-060200-10100
  • Successors: LDM-503-14
  • Due: 2023-02-14
  • Completion pending

Science pipeline developers and members of the verification and validation science team use nublado to support their interactive data exploration needs relating to ComCam commissioning. They are able to work effectively though in some cases their user experience can be improved. As well as access to EFD, catalogue and pixel data, they have access to any available APIs to the workflow/batch system for non-trivial data processing. There are curated libraries of continuously integrated notebooks modeling common tasks and they is a reasonable workflow to allow Github-based collaboration on those notebooks. Users can refer to documentation for how to get started and for answers to common problems. The service is stable with minimal interruptions, but may not always scale well to large number of users.

DM-SQRE-5: Notebook service ready for general science users

  • WBS: 02C.10.02
  • Level: 3
  • Predecessors: LSST-1560
  • Successors: LSST-1580
  • Due: 2024-01-12
  • Completion pending

Members of the science collaborations or other non-project staff with access to data use nublado to support their interactive scientific data exploration. They are able to share data with each other using LSP sharing mechanisms and can access data without a direct filesystem mount (eg VO services, remote Butler etc). The service scales well to large number of users provided the infrastructure is sized for them. Users can primarily rely on documentation for support, and use a helpdesk system for questions. There is robust functionality for performing large distributed catalog operations and for visually exploring LSSTCam sized images.

Bibliography

[1][DMTR-302]. Yusra AlSayyad. LDM-503-13a: Science Pipelines Fall 2020 Release Test Plan and Report. 2021. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-302.lsst.io/
[2](1, 2) [LDM-533]. Eric C. Bellm. Level 1 System Software Test Specification. 2017. URL, https://ls.st/LDM-533.
[3][LDM-534]. Jim Bosch, Hsin-Fang Chiang, Michelle Gower, and John D. Swinbank. LSST Level 2 System Test Specification. 2021. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-534.lsst.io/
[4][DMTR-121]. Michelle Butler. LDM-503-8 Spectrograph Data Acquisition Test Plan and Report. 2019. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-121.lsst.io/
[5][DMTR-102]. Michelle Butler. LDM-503-8b (Small Scale CCOB Data Access) Test Plan and Report. 2019. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-102.lsst.io/
[6][DMTR-181]. Michelle Butler. LDM-503-10: DAQ Validation Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-181.lsst.io/
[7][DMTR-182]. Michelle Butler. LDM-503-10b: Large Scale CCOB Data Access Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-182.lsst.io/
[8][DMTR-171]. Michelle Butler. LDM-503-6: ComCam Interface Verification Readiness Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-171.lsst.io/
[9](1, 2) [LDM-538]. Michelle Butler, Jim Parsons, and Michelle Gower. Raw Image Archiving Service Test Specification. 2018. URL, https://ls.st/LDM-538.
[10][DMTR-271]. Jeffrey Carlin. LDM-GEN3: Gen 3 Butler Acceptance Testing Test Plan and Report. 2022. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-271.lsst.io/
[11][LDM-540]. G. P. Dubois-Felsmann, L.P. Guy, J. Carlin, K.S. Krughoff, C. Slater, and M. Wood-Vasey. LSST Science Platform Test Specification. 2020. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-540.lsst.io/
[12][DMTR-161]. Gregory Dubois-Felsmann. LDM-503-10a: LSP with Authentication and TAP Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-161.lsst.io/
[13][DMTR-301]. Gregory Dubois-Felsmann. LDM-503-14a: RSP redeployed on the Interim Data Facility (IDF), ready for DP0.1 Test Plan and Report. 2021. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-301.lsst.io/
[14][LDM-554]. Gregory Dubois-Felsmann, David Ciardi, Fritz Mueller, and Frossie Economou. Data Management LSST Science Platform Requirements. 2019. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-554.lsst.io/
[15][LDM-542]. Gregory Dubois-Felsmann, Frossie Economou, Kian-Tat Lim, Fritz Mueller, Stephen R. Pietrowicz, and Xiuqin Wu. Science Platform Design. 2019. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-542.lsst.io/
[16][DMTR-231]. Robert Gruendl. LDM-503-11a: ComCam OPS Readiness Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-231.lsst.io/
[17](1, 2) [LDM-643]. M. Johnson and R.A. Gruendl. Proposed DM OPS Rehearsals. 2019. URL, https://ls.st/LDM-643.
[18][LSE-163]. M. Jurić, T. Axelrod, A.C. Becker, J. Becla, E. Bellm, J.F. Bosch, D. Ciardi, A.J. Connolly, G.P. Dubois-Felsmann, F. Economou, M. Freemon, M. Gelman, R. Gill, M. Graham, L.P. Guy, Ž. Ivezić, T. Jenness, J. Kantor, K.S. Krughoff, K-T Lim, R.H. Lupton, F. Mueller, D. Nidever, W. O’Mullane, M. Patterson, D. Petravick, D. Shaw, C. Slater, M. Strauss, J. Swinbank, J.A. Tyson, M. Wood-Vasey, and X. Wu. Data Products Definition Document. 2021. Vera C. Rubin Observatory . URL: https://lse-163.lsst.io/
[19][LSE-319]. M. Jurić, D. Ciardi, G.P. Dubois-Felsmann, and L.P. Guy. LSST Science Platform Vision Document. 2019. Vera C. Rubin Observatory . URL: https://lse-319.lsst.io/
[20][DMTR-291]. Simon Krughoff. DM-503-EFDa: EFD on Summit for M1/M3 Test Plan and Report. 2021. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-291.lsst.io/
[21][LSE-400]. K-T Lim. Header Service Interface. 2019. URL: https://lse-400.lsst.io
[22](1, 2) [LDM-148]. K.-T. Lim, J. Bosch, G. Dubois-Felsmann, T. Jenness, J. Kantor, W. O’Mullane, D. Petravick, and The DM Leadership Team. Data Management System Design. 2018. URL, https://ls.st/LDM-148.
[23][RTN-001]. William O’Mullane. Data Preview 0: Definition and planning. 2021. Vera C. Rubin Observatory Technical Note. URL: https://rtn-001.lsst.io/
[24](1, 2, 3, 4, 5, 6) [LDM-564]. William O’Mullane, Frossie Economou, Tim Jenness, and Andrew Loftus. Data Management Software Releases for Verification/Integration. 2018. URL, https://ls.st/LDM-564.
[25](1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31) [LDM-151]. J.D. Swinbank, T. Axelrod, A.C. Becker, J. Becla, E. Bellm, J.F. Bosch, H. Chiang, D.R. Ciardi, A.J. Connolly, G.P. Dubois-Felsmann, F. Economou, M. Fisher-Levine, M. Graham, Ž. Ivezić, M. Jurić, T. Jenness, R.L. Jones, J. Kantor, S. Krughoff, K-T. Lim, R.H. Lupton, F. Mueller, D. Petravick, P.A. Price, D.J. Reiss, D. Shaw, C. Slater, M. Wood-Vasey, X. Wu, P. Yoachim, and _for the LSST Data Management_. Data Management Science Pipelines Design. 2020. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-151.lsst.io/
[26][DMTR-112]. John Swinbank. LDM-503-07 (Camera Data Processing) Test Plan and Report. 2019. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-112.lsst.io/
[27][DMTR-192]. John Swinbank. LDM-503-11b: Science Pipelines Fall 2019 Release Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-192.lsst.io/
[28][DMTR-111]. John D. Swinbank. LDM-503-09a (Science Pipelines Fall 2018 Release) Test Plan and Report. 2019. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-111.lsst.io/