Provenance#

This document was generated based on the contents of the lsst-dm/milestones repository, version a6a16d8f, dated 2024-02-12. This corresponds to the status recorded in the project controls system for December 2023.

Notation#

Throughout this document, the identifiers of completed milestones are set in italics; those of milestones which are still pending, in roman.

Summary#

The DM Subsystem is currently tracking 388 milestones: 7 at Level 1, 84 at Level 2, 272 at Level 3, and 23 at Level 4. 2 have no level defined. Of these, 356 have been completed. Of the incomplete milestones, 6 are late relative to the baseline schedule, while the remainder are scheduled for the future.

_images/burndown.png

Fig. 1 Milestone completion as a function of date.#

Currently overdue milestones#

  • LDM-503-12: Ops rehearsal for commissioning #3 [Due 2023-12-01]

  • LDM-503-18q: Qserv construction complete [Due 2023-12-15]

  • LDM-503-EFDb: EFD/telemetry data replicated at the LDF, stored in InfluxDB. aggregated into parquet files [Due 2023-12-01]

  • DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects. [Due 2023-12-01]

  • DM-DRP-37: Artifact rejection and background matching during coadd construction. [Due 2023-12-01]

  • DM-DAX-15: DAX Construction Complete [Due 2023-12-15]

Milestones by due date#

Due in June 2025#

  • DMTC-6800-1350: Network End-to-End Verification Test

  • LDM-503-16: Ops rehearsal for data processing #3

  • LDM-503-17: Final operations rehearsal

  • LDM-503-17a: Test report: Final Pipelines Delivery

  • DM-READY: DM is ready for ORR

  • LDM-503-19: All 1a,1b DM requirements verified

  • DM-AP-14: Alert filtering system available.

  • DM-AP-16: Full integration of the Alert Production system within the operational environment.

  • DM-SQRE-5: Notebook service ready for general science users

Due in May 2025#

  • No milestones due.

Due in April 2025#

  • No milestones due.

Due in March 2025#

  • LDM-503-RSPb: Science Platform ready for Science Verification

Due in February 2025#

  • LDM-503-14: Test report: DM Readiness for Science Verification Surveys

  • LDM-503-12a: LSSTcam On Sky Readiness

  • LDM-503-EFDc: EFD data is queriable through TAP in the Science Platform

  • RSP-1200: RSP handles 1200 users

  • LDM-503-19a: All P1a DM requirements verified

  • DM-AP-15: Alert distribution system fully integrated.

  • DM-AP-17: Solar System Processing pipeline (SSP) available.

  • DM-DRP-24: Physically motivated PSF model, including separate characterization of contributions from the atmosphere and the telescope system.

  • DM-DRP-31: A photometric redshift is now provided for each object.

  • DM-DRP-36: Solar System Processing integrated to data release processing.

  • DM-DAX-9: Butler Provenance System

Due in January 2025#

  • No milestones due.

Due in December 2024#

  • No milestones due.

Due in November 2024#

  • No milestones due.

Due in October 2024#

  • No milestones due.

Due in September 2024#

Due in August 2024#

  • No milestones due.

Due in July 2024#

  • No milestones due.

Due in June 2024#

  • LDM-503-16a: Test report: Pipelines Release Spring 2023

Due in May 2024#

  • LDM-503-13: Ops rehearsal for data release processing #1

Due in April 2024#

Due in March 2024#

Due in February 2024#

  • DLP-526: Archive Center Integration Complete at SLAC

Due in January 2024#

  • No milestones due.

Due in December 2023#

  • LDM-503-12: Ops rehearsal for commissioning #3

  • LDM-503-11b: Test report: Pipelines Release Fall 2019

  • LDM-503-13a: Test report: Pipelines Release Fall 2020

  • LDM-503-15a: Test report: Pipelines Release Spring 2022

  • LDM-503-EFDb: EFD/telemetry data replicated at the LDF, stored in InfluxDB. aggregated into parquet files

  • LDM-503-18q: Qserv construction complete

  • DM-DRP-19: QA metrics are generated during pipeline execution.

  • DM-DRP-37: Artifact rejection and background matching during coadd construction.

  • DM-DRP-26: Overlap resolution at tract & patch boundaries.

  • DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects.

  • DM-DAX-15: DAX Construction Complete

Due in November 2023#

  • No milestones due.

Due in October 2023#

  • DM-DRP-8: Calibration product generation for the Auxiliary Telescope.

  • DM-DRP-9: Data reduction pipeline for the Auxiliary Telescope.

  • DM-DRP-23: Atmospheric characterization based on data from the Auxiliary Telescope now available.

Due in September 2023#

  • DM-NCSA-15: Batch Processing Service for offline spectrograph data processing

  • DM-NCSA-22: Information in consolidated database available to QA portal

  • DLP-465: Install networking for normal Base Center operations for full-camera scale

  • DLP-468: Provide networking for formal Chilean DAC production at full-camera scale

  • DLP-467: Install networking for full-camera-scale Chilean DAC serving on order 80 people

  • DLP-46: Full Capacity over Diverse path Ready

  • S19F: S19 Complete

  • S20F: S20 Complete

  • S21F: S21 Complete

  • F20F: F20 Complete

  • F21F: F21 Complete

  • LDM-503-05: Test report: Alert distribution validation

  • LDM-503-08: Test report: Spectrograph data acquisition

  • LDM-503-09: Test report: Ops rehearsal for commissioning #1

  • LDM-503-10: Test report: DAQ validation

  • LDM-503-11: Ops rehearsal for commissioning #2

  • LDM-503-15: Ops rehearsal for data release processing #2

  • LDM-503-10b: Large Scale CCOB Data Access

  • LDM-503-09a: Test report: Pipelines Release Fall 2018

  • LDM-503-11a: ComCam Ops Readiness

  • DM-PORTAL: Resume development of LSP portal aspect

  • DM-CLOUD: Decide on final data facility design for DR1

  • LDM-503-14a: Science Platform ready for DP0

  • LDM-GEN3: Gen3 parity demonstrated

  • DM-IT-FIB: Camera Fibers Ready

  • DM-SUIT-7: Mapping between SUIT systems & NCSA auth system

  • DM-SUIT-10: SUIT deployment procedure

  • DM-AP-6: Alert format defined & queue system available.

  • DM-AP-18: Alert packet generation

  • DM-DRP-12: Background estimation over the full visit.

  • DM-DRP-13: PSF estimation over the full visit.

  • DM-DRP-15: All varieties of coadd required for object detection and characterization are now produced during normal pipeline operation (although not necessarily at the ultimately required level of fidelity).

  • DM-DRP-21: Integrated image characterization pipeline for Data Release Production.

  • DM-DRP-22: Template generation integrated with Data Release Production pipelines.

  • DM-DRP-28: Difference images are now a first-class data product during data release processing.

  • DM-DRP-30: Forced photometry is now performed on individual processed visit images during data releases.

  • DM-DRP-34: Selection maps are generated during data releases.

  • DM-DAX-7: Generation 3 Butler, running on a POSIX file-system provided by NCSA

  • DM-DAX-8: Scatter-gather support in PipelineTask

  • DM-DAX-12: Generation 3 middleware at feature parity with Generation 2

  • DM-DAX-13: Generation 2 middleware retired

  • DM-DAX-14: Provenance system review completed

  • DM-SQRE-6: IVOA Server-side Operations for Data Access API

  • DM-SQRE-7: IVOA-compliant image metadata services

Due in August 2023#

  • LDM-503-EFDa: Engineering Facility Database at the summit capturing and enabling access to all telemetry with M1/M3 active

  • NET-neutral: Cisco ACI replaced

Due in July 2023#

  • DM-AP-9: JOINTCAL1: Jointcal at a functional level

  • DM-AP-12: Difference imaging is now agnostic to the PSF of the template image.

  • DM-DRP-17: Simultaneous photometric and astrometric fitting to multiple exposures.

Due in June 2023#

  • DM-NCSA-27: Deliver header service code

  • DM-NCSA-28: IIP control code

  • DM-NCSA-1: Provide regular reprocessing service for HSC data

  • DM-NCSA-2: Provide access to results of regular reprocessing (NB the form this takes depends upon available DAX functionality)

  • DM-NCSA-3: Provide database for metadata, provenance, location and demonstrate ingest at small scale

  • DM-NCSA-29: Deliver IIP image client code

  • DM-NCSA-4: Minimal support for the small operational schema including file metadata and provenance for every file, and record of in

  • DM-NCSA-5: Level 1 archiving system able to acquire pixel data from the Aux Tel DAQ, header metadata via OCS, assemble FITS image,

  • DM-NCSA-6: Ability to transfer files originating from Tucson to NCSA and ingest files at NCSA, including metadata and provenance

  • DM-NCSA-7: Capability to paint displays for Tucson and NCSA

  • DM-NCSA-9: Test instance of alert distribution hosting service and L1 database in Development & Integration Enclave

  • DM-NCSA-10: Sustained archiving service that is OCS commandable

  • DM-NCSA-11: Verified acquisition of raw and crosstalk-corrected exposures at raft scale, incl. correct metadata

  • DM-NCSA-13: Header Writing Service for Spectrograph use case

  • DM-NCSA-14: Data Backbone endpoints in Chile for ingestion and access, distribution over WAN, ingest at NCSA into custodial file sto

  • DM-NCSA-16: Perform ISR processing on ComCam-scale data.

  • DM-NCSA-20: ComCam Archiving Service

  • DM-NCSA-21: L1 Offline Processing Service, Raft Scale, ComCam

  • DLP-864: Coordination with NCSA/CC-IN2P3 JCC

  • DLP-469: Coordination with NCSA/CC-IN2P3 JCC

  • DLP-356: Set up Joint Coordination Council (JCC) for coordination with satellite processing center

  • DLP-355: Establish MOA with CC-IN2P3 satellite processing center

  • DLP-358: Refinement of facility integration plan and traceability of science and operational requirements

  • DLP-357: Initial refinement of facility integration plan

  • DLP-847: Meet camera DAQ/OCS milestones for FY16

  • DLP-368: Initial AP DMCS

  • DLP-366: Mid-scale processing of eligible camera data

  • DLP-364: Build prototype production sequence for the DMCS

  • DLP-361: Build prototype AP DMCS, including events

  • DLP-360: Prepare for OCS Middleware integration with AP DMCS

  • DLP-359: Improve fault tolerance in AP simulator

  • DLP-389: Implement advanced logging functionality, including event-based logging

  • DLP-850: Prototype authentication/authorization system complete

  • DLP-405: Investigate CAS integration with Internet2 AAA tools

  • DLP-404: Identify candidate security packages

  • DLP-412: Monitor development cluster

  • DLP-411: Monitor OpenStack

  • DLP-410: Implement Puppet for system configuration management

  • DLP-415: Evaluate iRODS, Datacat, and similar file system management technologies

  • DLP-400: Refactor basic event services

  • DLP-422: Evaluate OpenStack as a development environment

  • DLP-424: Integration of DevOps to OpenStack

  • DLP-423: DevOps assessment

  • DLP-471: Refresh sizing model

  • DLP-431: Refinement of facility integration plan and traceability of science and operational requirements

  • DLP-430: Initial refinement of facility integration plan

  • DLP-459: Qserv capability to hold data reduced at NCSA

  • DLP-458: Ingest of bulk data from real cameras to ComCam scale

  • DLP-457: Support runs on real cameras data and ImSim data

  • DLP-452: Support datasets on basic file system infrastructure with apropos policies

  • DLP-451: Deploy OpenStack capability

  • DLP-450: Support lsst-dev for developers

  • DLP-449: Upgrade storage server hardware

  • DLP-442: Procure infrastructure for Base Center to meet FY18 goals

  • DLP-441: Base Center requirements document

  • DLP-433: Procure infrastructure for Archive Center to FY18 scale

  • A4369.9: Ship Hardware to BDC

  • A4640: Ship ComCam Hardware to Tucson

  • A4680: June 2019 Travel to Chile - Travel End

  • A4750: Summit AA Systems Installed at the Summit

  • A4860: ComCam Systems in Tucson ready for NCSA

  • A5090: Expected Arrival of ComCam Systems to NCSA

  • DM-NET-5: Base - Archive Network Functional 100 Gbps

  • DLP-464: Install networking for normal Base Center operations for ComCam scale

  • DM-NET-2: Mountain - Base Network Functional 2 x 100 Gbps

  • IT-725-M: Base Data Center Network Complete

  • IT-716-M: Base Center Beneficial Occupancy

  • IT-718-M: Base Center Full Occupancy

  • IT-716-M2: Furniture installed in Offices, purchased cabling received

  • DLP-466: Install networking for ComCam-scale Chilean DAC serving on order 20 people

  • DM-NET-1: Rubin Network Re-engineering

  • DM-NET-3: Initial Network Ready (Summit)

  • DM-NET-4: Base LAN installed

  • DLP-42: La Serena to Santiago Path Design Ready

  • DLP-43: Fiber Delivery Ready

  • DLP-61: La Serena – Santiago link Ready

  • DLP-44: La Serena to Santiago with test traffic AURA DWDM LSST Fiber

  • DLP-45: Delivery of “functional Pre-operations Link” (at least 4Gbps), Ready

  • DMTC-8000-0760: Technical Implementation plan of fiber connection (primary link) - Complete

  • DMTC-8000-0806: 3T Fiber Installation Certification and Acceptance Test - Complete

  • DMTC-8000-0818: Acceptance Test with Equipment Installed - Complete

  • DMTC-8000-0826: Intermediate traffic testing and report - Complete

  • DMTC-8000-0836: La Serena to Santiago with live traffic and uptime over LSST Lambda Ready - Complete

  • DMTC-8000-0870: Report on functional fiber, including uptime, 1st year functional milestone - Complete

  • DMTC-8000-0910: Delivery of 4Gbs link from La Serena to Santiago Ready - Complete

  • DMTC-8000-0950: Technical Implementation Plan for 40Gbps diverse path - Complete

  • DMTC-8000-0990: Full capacity La Serena to Santiago over Primary link AND Secondary link - Complete

  • DMTC-7400-2420: REUNA La Serena - Santiago DWDM equipment - Complete

  • DMTC-7400-2510: REUNA La Serena - Santiago Optical Channel in operation - Complete

  • DMTC-7400-2480: AURA DWDM equipment in operation - Complete

  • DMTC-7400-0815: La Serena-Gatehouse path design Ready

  • DMTC-7400-2010: Gatehouse to La Serena path design Ready - Complete

  • DMTC-7400-2050: Fiber delivery - Complete

  • DMTC-7400-2090: Report on functional fiber connections, including AURA equipment - Complete

  • DMTC-7400-2130: End of commissioning. Acceptance of fiber connection - Complete

  • DLP-41: Mountain - Base Fiber Path Ready

  • DLP-37: Summits-Gatehouse Fiber Ready

  • DLP-40: Summits-Gatehouse Initial Capability Ready

  • DMTC-7400-2170: Gatehouse to Summit contract signed; Design Ready - Complete

  • DMTC-7400-2210: Fiber delivery by Telefonica - Complete

  • DMTC-7400-2250: End of commissioning and Acceptance - Complete

  • DLP-65: Mountain - Base LSST and AURA DWM End Node Installation and Test Ready

  • DLP-63: Vendor selection Ready

  • DMTC-7400-2290: Final Selection of equipment vendor - Complete

  • DMTC-7400-2330: LSST and AURA summit to base DWDM equipment - Complete

  • DLP-36: Santiago - Miami Early Integration Ready

  • DMTC-8100-2010: First Data passed on 100G ring without errors - Complete

  • DMTC-8100-2050: 100G ring operating at 95.0% availability - Complete

  • DMTC-8100-2090: 100G ring operating at 97.0% availability - Complete

  • DMTC-8100-2130: 100G ring operating at 98.0% availability - Complete

  • DMTC-8100-2170: 100G ring operating at 99.0% availability - Complete

  • DMTC-8100-2210: 100G ring operating at 99.99% availability - Complete

  • DMTC-8100-2112: Miami - Boca Raton path diverse fiber - Complete

  • DMTC-8100-2250: FIU-CIARA Management Contract Delivery #2 Complete

  • DMTC-8100-2290: FIU-CIARA Management Contract Delivery #3 Complete

  • DMTC-8100-2330: FIU-CIARA Management Contract Delivery #4 Complete

  • DMTC-8100-2370: FIU-CIARA Management Contract Delivery #5 Complete

  • DMTC-8100-2410: FIU-CIARA Management Contract Delivery #6 Complete

  • DMTC-8100-2450: FIU-CIARA Management Contract Delivery #7 Complete

  • DLP-33: Santiago - Boca Raton Full Integration Ready

  • DMTC-8100-2490: First light test report - Complete  Contract

  • DMTC-8100-2530: Data passing test report - Complete

  • DMTC-8100-2474: Agreement with Monet Cable Partner signed Complete

  • DLP-70: Miami - Chicago Early Integration Ready

  • DLP-69: Boca Raton - Chicago Early Integration Ready

  • DMTC-8200-0480: VLAN design finalized Miami-NCSA - Complete

  • DMTC-8200-0510: ESNet Request - Complete

  • DMTC-8200-0600: ESNet Implementation Phase 1- Complete

  • DM-NET-6: Summit LAN installed

  • DMTC-6800-1325: Network Verification Pre-Tests

  • DLP-533: Base Center Integration Complete

  • DLP-559: Mountain - Base Network Functional 1 Gbps

  • DLP-541: LSE-68 Camera-DM Image data interfaces - Phase 3

  • DLP-544: LSE-72 OCS-DM master interface - Phase 3

  • DLP-552: LSST Software Release 9.1 Complete, Ready for Full Camera

  • DLP-551: LSST Software Release 8.1 Complete, Ready for Commissioning Camera

  • DLP-558: LSST Software Release 7.1 Complete

  • DLP-554: LSST Software Release 4.0 Complete

  • DLP-556: LSST Software Release 5.1 Complete

  • DLP-557: LSST Software Release 6.1 Complete

  • DLP-538: LSE-130 Data items Exchanged between Cam & DM - Phase 3

  • DLP-555: LSST Software Release 5.0 Complete

  • DLP-566: LSST Software Release 6.0 Complete

  • DLP-567: LSST Software Release 7.0 Complete

  • DMTC-0000-0020: LSST Software Release X16 Complete

  • DMTC-0000-0030: LSST Software Release F16 Complete

  • DMTC-0000-3000: LSST Software Release S17 Complete

  • S17F: S17 Complete

  • S18F: S18 Complete

  • S22F: S22 Complete

  • F17F: F17 Complete

  • F19F: F19 Complete

  • F18F: F18 Complete

  • LDM-503-01: Test report: Science Platform with WISE data in PDAC

  • LDM-503-02: Test report: HSC reprocessing

  • LDM-503-03: Test report: Alert generation validation

  • LDM-503-04: Test report: Aux Tel DAQ integration functionality test

  • LDM-503-06: Test report: DM ComCam interface verification readiness

  • LDM-503-07: Test report: Camera data processing

  • LDM-503-04b: Test report: Aux Tel DAQ interface Integration Verification and Spectrograph Operations Rehearsal

  • LSST-1160: Header Service Integration

  • LSST-1190: Control Interfaces Verification: Single Visit

  • LSST-1200: Interface Verification: Single Visit

  • LSST-1220: EFD ETL Under DM Control

  • DM-STAFF: Staffing Checkpoint

  • LDM-503-08b: Small Scale CCOB Data Access

  • LDM-503-10a: Science Platform: TAP service with federated SSO A&A

  • DLP-579: Usability and developer efficiency

  • DM-SUIT-1: Search and display WISE sources (objects) in PDAC

  • DM-SUIT-2: Search WISE coaded data single exposure images in PDAC (the images are from IRSA at IPAC, not NCSA)

  • DM-SUIT-3: Time series analysis tool for WISE data

  • DM-SUIT-4: Multiple data traces in chart space

  • DM-SUIT-5: Search and display processed HSC data

  • DM-SUIT-8: SUIT portal integrated with workspace

  • DLP-578: Propose a workspace structure for discussion.

  • DLP-256: First draft definition of workspace

  • DLP-518: Evaluate best-of-breed tools. Technology exploration.

  • DLP-872: Convert major portion of GWT in Firefly to pure JavaScript

  • DLP-871: Firefly server side code refactor

  • DLP-574: Evaluate best-of-breed tools. Technology exploration.

  • DLP-564: Able to use some data-access APIs developed by database team

  • DLP-521: Define the data access APIs with database team

  • DLP-520: Access Qserv for catalog search in SUI prototype

  • DLP-519: Research the development framework for SUI

  • DLP-207: Extra LSST query capabilities in web UI

  • DLP-205: LSST web UI search external catalog

  • DLP-575: Evaluate best-of-breed tools. Technology exploration.

  • DLP-222: Provide time-domain data view

  • DLP-218: Provide API for image display in Firefly

  • DLP-216: Provide basic APIs to use Firefly visualization components

  • DLP-576: Evaluate best-of-breed tools. Technology exploration.

  • DLP-577: Evaluate best-of-breed tools. Technology exploration.

  • DM-AP-1: Basic single frame measurement pipeline.

  • DM-AP-2: Alard & Lupton-style image differencing.

  • DM-AP-3: Point source & dipole measurement on difference images.

  • DM-AP-4: DIASource association

  • DM-AP-5: DIAObject generation

  • DM-AP-7: Basic instrument signature removal (ISR) capability.

  • DM-AP-8: Advanced ISR, including ghost and linear feature masking, correction for the Brighter-Fatter effect and compensation for pixel response non-uniformity.

  • DM-AP-10: Advanced single frame measurement pipeline for Alert Production.

  • DM-AP-11: Difference imaging includes noise decorrelation and correction for differential chromatic refraction.

  • DM-AP-13: Trailed source measurement on difference images.

  • DLP-276: Dipole measurement suite

  • DLP-275: Deliver document on effects of DCR

  • DLP-642: Camera Team using DM code

  • DLP-639: Re-design camera geometry classes

  • DLP-644: Generalized Single Frame Astrometry Pipeline

  • DLP-638: Refactor AFW R&D Code

  • DM-DRP-1: HSC merger complete: all functionality deployed for the most recent HSC data release processing is now available within the LSST stack.

  • DM-DRP-2: Basic visualization and quality assessment tools operational on HSC-scale data volumes.

  • DM-DRP-3: PSF-homogenized coadd construction.

  • DM-DRP-4: Calibration product generation in support of basic ISR.

  • DM-DRP-5: Camera package supporting the LSST Camera.

  • DM-DRP-6: Camera package supporting the Auxiliary Telescope.

  • DM-DRP-7: Coordinate transformation tool provided for use with the Collimated Beam Projector.

  • DM-DRP-11: Pipelines code provides supports for database ingestion of results.

  • DM-DRP-14: Insertion of simulated sources into the data stream to check pipeline performance.

  • DM-DRP-16: Global photometric fitting (e.g. Burke et al. Forward Global Calibration Method).

  • DM-DRP-18: Initial multi-band deblending algorithm available.

  • DM-DRP-20: Refined set of LSST calibration products.

  • DM-DRP-29: Moving point source model fitting now available.

  • DM-DRP-32: Object classification system available.

  • DM-DRP-35: Simultaneous measurement across a suite of coadds representing different bandpasses, epocs, and flavors.

  • DM-DRP-38: Camera package supporting the Commissioning Camera.

  • DLP-486: Prototype merge & deduplication in tract & patch overlaps

  • DLP-103: Prototype multi-coadd peak association available

  • DLP-101: Prototype coadd detection available

  • DLP-100: Consistent multi-band deblending available

  • DLP-736: Performance and capability enhancements in support of catalog use cases

  • DLP-588: AFW R&D code refactored

  • DLP-288: HSC merger complete

  • DLP-73: Footprints redesigned and refactored

  • DLP-694: Prototype Brighter-Fatter correction available

  • DLP-724: Forced per-epoch photometry

  • DLP-716: Fundamental flux measurements

  • DLP-715: Fundamental centroid and shape measurements

  • DLP-700: Galaxy shear fitting parameters available

  • DLP-697: Single epoch CModel minimization

  • DLP-696: Single epoch Sersic model fitting

  • DLP-503: Prototype direct coaddition

  • DLP-502: Prototype image artifact identification available

  • DLP-152: Initial PSF-matched image coaddition available

  • DM-DAX-1: WISE data ingest to PDAC

  • DM-DAX-2: Query service supporting IVOA TAP protocol, w/ support for asynchronous queries

  • DM-DAX-5: Database ingest in support of HSC reprocessing (ie, large catalog ingest)

  • DM-DAX-6: Prototype level 1 database

  • DM-DAX-11: Transformed EFD schema

  • DM-DAX-10: Per-object gather support in Butler

  • DLP-836: New logging for DM

  • DLP-25: Prototype Web Service for SUI

  • DLP-837: AP-ready Butler Framework

  • DLP-835: New xrootd client

  • DLP-834: Automated Testbed for Qserv

  • DLP-818: CSS

  • DLP-817: Multi-node Multi-query Integration Testing Harness

  • DLP-805: Distributed Database and Table Management

  • DLP-801: Qserv uses C++-based Geometry Code

  • DLP-649: Shared Scans

  • DLP-635: Qserv Query Cancellation

  • DLP-473: Async Queries in Qserv

  • DLP-472: Qserv Data Distribution

  • DLP-802: Alert Production Database (APDB) design

  • DLP-800: Data Ingest into MySQL

  • DLP-725: Working Prototype of AP/L1 Database

  • DLP-346: SQuaSH Prototype using existing architectural components deployed

  • DLP-345: Prototype metrics collection system deployed

  • DLP-344: Release SQuaSH-in-a-box automated deployment platform

  • DLP-342: Publish updated introductory software guide to developing with the stack

  • DLP-341: Deploy stack user support and discussion forum

  • DLP-340: Operating system testing for public stack releases increased

  • DLP-352: Metrics of Interest integrated in running stack or post-tests

  • DLP-351: Intermediate Performance Metrics routinelly calculated and published

  • DLP-337: Release new stack documentation infrastructure

  • DLP-334: SQuaRE release improved or changed CI system

  • DM-SQRE-1: Project internal Jupyter notebook service

  • DM-SQRE-2: Notebook service capabilities are suitable for the commissioning team to develop notebooks for its needs

  • DM-SQRE-3: Notebook service stable for commissioning and other internal project users

Due in May 2023#

  • No milestones due.

Due in April 2023#

  • No milestones due.

Due in March 2023#

  • No milestones due.

Due in February 2023#

  • No milestones due.

Due in January 2023#

  • No milestones due.

Due in December 2022#

  • No milestones due.

Due in November 2022#

  • No milestones due.

Due in October 2022#

  • A9440: START: Data Security

Due in September 2022#

  • No milestones due.

Due in August 2022#

  • No milestones due.

Due in July 2022#

  • No milestones due.

Due in June 2022#

  • No milestones due.

Due in May 2022#

  • No milestones due.

Due in April 2022#

  • No milestones due.

Due in March 2022#

  • No milestones due.

Due in February 2022#

  • No milestones due.

Due in January 2022#

  • No milestones due.

Due in December 2021#

Due in November 2021#

  • No milestones due.

Due in October 2021#

  • No milestones due.

Due in September 2021#

  • No milestones due.

Due in August 2021#

  • No milestones due.

Due in July 2021#

  • No milestones due.

Due in June 2021#

Due in May 2021#

  • No milestones due.

Due in April 2021#

Due in March 2021#

  • No milestones due.

Due in February 2021#

  • No milestones due.

Due in January 2021#

  • No milestones due.

Due in December 2020#

  • No milestones due.

Due in November 2020#

  • No milestones due.

Due in October 2020#

  • No milestones due.

Due in September 2020#

  • No milestones due.

Due in August 2020#

  • No milestones due.

Due in July 2020#

  • No milestones due.

Due in June 2020#

Due in May 2020#

  • No milestones due.

Due in April 2020#

  • No milestones due.

Due in March 2020#

  • No milestones due.

Due in February 2020#

  • No milestones due.

Due in January 2020#

  • A6150: Start Milestone: January 1st, 2020

Due in December 2019#

Due in November 2019#

  • No milestones due.

Due in October 2019#

  • A1300: Start Milestone: October 1st, 2019

Due in September 2019#

  • No milestones due.

Due in August 2019#

  • A4670: June 2019 Travel to Chile - Travel Start

Due in July 2019#

  • No milestones due.

Due in June 2019#

  • A2330: Ship AA System to Chile

  • F19S: F19 Start

Due in May 2019#

  • No milestones due.

Due in April 2019#

  • No milestones due.

Due in March 2019#

  • No milestones due.

Due in February 2019#

  • No milestones due.

Due in January 2019#

  • No milestones due.

Due in December 2018#

Due in November 2018#

  • No milestones due.

Due in October 2018#

  • No milestones due.

Due in September 2018#

  • No milestones due.

Due in August 2018#

  • No milestones due.

Due in July 2018#

  • No milestones due.

Due in June 2018#

Due in May 2018#

  • No milestones due.

Due in April 2018#

  • No milestones due.

Due in March 2018#

  • No milestones due.

Due in February 2018#

  • No milestones due.

Due in January 2018#

  • No milestones due.

Due in December 2017#

Due in November 2017#

  • No milestones due.

Due in October 2017#

  • No milestones due.

Due in September 2017#

  • No milestones due.

Due in August 2017#

  • No milestones due.

Due in July 2017#

  • No milestones due.

Due in June 2017#

Due in May 2017#

  • No milestones due.

Due in April 2017#

  • No milestones due.

Due in March 2017#

  • No milestones due.

Due in February 2017#

  • No milestones due.

Due in January 2017#

  • No milestones due.

Due in December 2016#

Due in November 2016#

  • No milestones due.

Due in October 2016#

  • No milestones due.

Due in September 2016#

  • No milestones due.

Due in August 2016#

  • No milestones due.

Due in July 2016#

  • No milestones due.

Due in June 2016#

  • No milestones due.

Due in May 2016#

  • No milestones due.

Due in April 2016#

  • No milestones due.

Due in March 2016#

  • No milestones due.

Due in February 2016#

  • No milestones due.

Due in January 2016#

  • No milestones due.

Due in December 2015#

  • No milestones due.

Due in November 2015#

  • No milestones due.

Due in October 2015#

  • DLP-528: Archive Site Ready for Equipment Configuration

Milestones by WBS#

02C.00: Data Management Level 2 Milestones#

_images/graph_02C.00.png

Fig. 2 Relationships between milestones in WBS 02C.00 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-528: Archive Site Ready for Equipment Configuration#

  • WBS: 02C.00

  • Level: 1

  • Due: 2015-10-01

  • Completed: 2015-10-01

This is complete in. data/pmcs/201904-ME.xls.

S17S: S17 Start#

Warning

No description available

F17S: F17 Start#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: DLP-736

  • Successors: F17F

  • Due: 2017-06-01

  • Completed: 2017-06-01

Warning

No description available

S18S: S18 Start#

Warning

No description available

F18S: F18 Start#

Warning

No description available

S19S: S19 Start#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S18S

  • Successors: S20S

  • Due: 2018-12-01

  • Completed: 2018-12-01

Warning

No description available

F19S: F19 Start#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: F18S

  • Successors: F19F

  • Due: 2019-06-01

  • Completed: 2019-06-01

Warning

No description available

S20S: S20 Start#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S19S, S19F, F19F

  • Successors: S21S, S20F

  • Due: 2019-12-02

  • Completed: 2019-12-02

Warning

No description available

F20S: F20 Start#

Time-based milestone marking the start of the Fall 2020 development cycle.

S21S: S21 Start#

Warning

No description available

F21S: F21 Start#

Warning

No description available

S22S: S22 Start#

Warning

No description available

DLP-533: Base Center Integration Complete#

  • WBS: 02C.00

  • Level: 1

  • Successors: COMC-1564, COMC-0620-2090

  • Due: 2023-06-01

  • Completed: 2020-02-28

This milestone represents the availability of the base center in Las Serena. This includes DM staging machines, the commissioning cluster. It also includes the Chilean Data Access center machines. The later will only be deployed after the US DAC is up.

DLP-559: Mountain - Base Network Functional 1 Gbps#

  • WBS: 02C.00

  • Level: 1

  • Successors: DM-NET-2

  • Due: 2023-06-01

  • Completed: 2015-06-01

Warning

No description available

DLP-541: LSE-68 Camera-DM Image data interfaces - Phase 3#

  • WBS: 02C.00

  • Level: 2

  • Successors: S17S

  • Due: 2023-06-01

  • Completed: 2016-10-31

Complete with LCR-1788 June 2019.

DLP-544: LSE-72 OCS-DM master interface - Phase 3#

  • WBS: 02C.00

  • Level: 2

  • Successors: DLP-558

  • Due: 2023-06-01

  • Completed: 2016-10-31

Warning

No description available

DLP-552: LSST Software Release 9.1 Complete, Ready for Full Camera#

This pipeline software release will precede LDM-503-12a. This will have all functionality required to process data from LSSTCam. It will not have the full functionality for data release processing.

DLP-551: LSST Software Release 8.1 Complete, Ready for Commissioning Camera#

  • WBS: 02C.00

  • Level: 1

  • Predecessors: DLP-558

  • Successors: DLP-552

  • Due: 2023-06-01

  • Completed: 2018-07-12

Warning

No description available

DLP-558: LSST Software Release 7.1 Complete#

Warning

No description available

DLP-554: LSST Software Release 4.0 Complete#

  • WBS: 02C.00

  • Level: 2

  • Successors: DLP-556

  • Due: 2023-06-01

  • Completed: 2014-09-30

Warning

No description available

DLP-556: LSST Software Release 5.1 Complete#

Warning

No description available

DLP-557: LSST Software Release 6.1 Complete#

Warning

No description available

DLP-538: LSE-130 Data items Exchanged between Cam & DM - Phase 3#

approval of LCR-1494 in June 2019 allows this to be closed, LSE-400 [Lim, 2019] covers the items.

DLP-555: LSST Software Release 5.0 Complete#

  • WBS: 02C.00

  • Level: 2

  • Due: 2023-06-01

  • Completed: 2015-03-06

Warning

No description available

DLP-566: LSST Software Release 6.0 Complete#

Warning

No description available

DLP-567: LSST Software Release 7.0 Complete#

  • WBS: 02C.00

  • Level: 2

  • Due: 2023-06-01

  • Completed: 2017-02-28

Warning

No description available

DMTC-0000-0020: LSST Software Release X16 Complete#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: DLP-694

  • Due: 2023-06-01

  • Completed: 2016-07-12

Warning

No description available

DMTC-0000-0030: LSST Software Release F16 Complete#

Warning

No description available

DMTC-0000-3000: LSST Software Release S17 Complete#

Warning

No description available

S17F: S17 Complete#

Warning

No description available

S18F: S18 Complete#

Warning

No description available

S22F: S22 Complete#

  • WBS: 02C.00

  • Level: 2

  • Successors: LSST-1590, DM11, LDM-503-15a

  • Due: 2023-06-01

  • Completed: 2022-05-31

Time-based milestone marking the completion of the Spring 2022 development cycle.

F17F: F17 Complete#

Warning

No description available

F19F: F19 Complete#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: F19S

  • Successors: F20S, S20S, DM-STAFF

  • Due: 2023-06-01

  • Completed: 2019-11-29

Time-based milestone marking the completion of the Fall 2019 development cycle.

F18F: F18 Complete#

Warning

No description available

LDM-503-01: Test report: Science Platform with WISE data in PDAC#

This test demonstrates the deployment of the Portal and API Aspects of the LSST Science Platform (LSE-319 [Jurić et al., 2019], LDM-542 [Dubois-Felsmann et al., 2019], LDM-554 [Dubois-Felsmann et al., 2019]) to serve all-sky precursor data, including catalogs totaling of order 50–100 billion rows, in a prototype LSST Data Access Center hosted in the LSST Data Facility at NCSA.

LDM-503-02: Test report: HSC reprocessing#

This test demonstrates the successful execution of a prototype Data Release Production science payload (LDM-148 [Lim et al., 2020], LDM-151 [Swinbank et al., 2020]), processing data from precursor surveys at a relatively small scale using compute resources at the LSST Data Facility at NCSA.

LDM-503-03: Test report: Alert generation validation#

This test demonstrates the successful execution of a prototype Alert Generation science payload (LDM-148 [Lim et al., 2020], LDM-151 [Swinbank et al., 2020]), processing data from precursor surveys at a relatively small scale using compute resources at the LSST Data Facility at NCSA.

LDM-503-04: Test report: Aux Tel DAQ integration functionality test#

This test demonstrates that LSST data acquisition system can capture raw data from instrumentation and construct a well-formed image with proper headers.

LDM-503-06: Test report: DM ComCam interface verification readiness#

This milestone records successful transfer of an image equivalent to one raft from the DAQ at the summit to reliable storage in the LSST Data Facility at NCSA, from where it will be made available for scientific evaluation through the LSST Science Platform.

LDM-503-07: Test report: Camera data processing#

This test demonstrates that the LSST Science Pipelines can be successfully used to load and perform basic processing of data from the LSST Camera test systems. In the process, it establishes that data is properly made available at the LSST Data Facility at NCSA, and is accessible through the LSST Science Platform.

LDM-503-04b: Test report: Aux Tel DAQ interface Integration Verification and Spectrograph Operations Rehearsal#

This test demonstrates that the well-formed image generated in LDM-503-04 can be written to the permanent record of the survey.

LSST-1160: Header Service Integration#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: DM-NCSA-27

  • Successors: LSST-1190

  • Due: 2023-06-01

  • Completed: 2018-03-31

Warning

No description available

LSST-1190: Control Interfaces Verification: Single Visit#

Warning

No description available

LSST-1200: Interface Verification: Single Visit#

Warning

No description available

LSST-1220: EFD ETL Under DM Control#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: LSST-1190

  • Successors: LSST-1580

  • Due: 2023-06-01

  • Completed: 2019-03-29

Warning

No description available

DM-STAFF: Staffing Checkpoint#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: F19F

  • Successors: LDM-503-13

  • Due: 2023-06-01

  • Completed: 2019-11-29

DM Management evaluate current DM staffing against project schedule and budget. If there is a shortfall through mid-2022 (the “ops gap”), this is the point at which we need to fill it.

LDM-503-08b: Small Scale CCOB Data Access#

This test demonstrates the integration of the CCOB with the permanent record of the survey.

LDM-503-10a: Science Platform: TAP service with federated SSO A&A#

This test demonstrates the successful integration of a single-sign-on federated authentication system, and a basic authorization system, with the three Aspects of the LSST Science Platform (Portal, Notebook, and API), with the API Aspect containing at least a TAP service.

DLP-579: Usability and developer efficiency#

  • WBS: 02C.00

  • Level: 3

  • Successors: DLP-557

  • Due: 2023-06-01

  • Completed: 2017-12-01

Warning

No description available

NET-neutral: Cisco ACI replaced#

  • WBS: 02C.00

  • Level: Undefined

  • Successors: LDM-503-NET

  • Due: 2023-08-15

  • Completed: 2023-01-31

Warning

No description available

LDM-503-EFDa: Engineering Facility Database at the summit capturing and enabling access to all telemetry with M1/M3 active#

An instance of the Engineering and Facility Database at the summit capturing and enabling access to all telemetry, including from an active M1M3. This may be distinct from the Auxiliary Telescope EFD instance. Demonstrate the ability to run for five days, allowing data retrieval and plotting via notebooks.

LDM-503-15: Ops rehearsal for data release processing #2#

This milestone builds upon LDM-503-13, and will demonstrate the production of a data release under simulated operational conditions using data from science verification activities.

S19F: S19 Complete#

Time-based milestone marking the completion of the Spring 2019 development cycle.

S20F: S20 Complete#

Time-based milestone marking the completion of the Spring 2020 development cycle.

S21F: S21 Complete#

Time-based milestone marking the completion of the Spring 2021 development cycle.

F20F: F20 Complete#

Time-based milestone marking the completion of the Fall 2020 development cycle.

F21F: F21 Complete#

Time-based milestone marking the completion of the Fall 2021 development cycle.

LDM-503-05: Test report: Alert distribution validation#

This test demonstrates that a full-LSST-scale (10,000 messages every 39 seconds) alert stream can be distributed to end users. It further shows that simple filters based on alert contents can be applied to that stream.

LDM-503-08: Test report: Spectrograph data acquisition#

This test demonstrates data acquisition from the Auxiliary Telescope spectrograph, transmission of all image files with associated metadata to NCSA for screening, and then further processing if needed.

LDM-503-09: Test report: Ops rehearsal for commissioning #1#

This milestone represents the completion of a multi-day exercise in which (simulated) observations are performed and data is processed in as close to an operational mode as is possible at this stage in construction. In particular, it will demonstrate transfer, archiving and ingestion of raw data; offline processing of calibration and science data; and curation of the resultant data products.

LDM-503-10: Test report: DAQ validation#

Test of the extended data acquisition (DAQ) network in Chile from the Summit to the Base Data Center.

LDM-503-11: Ops rehearsal for commissioning #2#

Template generation, including code required to build templates in both Data Release and Alert Production science payloads, can now product templates which are corrected for the effects of Differential Chromatic Refraction (DCR), as described in LDM-151 [Swinbank et al., 2020] §6.18. Further, noise in image differences can be “decorrelated”, as described in LDM-151 [Swinbank et al., 2020] §6.19.1. Both of these represents initial capabilities, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

LDM-503-10b: Large Scale CCOB Data Access#

Demonstrate the ability to transfer data from the CCOB with 21 rafts from SLAC and ingested at NCSA and make available through an instance of the LSP.

LDM-503-09a: Test report: Pipelines Release Fall 2018#

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2018 and early 2019. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-503-11a: ComCam Ops Readiness#

Demonstrate that all DM functionality required to support ComCam operations has been delivered.

DM-PORTAL: Resume development of LSP portal aspect#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S22S

  • Successors: LSST-1580

  • Due: 2023-09-05

  • Completed: 2022-01-31

Warning

No description available

DM-CLOUD: Decide on final data facility design for DR1#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S22S

  • Successors: LSST-1580

  • Due: 2023-09-05

  • Completed: 2022-01-31

Warning

No description available

LDM-503-14a: Science Platform ready for DP0#

The Rubin Science Platform is ready to support Data Preview 0.1 as deployed on the IDF. The RSP capabilities are largely those demonstrated under previous milestones, but ported to the IDF and with deployment and authentication/authorization infrastructure improvements that support the scale of DP0.1 usage. The Notebook Aspect provides access to the Rubin Science Pipelines stack and to data exposed in a cloud-deployed Gen3 Butler repository; the API Aspect provides TAP queries against a Qserv deployment of catalog tables, but not image metadata; and the Portal Aspect provides a UI for visual and ADQL construction of queries against those tables. Image and image metadata access services in the API and Portal Aspects are not included.

LDM-GEN3: Gen3 parity demonstrated#

Show parity of Butler Gen3 with Butler Gen2 systems. We should be able to run processing pipelines with Gen3 which previously ran on Gen2. Demonstrate the ability to write new pipelines using the framework. Framework should be ready to support Preops processing, specifically DP0.2 (RTN-001 [O'Mullane, 2024]).

DM-IT-FIB: Camera Fibers Ready#

  • WBS: 02C.00

  • Level: 2

  • Predecessors: S21F

  • Successors: COMC-060200-20100

  • Due: 2023-09-05

  • Completed: 2021-10-29

Camera fibers ready deployed from the Computer Room to the top of the TMA.

LDM-503-12: Ops rehearsal for commissioning #3#

This milestone builds upon LDM-503-11, and serves to demonstrate that all necessary procedures and workflows are in place to support commissioning activities.

LDM-503-EFDb: EFD/telemetry data replicated at the LDF, stored in InfluxDB. aggregated into parquet files#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-26344]

  • Predecessors: LDM-503-EFDa, COMC-0030

  • Successors: LSST-1520

  • Due: 2023-12-01

  • Completion pending [DM-26347]

Demonstrate that EFD data from the Summit appears at a Data Facility. Show that the Parquet files generated contain all low frequency EFD values and summaries of high frequency values.

LDM-503-11b: Test report: Pipelines Release Fall 2019#

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2019 and early 2020. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-503-13a: Test report: Pipelines Release Fall 2020#

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2020 and early 2021. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-503-15a: Test report: Pipelines Release Spring 2022#

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2021 and early 2022. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-503-18q: Qserv construction complete#

  • WBS: 02C.00

  • Level: 4

  • Test specification: Undefined [DM-38758]

  • Successors: DM-READY

  • Due: 2023-12-15

  • Completion pending [DM-38759]

This milestone represents the point at which Qserv completes construction and enters full operation and maintenance. A testplan should be executed showing the requirements are verified.

DLP-526: Archive Center Integration Complete at SLAC#

  • WBS: 02C.00

  • Level: 1

  • Successors: DM2, COMC-1664

  • Due: 2024-02-29

  • Completion pending

This milestone represents the availability of an archive center to host LSST DR1 data. This includes an science platform interface with the implied services for Data Access, a portal and Notebooks.

LDM-503-NET: Network Verification End-to-End#

Network verified.

LDM-503-RSPa: Science Platform ready for DP0.2 with Image Services#

The Rubin Science Platform is ready to support Data Preview 0.2 as deployed on the IDF. The RSP serves an end-to-end Rubin-Gen3-processed dataset with a substantial set of image and SDM-ified catalog data products. Beyond the capabilities provided under LDM-503-14a, this deployment provides: initial IVOA-oriented image metadata and image services in the API Aspect, specifically an ObsTAP service and a basic SODA-compatible image cutout service; Portal Aspect support for accessing these services and browsing images from the released dataset; and access to a User File Workspace from all three Aspects.

LDM-503-13: Ops rehearsal for data release processing #1#

This milestone builds upon LDM-503-12, but with its scope extended beyond procedures which are necessary in commissioning to cover the operational era.

LDM-503-16a: Test report: Pipelines Release Spring 2023#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-32560]

  • Predecessors: COMC-0240

  • Successors: LDM-503-17a

  • Due: 2024-06-07

  • Completed: 2023-08-31 [DM-32561]

This milestone describes a software release of the LSST Science Pipelines in support of activities in late 2023 and early 2023. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

LDM-503-14: Test report: DM Readiness for Science Verification Surveys#

This milestone represents the point at which all code and/or services which are supplied by DM in support of science verification will be tested and demonstrated to meet their requirements.

LDM-503-12a: LSSTcam On Sky Readiness#

Demonstrate that all DM functionality required to support operations with the full LSST Camera has been delivered.

LDM-503-EFDc: EFD data is queriable through TAP in the Science Platform#

Demonstrate that engineering data for a specific observation time can be retrieved via TAP on the Science Platform or directly to a TAP tool like Topcat. Test DMS-REQ-0102 (Priority 1a) to permanently archive metadata from OCS and make available to rights holders. Also test DMS-REQ-0358 (Priority 1a), demonstrating support of simultaneous queries.

RSP-1200: RSP handles 1200 users#

  • WBS: 02C.00

  • Level: 4

  • Predecessors: LSST-1520

  • Successors: LDM-503-17

  • Due: 2025-02-12

  • Completed: 2023-09-29

Warning

No description available

LDM-503-19a: All P1a DM requirements verified#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-38727]

  • Predecessors: LSST-1520

  • Successors: LSST-1580, LDM-503-14, SITCOM-114

  • Due: 2025-02-12

  • Completion pending [DM-38728]

This milestone represents the point at which all code and/or services needed for commissioning (i.e. 1a requirements) are verified.

LDM-503-RSPb: Science Platform ready for Science Verification#

The Rubin Science Platform is ready to support Science Verification activities in the final phase of Rubin Observatory commissioning. In additional to the capabilities demonstrated in Milestone-2 (LDM-503-RSPa), the RSP will be scalable to the user load required to support Science Verification; provide access to a regularly updating stream of newly-acquired LSSTCam data; provide access to next-to-data computing and user batch computing to support intensive data analysis; provide access to a User Database Workspace; provide a means for integrating with a TBD focal-plane-scale visualization tool; provide capabilities in all three Aspects for following links between related datasets and exploiting provenance information; provide access to the EFD for recently acquired data, extending the capability demonstrated for LDM-503-EFDc; and provide substantial user-facing documentation, with links between the data and documentation.

LDM-503-17a: Test report: Final Pipelines Delivery#

This milestone describes the final Science Pipelines release from the DM Construction Project. The contents of the release are described in LDM-564 [O'Mullane et al., 2023]. The release will be accompanied by a characterization report.

DM-READY: DM is ready for ORR#

Warning

No description available

LDM-503-19: All 1a,1b DM requirements verified#

This milestone represents the point at which all code and/or services needed for operations (i.e. 1a and 1b requirements) are verified.

LDM-503-16: Ops rehearsal for data processing #3#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-17138]

  • Predecessors: LSST-1520, LDM-503-15, SITCOM-130

  • Successors: DM9, LDM-503-17, SITCOM-119

  • Due: 2025-06-17

  • Completion pending [DM-17139]

This milestone builds upon LDM-503-15 to demonstrate the production of prompt products over several nights.

LDM-503-17: Final operations rehearsal#

  • WBS: 02C.00

  • Level: 2

  • Test specification: Undefined [DM-17140]

  • Predecessors: DM2, DM11, DM9, LDM-503-16, RSP-1200, COMC-0620-9000

  • Successors: LSST-1580

  • Due: 2025-06-17

  • Completion pending [DM-17141]

This milestone builds upon LDM-503-16 to demonstrate successful operation of all aspects of the DM subsystem under simulated operational conditions.

02C.03: Alert Production#

_images/graph_02C.03.png

Fig. 3 Relationships between milestones in WBS 02C.03 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DM-AP-1: Basic single frame measurement pipeline.#

Warning

No description available

DM-AP-2: Alard & Lupton-style image differencing.#

Warning

No description available

DM-AP-3: Point source & dipole measurement on difference images.#

Warning

No description available

DM-AP-4: DIASource association#

  • WBS: 02C.03

  • Level: 3

  • Successors: LDM-503-03

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DM-AP-5: DIAObject generation#

  • WBS: 02C.03

  • Level: 3

  • Successors: LDM-503-03

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DM-AP-10: Advanced single frame measurement pipeline for Alert Production.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: F20S, DM-AP-1

  • Successors: LSST-1580

  • Due: 2023-06-01

  • Completed: 2023-05-31

The Single Frame Processing system is now substantially complete and provides all functionality described in LDM-151 [Swinbank et al., 2020] §3.1. It supports all expected interfaces, data-flows, and products, but the details of the algorithms are expected to undergo continued refinement.

DM-AP-11: Difference imaging includes noise decorrelation and correction for differential chromatic refraction.#

Template generation, including code required to build templates in both Data Release and Alert Production science payloads, can now product templates which are corrected for the effects of Differential Chromatic Refraction (DCR), as described in LDM-151 [Swinbank et al., 2020] §6.18. Further, noise in image differences can be “decorrelated”, as described in LDM-151 [Swinbank et al., 2020] §6.19.1. Both of these represents initial capabilities, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-AP-13: Trailed source measurement on difference images.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: DM-AP-3

  • Successors: LDM-503-13a

  • Due: 2023-06-01

  • Completed: 2021-10-29

The Alert Generation science payload now provides the capability to fit a trailed point source model to detections made on difference images, as described in LDM-151 [Swinbank et al., 2020] §3.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DLP-276: Dipole measurement suite#

  • WBS: 02C.03.04

  • Level: 3

  • Predecessors: DLP-275

  • Due: 2023-06-01

  • Completed: 2016-07-12

Warning

No description available

DLP-275: Deliver document on effects of DCR#

  • WBS: 02C.03.04

  • Level: 3

  • Successors: DLP-276

  • Due: 2023-06-01

  • Completed: 2016-04-08

Warning

No description available

DLP-642: Camera Team using DM code#

  • WBS: 02C.03.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2014-12-10

Warning

No description available

DLP-639: Re-design camera geometry classes#

  • WBS: 02C.03.01

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-13

Warning

No description available

DLP-644: Generalized Single Frame Astrometry Pipeline#

  • WBS: 02C.03.08

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-02-24

Warning

No description available

DLP-638: Refactor AFW R&D Code#

  • WBS: 02C.03.05

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-13

Warning

No description available

DM-AP-7: Basic instrument signature removal (ISR) capability.#

Warning

No description available

DM-AP-8: Advanced ISR, including ghost and linear feature masking, correction for the Brighter-Fatter effect and compensation for pixel response non-uniformity.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: F18F

  • Successors: LDM-503-11b

  • Due: 2023-06-30

  • Completed: 2019-01-17

Warning

No description available

DM-AP-12: Difference imaging is now agnostic to the PSF of the template image.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: DM-AP-11

  • Successors: LDM-503-15a

  • Due: 2023-07-03

  • Completed: 2023-05-31

The Alert Generation science payload is now agnostic to the PSF of the template, as described in LDM-151 [Swinbank et al., 2020] §3.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-AP-9: JOINTCAL1: Jointcal at a functional level#

Warning

No description available

DM-AP-6: Alert format defined & queue system available.#

Warning

No description available

DM-AP-18: Alert packet generation#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: S20F

  • Successors: LDM-503-05, DM-AP-6

  • Due: 2023-09-05

  • Completed: 2020-05-29

The AP Pipeline is now capable of generating per-DIASource alert packets, which comply with the current version of the LSST alert schema. This milestone only covers packet generation; success involves only those serialized alerts being e.g. written to disk, and does not imply that an alert transport system is available.

DM-AP-15: Alert distribution system fully integrated.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: LDM-503-12a

  • Successors: DM16, DM11, DM-AP-16

  • Due: 2025-02-12

  • Completion pending

The Alert Distribution system (LDM-151 [Swinbank et al., 2020] §3.3) is now available for deployment at the LSST Data Facility. The system is substantially complete, and supports all expected interfaces, data-flows, and products, but the details of the algorithms may undergo continued refinement.

DM-AP-17: Solar System Processing pipeline (SSP) available.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: LDM-503-12a, SITCOM-122

  • Successors: DM-DRP-36

  • Due: 2025-02-12

  • Completion pending

All of the algorithmic components which comprise the Solar System Processing (SSP) pipeline, as described in LDM-151 [Swinbank et al., 2020] §§6.25–6.28, are available for integration into science payloads. These components must provide realistic interfaces, data-flows and products, but it is expected that the details of the algorithms will undergo continued refinement.

DM-AP-14: Alert filtering system available.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: LDM-503-17a, DM-AP-6

  • Successors: LSST-1580

  • Due: 2025-06-10

  • Completion pending

The Alert Filtering Service (LDM-151 [Swinbank et al., 2020] §3.3.4; LSE-163 [Jurić et al., 2023] §3.5.2) can now be deployed as part of a prototype Alert Distribution system (LDM-151 [Swinbank et al., 2020] §3.3). This service is substantially complete, and supports all expected interfaces, data-flows, and products, but may not yet provide the complete set of filtering capabilities and/or may not perform at the level required to distribute the full LSST alert stream.

DM-AP-16: Full integration of the Alert Production system within the operational environment.#

  • WBS: 02C.03

  • Level: 3

  • Predecessors: LDM-503-17a, DM-AP-15, COMC-0240

  • Successors: LSST-1580

  • Due: 2025-06-10

  • Completion pending

The Alert Production system (comprising Single Frame Processing, LDM-151 [Swinbank et al., 2020] §3.1; Alert Generation, LDM-151 [Swinbank et al., 2020] §3.3; and Precovery and Forced Photometry, LDM-151 [Swinbank et al., 2020] §3.4) is now available for deployment at the LSST Data Facility. The system is substantially complete, and supports all expected interfaces, data-flows and products, but the details of the algorithms may undergo continued refinement.

02C.04: Data Release Production#

_images/graph_02C.04.png

Fig. 4 Relationships between milestones in WBS 02C.04 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DM-DRP-1: HSC merger complete: all functionality deployed for the most recent HSC data release processing is now available within the LSST stack.#

Warning

No description available

DM-DRP-2: Basic visualization and quality assessment tools operational on HSC-scale data volumes.#

Warning

No description available

DM-DRP-4: Calibration product generation in support of basic ISR.#

Warning

No description available

DM-DRP-6: Camera package supporting the Auxiliary Telescope.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-08

  • Due: 2023-06-01

  • Completed: 2018-01-31

Warning

No description available

DM-DRP-7: Coordinate transformation tool provided for use with the Collimated Beam Projector.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-09a, T&SC-2100-2252

  • Due: 2023-06-01

  • Completed: 2018-07-05

Warning

No description available

DM-DRP-11: Pipelines code provides supports for database ingestion of results.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F

  • Successors: F20F, DM-DAX-5

  • Due: 2023-06-01

  • Completed: 2019-12-31

The Data Release Production science payload now generates results which are compliant with the LSST Science Data Model, and which are suitable for database ingest.

DM-DRP-14: Insertion of simulated sources into the data stream to check pipeline performance.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-11b

  • Due: 2023-06-01

  • Completed: 2017-12-01

Warning

No description available

DM-DRP-16: Global photometric fitting (e.g. Burke et al. Forward Global Calibration Method).#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-09a

  • Due: 2023-06-01

  • Completed: 2018-05-31

Warning

No description available

DM-DRP-18: Initial multi-band deblending algorithm available.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-11b

  • Due: 2023-06-01

  • Completed: 2017-12-01

Warning

No description available

DM-DRP-20: Refined set of LSST calibration products.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DRP-4

  • Successors: LSST-1580, SITCOM-120

  • Due: 2023-06-01

  • Completed: 2021-07-29

The Calibration Products Production science payload is now capable of generating all calibration products which are required to enter commissioning.

DM-DRP-29: Moving point source model fitting now available.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F18F

  • Successors: LSST-1580

  • Due: 2023-06-01

  • Completed: 2022-12-31

The Data Release Production science payload now includes the capability of fitting moving a moving point source model that includes parallax and proper motion as free parameters to all objects, as described in LDM-151 [Swinbank et al., 2020] §6.7.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-32: Object classification system available.#

  • WBS: 02C.04

  • Level: 3

  • Successors: LDM-503-09a

  • Due: 2023-06-01

  • Completed: 2018-10-31

Warning

No description available

DM-DRP-35: Simultaneous measurement across a suite of coadds representing different bandpasses, epocs, and flavors.#

  • WBS: 02C.04

  • Level: 3

  • Successors: DM-DRP-31

  • Due: 2023-06-01

  • Completed: 2021-10-29

The Data Release Production science payload now includes the capability to perform consistent measurement of all Objects in a patch across multiple coadds, as described in LDM-151 [Swinbank et al., 2020] §5.3.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DLP-486: Prototype merge & deduplication in tract & patch overlaps#

  • WBS: 02C.04.05

  • Level: 3

  • Successors: DLP-566

  • Due: 2023-06-01

  • Completed: 2016-04-11

Warning

No description available

DLP-103: Prototype multi-coadd peak association available#

  • WBS: 02C.04.05

  • Level: 3

  • Successors: DLP-566

  • Due: 2023-06-01

  • Completed: 2016-04-11

Warning

No description available

DLP-101: Prototype coadd detection available#

  • WBS: 02C.04.05

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-04-11

Warning

No description available

DLP-100: Consistent multi-band deblending available#

  • WBS: 02C.04.05

  • Level: 3

  • Successors: DLP-566

  • Due: 2023-06-01

  • Completed: 2016-04-11

Warning

No description available

DLP-736: Performance and capability enhancements in support of catalog use cases#

  • WBS: 02C.04.01

  • Level: 3

  • Predecessors: DMTC-0000-3000, S17F

  • Successors: F17S

  • Due: 2023-06-01

  • Completed: 2016-12-19

Warning

No description available

DLP-588: AFW R&D code refactored#

  • WBS: 02C.04.01

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-288: HSC merger complete#

  • WBS: 02C.04.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-12-07

Warning

No description available

DLP-73: Footprints redesigned and refactored#

  • WBS: 02C.04.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2017-05-09

Warning

No description available

DLP-694: Prototype Brighter-Fatter correction available#

  • WBS: 02C.04.02

  • Level: 3

  • Successors: DMTC-0000-0020

  • Due: 2023-06-01

  • Completed: 2016-05-31

Warning

No description available

DLP-724: Forced per-epoch photometry#

  • WBS: 02C.04.06

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2014-09-12

Warning

No description available

DLP-716: Fundamental flux measurements#

  • WBS: 02C.04.06

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2014-09-12

Warning

No description available

DLP-715: Fundamental centroid and shape measurements#

  • WBS: 02C.04.06

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2015-12-04

Warning

No description available

DLP-700: Galaxy shear fitting parameters available#

  • WBS: 02C.04.06

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-05-31

Warning

No description available

DLP-697: Single epoch CModel minimization#

  • WBS: 02C.04.06

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-696: Single epoch Sersic model fitting#

  • WBS: 02C.04.06

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-503: Prototype direct coaddition#

  • WBS: 02C.04.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-31

Warning

No description available

DLP-502: Prototype image artifact identification available#

  • WBS: 02C.04.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-31

Warning

No description available

DLP-152: Initial PSF-matched image coaddition available#

  • WBS: 02C.04.04

  • Level: 3

  • Predecessors: S17F

  • Due: 2023-06-01

  • Completed: 2016-12-19

Warning

No description available

DM-DRP-3: PSF-homogenized coadd construction.#

Warning

No description available

DM-DRP-5: Camera package supporting the LSST Camera.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F

  • Successors: LDM-503-09a

  • Due: 2023-06-29

  • Completed: 2018-06-29

Warning

No description available

DM-DRP-38: Camera package supporting the Commissioning Camera.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F

  • Successors: LDM-503-09a

  • Due: 2023-06-29

  • Completed: 2018-06-29

Warning

No description available

DM-DRP-17: Simultaneous photometric and astrometric fitting to multiple exposures.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F, DM-AP-9

  • Successors: LDM-503-09a

  • Due: 2023-07-20

  • Completed: 2018-11-29

Warning

No description available

DM-DRP-12: Background estimation over the full visit.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DAX-8

  • Successors: LSST-1580

  • Due: 2023-09-05

  • Completed: 2020-06-30

Warning

No description available

DM-DRP-13: PSF estimation over the full visit.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DAX-8

  • Successors: LSST-1580, DM-DRP-15

  • Due: 2023-09-05

  • Completed: 2023-02-28

The Data Release Production science payload now includes code for estimating the PSF over the full focal plane, as described in LDM-151 [Swinbank et al., 2020] §6.11.3. This does not require that the PSF model be “physically motivated” (ie, decomposed into atmospheric and optical components). Further, this represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-15: All varieties of coadd required for object detection and characterization are now produced during normal pipeline operation (although not necessarily at the ultimately required level of fidelity).#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DRP-13

  • Successors: LDM-503-11b

  • Due: 2023-09-05

  • Completed: 2018-11-30

Warning

No description available

DM-DRP-21: Integrated image characterization pipeline for Data Release Production.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-AP-9, DM-DAX-8

  • Successors: LSST-1580

  • Due: 2023-09-05

  • Completed: 2021-10-29

The Data Release Production science payload now includes an integrated image characterization pipeline, as described in LDM-151 [Swinbank et al., 2020] §5.1, including iterative refinement of the image characterization. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-22: Template generation integrated with Data Release Production pipelines.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F20S

  • Successors: LDM-503-13a

  • Due: 2023-09-05

  • Completed: 2021-03-31

The Data Release Production science payload is now capable of generating templates for use in image differencing, as described in LDM-151 [Swinbank et al., 2020] §§5.2.3 & 5.2.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-28: Difference images are now a first-class data product during data release processing.#

The Data Release Production science payload now includes the capability to generate difference images, and to perform source detection and characterization on those images, as described in LDM-151 [Swinbank et al., 2020] §5.2.5. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-30: Forced photometry is now performed on individual processed visit images during data releases.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F20F

  • Successors: LDM-503-13a

  • Due: 2023-09-05

  • Completed: 2021-02-01

The Data Release Production science payload now includes the capability of performing “forced” photometry at the positions of known Objects on Processed Visit Images, as described in LDM-151 [Swinbank et al., 2020] §5.5.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-34: Selection maps are generated during data releases.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F20F

  • Successors: LDM-503-13a

  • Due: 2023-09-05

  • Completed: 2021-05-24

The Data Release Production science payload now includes the capability of generating multi-scale maps that describe LSST’s depth and efficiency at detecting different classes of object as part of a data release, as described in LDM-151 [Swinbank et al., 2020] §5.6.3. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-8: Calibration product generation for the Auxiliary Telescope.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F, COMC-060200-10000

  • Successors: LSST-1580

  • Due: 2023-10-13

  • Completed: 2021-07-31

The Calibration Products Production science payload is now capable of generating those products which are required to calibrate the Auxiliary Telescope. This represent an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithms being used is expected.

DM-DRP-9: Data reduction pipeline for the Auxiliary Telescope.#

Warning

No description available

DM-DRP-23: Atmospheric characterization based on data from the Auxiliary Telescope now available.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F21S, DM-DRP-9, COMC-060200-10000

  • Successors: LSST-1580

  • Due: 2023-10-13

  • Completed: 2021-08-31

The Calibration Products Production science payload is now capable of producing an atmospheric transmission model based on observations from the Auxiliary Telescope, as described in LDM-151 [Swinbank et al., 2020] §4.6.1. This milestone does not presume that the Auxiliary Telescope itself is available for testing: it is satisfied by the availability of a pipeline that can process appropriate simulated or precursor data. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-37: Artifact rejection and background matching during coadd construction.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: F18F, DM-DRP-3

  • Successors: LSST-1580

  • Due: 2023-12-01

  • Completion pending

The Data Release Production science payload now includes code for generating final estimates of the sky background and artifact masks, as described in LDM-151 [Swinbank et al., 2020] §5.2.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-27: Object generation: association and assembly of (DIA, coadd, etc) sources to form objects.#

The Data Release Production science payload now includes the capability to build Objects by cross-matching the Source and DIASource tables, as described in LDM-151 [Swinbank et al., 2020] §5.3.2. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-19: QA metrics are generated during pipeline execution.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: S18F

  • Successors: LDM-503-11b

  • Due: 2023-12-01

  • Completed: 2018-12-13

Warning

No description available

DM-DRP-26: Overlap resolution at tract & patch boundaries.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-DRP-27

  • Successors: LDM-503-13a

  • Due: 2023-12-05

  • Completed: 2021-02-01

The Data Release Production science payload is now capable of finalizing the definition of Objects which fall within the overlap regions at the edges of tracts and patches, as described in LDM-151 [Swinbank et al., 2020] §5.4. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-24: Physically motivated PSF model, including separate characterization of contributions from the atmosphere and the telescope system.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: LSST-1520

  • Successors: LSST-1580

  • Due: 2025-02-12

  • Completion pending

Per LDM-151 [Swinbank et al., 2020] §6.11.3, the PSF model used by the Data Release Production science payload is now decomposed into separate contributions from the telescope optics and the atmosphere.

DM-DRP-31: A photometric redshift is now provided for each object.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: LSST-1520, DM-DRP-35

  • Successors: DM-DRP-36

  • Due: 2025-02-12

  • Completion pending

The Data Release Production science payload now includes the capability of calculating photometric redshifts as part of a data release, as (briefly) described in LDM-151 [Swinbank et al., 2020] §5.6.5. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

DM-DRP-36: Solar System Processing integrated to data release processing.#

  • WBS: 02C.04

  • Level: 3

  • Predecessors: DM-AP-17, DM-DRP-31

  • Successors: LSST-1580

  • Due: 2025-02-12

  • Completion pending

The Data Release Production science payload now generates SSObjects, by delegating to the Solar System Processing (SSP) pipeline being delivered by the Alert Production team, as described in LDM-151 [Swinbank et al., 2020] §5.6.1. This represents an initial capability, providing realistic interfaces, data-flows and products; ongoing refinement of the algorithm being used is expected.

02C.05: Science User Interface and Tools#

_images/graph_02C.05.png

Fig. 5 Relationships between milestones in WBS 02C.05 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DM-SUIT-1: Search and display WISE sources (objects) in PDAC#

  • WBS: 02C.05

  • Level: 3

  • Predecessors: DM-DAX-1

  • Successors: LDM-503-01

  • Due: 2023-06-01

  • Completed: 2017-11-30

Warning

No description available

DM-SUIT-2: Search WISE coaded data single exposure images in PDAC (the images are from IRSA at IPAC, not NCSA)#

  • WBS: 02C.05

  • Level: 3

  • Predecessors: DM-DAX-2

  • Successors: LDM-503-01

  • Due: 2023-06-01

  • Completed: 2017-11-30

Warning

No description available

DM-SUIT-3: Time series analysis tool for WISE data#

  • WBS: 02C.05

  • Level: 3

  • Successors: LDM-503-01

  • Due: 2023-06-01

  • Completed: 2017-11-30

Warning

No description available

DM-SUIT-4: Multiple data traces in chart space#

  • WBS: 02C.05

  • Level: 3

  • Successors: LDM-503-01

  • Due: 2023-06-01

  • Completed: 2017-11-30

Warning

No description available

DM-SUIT-5: Search and display processed HSC data#

Portal provides access to a collection of public-release HSC data processed by LSST DM, including calibrated single-epoch images, coadds, and catalogs (including at least an Object-like catalog). Access is obtained via IVOA interfaces including TAP for catalogs and ObsTAP for image metadata. (NB: Still achievable post-DM-10).

DM-SUIT-8: SUIT portal integrated with workspace#

  • WBS: 02C.05

  • Level: 3

  • Successors: DM-SUIT-10

  • Due: 2023-06-01

  • Completed: 2020-05-29

Portal is integrated with LSST-provided WebDAV or VOSpace service exposing the User File Workspace. Portal provides functions for saving image and catalog query results to the logged-in user’s workspace, with the user’s identity correctly applied. Portal provides functions for accessing data in the user’s workspace and displaying it using the Portal’s image and table visualization tools, as well as using data from the workspace in other Portal contexts where a file can be uploaded, e.g., to perform a multi-object query. (NB: Still achievable post-DM-10, though only with WebDAV.).

DLP-578: Propose a workspace structure for discussion.#

  • WBS: 02C.05.05

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-256: First draft definition of workspace#

  • WBS: 02C.05.05

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-11-17

Warning

No description available

DLP-518: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.05

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-03-20

Warning

No description available

DLP-872: Convert major portion of GWT in Firefly to pure JavaScript#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-04-08

Warning

No description available

DLP-871: Firefly server side code refactor#

  • WBS: 02C.05.01

  • Level: 3

  • Predecessors: DMTC-0000-0030

  • Successors: S17S

  • Due: 2023-06-01

  • Completed: 2017-07-31

Warning

No description available

DLP-574: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.01

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-564: Able to use some data-access APIs developed by database team#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-11-17

Warning

No description available

DLP-521: Define the data access APIs with database team#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-03-11

Warning

No description available

DLP-520: Access Qserv for catalog search in SUI prototype#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2014-12-02

Warning

No description available

DLP-519: Research the development framework for SUI#

  • WBS: 02C.05.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2014-10-14

Warning

No description available

DLP-207: Extra LSST query capabilities in web UI#

  • WBS: 02C.05.01

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-08-01

Warning

No description available

DLP-205: LSST web UI search external catalog#

  • WBS: 02C.05.01

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-08-01

Warning

No description available

DLP-575: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.02

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-222: Provide time-domain data view#

  • WBS: 02C.05.02

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-08-01

Warning

No description available

DLP-218: Provide API for image display in Firefly#

  • WBS: 02C.05.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-11-17

Warning

No description available

DLP-216: Provide basic APIs to use Firefly visualization components#

  • WBS: 02C.05.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-11-02

Warning

No description available

DLP-576: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.03

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-577: Evaluate best-of-breed tools. Technology exploration.#

  • WBS: 02C.05.04

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DM-SUIT-7: Mapping between SUIT systems & NCSA auth system#

  • WBS: 02C.05

  • Level: 3

  • Predecessors: S19F

  • Successors: DM-SUIT-10

  • Due: 2023-09-01

  • Completed: 2019-05-31

Portal is integrated with NCSA-provided federated-identity authentication and authorization system: the Portal recognizes that a user is logged in and displays that state, and passes the user’s identity along to LSST data services that it accesses. (NB: Still achievable post-DM-10) Claimed in May, based on installation on lsst-lsp-int.

DM-SUIT-10: SUIT deployment procedure#

Portal software is adapted for deployment in Kubernetes. The set of container images required for deployment is documented in an LTD-style manner. Configuration files for Kubernetes are provided in Github and documented to a level that permits deployments to be performed by non-IPAC staff. (NB: Still achievable post-DM-10).

02C.06: Science Data Archive and Application Services#

_images/graph_02C.06.png

Fig. 6 Relationships between milestones in WBS 02C.06 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DM-DAX-1: WISE data ingest to PDAC#

Warning

No description available

DM-DAX-2: Query service supporting IVOA TAP protocol, w/ support for asynchronous queries#

An IVOA TAP service supporting both synchronous and asynchronous query models, integrated with the LSP authentication architecture, and able to access catalog data hosted in a downstream in Qserv instance.

DM-DAX-5: Database ingest in support of HSC reprocessing (ie, large catalog ingest)#

Existing Qserv ingest tooling is highly manual and poorly parallelized. Develop new tooling to address these issues and support ingest campaigns at the scale needed for DRP.

DM-DAX-6: Prototype level 1 database#

Warning

No description available

DM-DAX-11: Transformed EFD schema#

  • WBS: 02C.06

  • Level: 3

  • Successors: LSST-1580

  • Due: 2023-06-01

  • Completed: 2018-05-31

Warning

No description available

DM-DAX-10: Per-object gather support in Butler#

  • WBS: 02C.06

  • Level: 3

  • Predecessors: F18F

  • Successors: DM-DAX-5

  • Due: 2023-06-01

  • Completed: 2019-11-29

Demonstrate Butler configuration able to gather all data pertaining to a particular object, such as retrieving cutouts from all warps at a given box on a skymap.

DLP-836: New logging for DM#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2015-11-17

Warning

No description available

DLP-25: Prototype Web Service for SUI#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-09-01

Warning

No description available

DLP-837: AP-ready Butler Framework#

  • WBS: 02C.06.02

  • Level: 3

  • Successors: F20F

  • Due: 2023-06-01

  • Completed: 2019-08-30

Warning

No description available

DLP-835: New xrootd client#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: DLP-834

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-834: Automated Testbed for Qserv#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: LSST-1440

  • Successors: DLP-835

  • Due: 2023-06-01

  • Completed: 2014-08-01

Warning

No description available

DLP-818: CSS#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-11-17

Warning

No description available

DLP-817: Multi-node Multi-query Integration Testing Harness#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-11-17

Warning

No description available

DLP-805: Distributed Database and Table Management#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-10-29

Warning

No description available

DLP-801: Qserv uses C++-based Geometry Code#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-11-17

Warning

No description available

DLP-649: Shared Scans#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2017-05-26

Warning

No description available

DLP-635: Qserv Query Cancellation#

  • WBS: 02C.06.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-01-12

Warning

No description available

DLP-473: Async Queries in Qserv#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: S17F, F17F

  • Successors: DLP-725

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DLP-472: Qserv Data Distribution#

  • WBS: 02C.06.02

  • Level: 3

  • Predecessors: DMTC-0000-3000, S17F

  • Due: 2023-06-01

  • Completed: 2018-06-29

Warning

No description available

DLP-802: Alert Production Database (APDB) design#

  • WBS: 02C.06.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2021-05-31

Warning

No description available

DLP-800: Data Ingest into MySQL#

  • WBS: 02C.06.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-11-17

Warning

No description available

DLP-725: Working Prototype of AP/L1 Database#

  • WBS: 02C.06.01

  • Level: 3

  • Predecessors: F17F, DLP-473

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DM-DAX-7: Generation 3 Butler, running on a POSIX file-system provided by NCSA#

  • WBS: 02C.06

  • Level: 3

  • Predecessors: DM-DAX-12

  • Successors: DM-DAX-13

  • Due: 2023-09-05

  • Completed: 2020-06-30

Given a stood-up data backbone instance, demonstrate instantiation/configuration of a Butler Gen3 instance capable of retrieving images from that backbone instance. (As currently conceived, the data backbone is either a GPFS filesystem or object store, in combination with metadata that is a superset of that maintained in a Butler Gen3 registry. Depending on particular technologies and implementation strategies chosen for the data backbone, this may be either a “simple matter of configuration” of existing Butler connectors, or may require some additional custom connectors to be written.).

DM-DAX-8: Scatter-gather support in PipelineTask#

Demonstrate a functional PipelineTask with scatter/gather functionality, such as Jointcal, SkyCorrect, or FGCM.

DM-DAX-12: Generation 3 middleware at feature parity with Generation 2#

Generation 3 middleware, consisting of Butler updates and the PipelineTask framework, are now able to support existing science payloads with no loss of functionality over previous versions. That is, Gen 3 may not be feature complete, but Gen 2 can be retired with no functionality regressions across the subsystem.

DM-DAX-13: Generation 2 middleware retired#

The Generation 2 middleware is deprecated, and may be removed from the codebase. No current science payload and no current execution service makes use of it.

DM-DAX-14: Provenance system review completed#

  • WBS: 02C.06

  • Level: 3

  • Predecessors: F20F

  • Successors: DM-DAX-9

  • Due: 2023-09-05

  • Completed: 2022-01-31

Warning

No description available

DM-DAX-15: DAX Construction Complete#

  • WBS: 02C.06

  • Level: 3

  • Successors: LSST-1580

  • Due: 2023-12-15

  • Completion pending

Warning

No description available

DM-DAX-9: Butler Provenance System#

  • WBS: 02C.06

  • Level: 3

  • Predecessors: LSST-1520, DM-DAX-12, DM-DAX-14, COMC-060200-20930

  • Successors: LDM-503-17a

  • Due: 2025-02-12

  • Completion pending

Warning

No description available

02C.07: LSST Data Facility#

_images/graph_02C.07.png

Fig. 7 Relationships between milestones in WBS 02C.07 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DMTC-4500-0010: Start NCSA S18#

  • WBS: 02C.07

  • Level: 3

  • Due: 2017-12-01

  • Completed: 2017-12-01

Time-based milestone marking the start of NCSA’s Spring 2018 development cycle.

A2330: Ship AA System to Chile#

  • WBS: 02C.07.09

  • Level: 4

  • Due: 2019-06-28

  • Completed: 2019-06-28

Date for Authentication and Authorization hardware to ship from NCSA to Chile.

A4670: June 2019 Travel to Chile - Travel Start#

  • WBS: 02C.07.09

  • Level: 4

  • Predecessors: A4369.9

  • Due: 2019-08-17

  • Completed: 2019-08-17

Start of travel to Chile to install hardware and systems at the Base Data Center.

A1300: Start Milestone: October 1st, 2019#

  • WBS: 02C.07.05

  • Level: 4

  • Due: 2019-10-01

  • Completed: 2019-10-01

Warning

No description available

A6150: Start Milestone: January 1st, 2020#

  • WBS: 02C.07

  • Level: 3

  • Due: 2020-01-03

  • Completed: 2020-01-03

Warning

No description available

DM-NCSA-27: Deliver header service code#

Warning

No description available

DM-NCSA-28: IIP control code#

Warning

No description available

DM-NCSA-1: Provide regular reprocessing service for HSC data#

Warning

No description available

DM-NCSA-2: Provide access to results of regular reprocessing (NB the form this takes depends upon available DAX functionality)#

Warning

No description available

DM-NCSA-3: Provide database for metadata, provenance, location and demonstrate ingest at small scale#

Warning

No description available

DM-NCSA-29: Deliver IIP image client code#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: DM-NCSA-28

  • Successors: LSST-1200

  • Due: 2023-06-01

  • Completed: 2018-03-31

Warning

No description available

DM-NCSA-4: Minimal support for the small operational schema including file metadata and provenance for every file, and record of in#

Warning

No description available

DM-NCSA-5: Level 1 archiving system able to acquire pixel data from the Aux Tel DAQ, header metadata via OCS, assemble FITS image,#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-04b

  • Due: 2023-06-01

  • Completed: 2018-05-31

Warning

No description available

DM-NCSA-6: Ability to transfer files originating from Tucson to NCSA and ingest files at NCSA, including metadata and provenance#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-04b

  • Due: 2023-06-01

  • Completed: 2018-10-31

Warning

No description available

DM-NCSA-7: Capability to paint displays for Tucson and NCSA#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-04b

  • Due: 2023-06-01

  • Completed: 2018-06-29

Warning

No description available

DM-NCSA-9: Test instance of alert distribution hosting service and L1 database in Development & Integration Enclave#

Hosting an instance of the alert distribution service delivered from University of Washington on our test containerized application management service platform, running feeds from the pipeline to the distribution services. Note: This was informally tied to LDM-503-05, which was de-scoped to a small demonstration of running an instance of Kafka at NCSA, so no L1 database was needed. We believe this was completed in August 2018.

DM-NCSA-10: Sustained archiving service that is OCS commandable#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-06

  • Due: 2023-06-01

  • Completed: 2019-05-31

The capability to use the existing OCS/CCS to command data acquisition and hand-off to a test Observatory Operations Data Service (OODS).

DM-NCSA-11: Verified acquisition of raw and crosstalk-corrected exposures at raft scale, incl. correct metadata#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: F18F, T&SC-5900-0400

  • Successors: DLP-552, LDM-503-06

  • Due: 2023-06-01

  • Completed: 2020-06-30

Exercise the handoff between the L1 archiving and data backbone endpoint, including a monitoring service that tracks files that have been transferred and received.

DM-NCSA-13: Header Writing Service for Spectrograph use case#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-08

  • Due: 2023-06-01

  • Completed: 2019-05-31

The Header builder integrated into OCS and demonstrated. Header builder has identified sources of information for all modes of observing with the Spectrograph.

DM-NCSA-14: Data Backbone endpoints in Chile for ingestion and access, distribution over WAN, ingest at NCSA into custodial file sto#

  • WBS: 02C.07

  • Level: 3

  • Successors: LDM-503-08

  • Due: 2023-06-01

  • Completed: 2021-11-17

Provide a data Backbone endpoint in Chile, allowing distribution over a wide-area network and ingestion at NCSA.

DM-NCSA-16: Perform ISR processing on ComCam-scale data.#

Process ComCam-scale data through the ISR pipeline to generate QC parameters used by the observatory.

DM-NCSA-20: ComCam Archiving Service#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: T&SC-5900-0400

  • Successors: LDM-503-11a

  • Due: 2023-06-01

  • Completed: 2020-06-30

Implement a Service that acquires raw data from a camera at raft scale and sends to NCSA for ingestion into the Data Backbone. This is a preliminary milestone to demonstrate initial capabilities.

DM-NCSA-21: L1 Offline Processing Service, Raft Scale, ComCam#

  • WBS: 02C.07

  • Level: 3

  • Successors: LSST-1580

  • Due: 2023-06-01

  • Completed: 2023-05-31

Demonstrated running of processing campaigns in offline mode needed for the ComCam phase of commissioning to debug the instrument. Likely includes calibration, ISR, and other pipelines.

DLP-864: Coordination with NCSA/CC-IN2P3 JCC#

  • WBS: 02C.07.00

  • Level: 3

  • Predecessors: DLP-469

  • Successors: S18F

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DLP-469: Coordination with NCSA/CC-IN2P3 JCC#

  • WBS: 02C.07.00

  • Level: 3

  • Successors: DLP-864

  • Due: 2023-06-01

  • Completed: 2016-06-07

Warning

No description available

DLP-356: Set up Joint Coordination Council (JCC) for coordination with satellite processing center#

  • WBS: 02C.07.00

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-26

Warning

No description available

DLP-355: Establish MOA with CC-IN2P3 satellite processing center#

  • WBS: 02C.07.00

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-31

Warning

No description available

DLP-358: Refinement of facility integration plan and traceability of science and operational requirements#

  • WBS: 02C.07.01

  • Level: 3

  • Predecessors: DLP-357

  • Due: 2023-06-01

  • Completed: 2016-04-08

Warning

No description available

DLP-357: Initial refinement of facility integration plan#

  • WBS: 02C.07.01

  • Level: 3

  • Successors: DLP-358

  • Due: 2023-06-01

  • Completed: 2016-06-15

Warning

No description available

DLP-847: Meet camera DAQ/OCS milestones for FY16#

  • WBS: 02C.07.01

  • Level: 3

  • Predecessors: DMTC-0000-0030

  • Successors: F17F

  • Due: 2023-06-01

  • Completed: 2017-09-01

Warning

No description available

DLP-368: Initial AP DMCS#

  • WBS: 02C.07.01

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DLP-366: Mid-scale processing of eligible camera data#

  • WBS: 02C.07.01

  • Level: 3

  • Predecessors: DMTC-0000-0030

  • Successors: F17F

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DLP-364: Build prototype production sequence for the DMCS#

  • WBS: 02C.07.01

  • Level: 3

  • Successors: DLP-566

  • Due: 2023-06-01

  • Completed: 2016-06-07

Warning

No description available

DLP-361: Build prototype AP DMCS, including events#

  • WBS: 02C.07.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-26

Warning

No description available

DLP-360: Prepare for OCS Middleware integration with AP DMCS#

  • WBS: 02C.07.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2014-11-21

Warning

No description available

DLP-359: Improve fault tolerance in AP simulator#

  • WBS: 02C.07.01

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2014-11-19

Warning

No description available

DLP-389: Implement advanced logging functionality, including event-based logging#

  • WBS: 02C.07.01

  • Level: 3

  • Successors: DLP-566

  • Due: 2023-06-01

  • Completed: 2015-09-29

Warning

No description available

DLP-850: Prototype authentication/authorization system complete#

  • WBS: 02C.07.02

  • Level: 3

  • Successors: S17S

  • Due: 2023-06-01

  • Completed: 2016-06-07

Warning

No description available

DLP-405: Investigate CAS integration with Internet2 AAA tools#

  • WBS: 02C.07.02

  • Level: 3

  • Successors: S17S

  • Due: 2023-06-01

  • Completed: 2016-07-15

Warning

No description available

DLP-404: Identify candidate security packages#

  • WBS: 02C.07.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-06-07

Warning

No description available

DLP-412: Monitor development cluster#

Warning

No description available

DLP-411: Monitor OpenStack#

  • WBS: 02C.07.02

  • Level: 3

  • Successors: DLP-412

  • Due: 2023-06-01

  • Completed: 2016-06-07

Warning

No description available

DLP-410: Implement Puppet for system configuration management#

  • WBS: 02C.07.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-31

Warning

No description available

DLP-415: Evaluate iRODS, Datacat, and similar file system management technologies#

  • WBS: 02C.07.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-31

Warning

No description available

DLP-400: Refactor basic event services#

  • WBS: 02C.07.02

  • Level: 3

  • Successors: DLP-556

  • Due: 2023-06-01

  • Completed: 2015-01-26

Warning

No description available

DLP-422: Evaluate OpenStack as a development environment#

  • WBS: 02C.07.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-01-29

Warning

No description available

DLP-424: Integration of DevOps to OpenStack#

  • WBS: 02C.07.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-07-15

Warning

No description available

DLP-423: DevOps assessment#

  • WBS: 02C.07.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-31

Warning

No description available

DLP-471: Refresh sizing model#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-06-07

Warning

No description available

DLP-431: Refinement of facility integration plan and traceability of science and operational requirements#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-07-15

Warning

No description available

DLP-430: Initial refinement of facility integration plan#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-08-31

Warning

No description available

DLP-459: Qserv capability to hold data reduced at NCSA#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-07-25

Warning

No description available

DLP-458: Ingest of bulk data from real cameras to ComCam scale#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-10-18

Warning

No description available

DLP-457: Support runs on real cameras data and ImSim data#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DLP-452: Support datasets on basic file system infrastructure with apropos policies#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-07-15

Warning

No description available

DLP-451: Deploy OpenStack capability#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-07-30

Warning

No description available

DLP-450: Support lsst-dev for developers#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-02-02

Warning

No description available

DLP-449: Upgrade storage server hardware#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-07-15

Warning

No description available

DLP-442: Procure infrastructure for Base Center to meet FY18 goals#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Due: 2023-06-01

  • Completed: 2018-04-30

Warning

No description available

DLP-441: Base Center requirements document#

  • WBS: 02C.07.04

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-05-29

Warning

No description available

DLP-433: Procure infrastructure for Archive Center to FY18 scale#

  • WBS: 02C.07.04

  • Level: 3

  • Predecessors: F17F

  • Due: 2023-06-01

  • Completed: 2018-04-30

Warning

No description available

A4369.9: Ship Hardware to BDC#

  • WBS: 02C.07.09

  • Level: 4

  • Successors: A4670

  • Due: 2023-06-01

  • Completed: 2019-07-17

Date for Base Data Center hardware to ship from NCSA to Chile.

A4640: Ship ComCam Hardware to Tucson#

  • WBS: 02C.07.09

  • Level: 4

  • Due: 2023-06-01

  • Completed: 2019-06-27

Date for ComCam support systems to ship from NCSA to Tucson.

A4680: June 2019 Travel to Chile - Travel End#

  • WBS: 02C.07.09

  • Level: 4

  • Successors: LDM-503-09

  • Due: 2023-06-01

  • Completed: 2019-08-24

End of travel to Chile to install hardware and systems at the Base Data Center.

A4750: Summit AA Systems Installed at the Summit#

  • WBS: 02C.07.09

  • Level: 4

  • Due: 2023-06-01

  • Completed: 2019-08-29

Authorization and Authentication systems installed at the Summit.

A4860: ComCam Systems in Tucson ready for NCSA#

  • WBS: 02C.07.09

  • Level: 4

  • Successors: LDM-503-09, A5860

  • Due: 2023-06-01

  • Completed: 2019-08-01

ComCam Test Stand ready for access by NCSA staff.

A5090: Expected Arrival of ComCam Systems to NCSA#

  • WBS: 02C.07.09

  • Level: 4

  • Predecessors: T&SC-6500-8100

  • Due: 2023-06-08

  • Completed: 2019-04-30

Arrival at NCSA of hardware to support ComCam testing and integration.

DM-NCSA-15: Batch Processing Service for offline spectrograph data processing#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: DM-DAX-12

  • Successors: LDM-503-08

  • Due: 2023-09-05

  • Completed: 2020-05-29

Demonstrate the ability to process offline Spectrograph data using a science pipeline in the Batch Processing Service. Output(s) will be available in the Data Backbone.

DM-NCSA-22: Information in consolidated database available to QA portal#

  • WBS: 02C.07

  • Level: 3

  • Predecessors: DM-DAX-12

  • Successors: LDM-503-11a

  • Due: 2023-09-05

  • Completed: 2020-08-31

Scientific and operational data products stored in the consolidated database during ComCam data processing are available for QA assessment of ComCam data through normal analysis access environments.

02C.08: International Communications and Base Site#

_images/graph_02C.08.png

Fig. 8 Relationships between milestones in WBS 02C.08 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DM-NET-5: Base - Archive Network Functional 100 Gbps#

  • WBS: 02C.08

  • Level: 2

  • Predecessors: DLP-464, COMC-0021

  • Successors: CAMM8330, CAMM8340, COMC-0620-2240

  • Due: 2023-06-01

  • Completed: 2021-02-01

All segments of the Base - Archive Network, including La - Serena - Santiago - Sao Paolo - Miami/Boca Raton - Chicago - Champaign are operationally available at 100 Gbps line rate (effective rate is typically lower, ~ 80%).

DLP-464: Install networking for normal Base Center operations for ComCam scale#

  • WBS: 02C.08.01

  • Level: 3

  • Predecessors: F18F

  • Successors: CAMM8340, DM-NET-5

  • Due: 2023-06-01

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for commissioning camera observing.

DM-NET-2: Mountain - Base Network Functional 2 x 100 Gbps#

  • WBS: 02C.08.01

  • Level: 1

  • Predecessors: DLP-559

  • Successors: LDM-503-08, COMC-0620-2050

  • Due: 2023-06-01

  • Completed: 2018-04-02

Warning

No description available

IT-725-M: Base Data Center Network Complete#

  • WBS: 02C.08.01

  • Level: 4

  • Successors: LDM-503-10

  • Due: 2023-06-01

  • Completed: 2021-08-20

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for ATS observing. (Level 3 equivalent of Level 2 milestone DM-NET-4).

IT-716-M: Base Center Beneficial Occupancy#

  • WBS: 02C.08.01

  • Level: 4

  • Predecessors: T&SC-0400-1600

  • Due: 2023-06-01

  • Completed: 2019-03-18

Warning

No description available

IT-718-M: Base Center Full Occupancy#

  • WBS: 02C.08.01

  • Level: 4

  • Due: 2023-06-01

  • Completed: 2019-06-30

Externally linked to TS milestone for full occupancy of Base Data Center, including all construction, utilities, access and dust control.

IT-716-M2: Furniture installed in Offices, purchased cabling received#

  • WBS: 02C.08.01

  • Level: 4

  • Due: 2023-06-01

  • Completed: 2019-10-31

Externally linked to TS milestone for installation of furniture of Base Office Building and Base Data Center. All cabling to connect offices to Base LAN has been received.

DLP-466: Install networking for ComCam-scale Chilean DAC serving on order 20 people#

  • WBS: 02C.08.02

  • Level: 3

  • Predecessors: F18F

  • Successors: LSST-1580, CAMM8330, CAMM8340

  • Due: 2023-06-01

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with ATS observing.

DM-NET-1: Rubin Network Re-engineering#

  • WBS: 02C.08.02

  • Level: 2

  • Predecessors: LSST-1440

  • Due: 2023-06-01

  • Completed: 2015-08-05

Warning

No description available

DM-NET-3: Initial Network Ready (Summit)#

  • WBS: 02C.08.03

  • Level: 2

  • Predecessors: T&SC-2600-1545

  • Successors: LDM-503-08

  • Due: 2023-06-01

  • Completed: 2018-03-05

Warning

No description available

DM-NET-4: Base LAN installed#

  • WBS: 02C.08.03

  • Level: 2

  • Successors: LDM-503-11

  • Due: 2023-06-01

  • Completed: 2020-01-31

Installed the Base LAN equipment in the Base Date Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for ATS observing. (Level 2 equivalent of Level 3 milestone IT-725-M).

DLP-42: La Serena to Santiago Path Design Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-04-30

Warning

No description available

DLP-44: La Serena to Santiago with test traffic AURA DWDM LSST Fiber#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8000-0826

  • Due: 2023-06-01

  • Completed: 2017-12-31

Warning

No description available

DMTC-8000-0806: 3T Fiber Installation Certification and Acceptance Test - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DLP-45

  • Due: 2023-06-01

  • Completed: 2017-03-15

Warning

No description available

DMTC-8000-0818: Acceptance Test with Equipment Installed - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2480

  • Due: 2023-06-01

  • Completed: 2017-12-01

Warning

No description available

DMTC-8000-0826: Intermediate traffic testing and report - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-44

  • Due: 2023-06-01

  • Completed: 2018-03-02

Warning

No description available

DMTC-8000-0836: La Serena to Santiago with live traffic and uptime over LSST Lambda Ready - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DLP-61

  • Due: 2023-06-01

  • Completed: 2019-07-23

Warning

No description available

DMTC-8000-0870: Report on functional fiber, including uptime, 1st year functional milestone - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2019-07-23

Contract Deliverable Report #8 Report on functional fiber, including uptime, 1st year functional milestone - Complete. Covers first year of operational service of La Serena - Santiago primary fiber, including availability and outage information.

DMTC-8000-0950: Technical Implementation Plan for 40Gbps diverse path - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2019-03-25

Contract Deliverable Report #10 Technical Implementation Plan for 40Gbps diverse path - Complete. Covers the fiber and equipment, ownership/service levels for La Serena - Santiago secondary path.

DMTC-7400-2420: REUNA La Serena - Santiago DWDM equipment - Complete#

Warning

No description available

DMTC-7400-2510: REUNA La Serena - Santiago Optical Channel in operation - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2480

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DMTC-7400-2480: AURA DWDM equipment in operation - Complete#

Warning

No description available

DMTC-7400-0815: La Serena-Gatehouse path design Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-12-25

Warning

No description available

DMTC-7400-2010: Gatehouse to La Serena path design Ready - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-41

  • Due: 2023-06-01

  • Completed: 2015-12-16

Warning

No description available

DMTC-7400-2050: Fiber delivery - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-03-31

Warning

No description available

DMTC-7400-2090: Report on functional fiber connections, including AURA equipment - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2017-12-01

Warning

No description available

DMTC-7400-2130: End of commissioning. Acceptance of fiber connection - Complete#

Warning

No description available

DLP-41: Mountain - Base Fiber Path Ready#

Warning

No description available

DLP-37: Summits-Gatehouse Fiber Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2210

  • Due: 2023-06-01

  • Completed: 2017-05-19

Warning

No description available

DLP-40: Summits-Gatehouse Initial Capability Ready#

Warning

No description available

DMTC-7400-2170: Gatehouse to Summit contract signed; Design Ready - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-41

  • Due: 2023-06-01

  • Completed: 2016-01-15

Warning

No description available

DMTC-7400-2210: Fiber delivery by Telefonica - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-37

  • Due: 2023-06-01

  • Completed: 2017-02-28

Warning

No description available

DMTC-7400-2250: End of commissioning and Acceptance - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-40

  • Due: 2023-06-01

  • Completed: 2017-06-30

Warning

No description available

DLP-65: Mountain - Base LSST and AURA DWM End Node Installation and Test Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2330

  • Due: 2023-06-01

  • Completed: 2018-03-31

Warning

No description available

DLP-63: Vendor selection Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-7400-2290

  • Due: 2023-06-01

  • Completed: 2016-03-31

Warning

No description available

DMTC-7400-2290: Final Selection of equipment vendor - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-63

  • Due: 2023-06-01

  • Completed: 2016-10-01

Warning

No description available

DMTC-7400-2330: LSST and AURA summit to base DWDM equipment - Complete#

Warning

No description available

DLP-36: Santiago - Miami Early Integration Ready#

Warning

No description available

DMTC-8100-2010: First Data passed on 100G ring without errors - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2015-10-01

Warning

No description available

DMTC-8100-2050: 100G ring operating at 95.0% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-10-01

Warning

No description available

DMTC-8100-2090: 100G ring operating at 97.0% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-36

  • Due: 2023-06-01

  • Completed: 2018-06-29

Warning

No description available

DMTC-8100-2130: 100G ring operating at 98.0% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2019-02-13

Warning

No description available

DMTC-8100-2170: 100G ring operating at 99.0% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2020-02-19

Contract Deliverable Report #6 100G managed ring Contract Delivery #6 Complete. Covers availability and outages.

DMTC-8100-2210: 100G ring operating at 99.99% availability - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2021-07-30

Contract Deliverable Report #7 100G managed ring Contract Delivery #7 Complete. Covers availability and outages.

DMTC-8100-2112: Miami - Boca Raton path diverse fiber - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-36

  • Due: 2023-06-01

  • Completed: 2019-02-01

Warning

No description available

DMTC-8100-2250: FIU-CIARA Management Contract Delivery #2 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-09-30

Warning

No description available

DMTC-8100-2290: FIU-CIARA Management Contract Delivery #3 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2017-09-01

Warning

No description available

DMTC-8100-2330: FIU-CIARA Management Contract Delivery #4 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2018-10-31

Warning

No description available

DMTC-8100-2370: FIU-CIARA Management Contract Delivery #5 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2019-12-31

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DMTC-8100-2410: FIU-CIARA Management Contract Delivery #6 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2020-09-30

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DMTC-8100-2450: FIU-CIARA Management Contract Delivery #7 Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2021-09-30

Level of effort management task, annual report delivered, covering all activities, meetings, conferences/workshops, circuits, tools, tests, staffing, budget/expenditures.

DLP-33: Santiago - Boca Raton Full Integration Ready#

Spectrum link operational at 100 Gbps from Santiago - Boca Raton, including terrestrial segment from Santiago - Sao Paolo, and Monet submarine cable from Sao Paolo - Boca Raton.

DMTC-8100-2490: First light test report - Complete  Contract#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2018-06-01

Warning

No description available

DMTC-8100-2530: Data passing test report - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-33

  • Due: 2023-06-01

  • Completed: 2019-05-31

Contract Deliverable Report #4 Spectrum Complete. Test of first data passing over Santiago - Boca Raton Spectrum link.

DMTC-8100-2474: Agreement with Monet Cable Partner signed Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2017-04-28

Warning

No description available

DLP-70: Miami - Chicago Early Integration Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8200-0510

  • Due: 2023-06-01

  • Completed: 2019-11-01

Miami - Atlanta, and ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DLP-69: Boca Raton - Chicago Early Integration Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8200-0510

  • Due: 2023-06-01

  • Completed: 2019-11-14

Boca Raton - Atlanta and ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DMTC-8200-0480: VLAN design finalized Miami-NCSA - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2017-03-23

Warning

No description available

DMTC-8200-0510: ESNet Request - Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Successors: DLP-70, DLP-69

  • Due: 2023-06-01

  • Completed: 2018-09-26

Warning

No description available

DMTC-8200-0600: ESNet Implementation Phase 1- Complete#

  • WBS: 02C.08.03

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2019-11-14

ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available and tested at capacity.

DM-NET-6: Summit LAN installed#

  • WBS: 02C.08.00

  • Level: 3

  • Predecessors: T&SC-2600-1545

  • Successors: LDM-503-08

  • Due: 2023-06-01

  • Completed: 2018-04-02

Warning

No description available

DMTC-6800-1325: Network Verification Pre-Tests#

  • WBS: 02C.08.00

  • Level: 4

  • Successors: DLP-33

  • Due: 2023-06-01

  • Completed: 2020-02-28

Warning

No description available

DLP-43: Fiber Delivery Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8000-0760

  • Due: 2023-06-01

  • Completed: 2017-02-28

Warning

No description available

DLP-46: Full Capacity over Diverse path Ready#

  • WBS: 02C.08.03

  • Level: 3

  • Predecessors: DMTC-8000-0990, S19F

  • Due: 2023-09-01

  • Completed: 2021-08-31

Both primary (100 Gbps) and secondary (40 - 100 Gbps) circuits from La Serena - Santiago are operational and tested with loads to full capacity.

DLP-465: Install networking for normal Base Center operations for full-camera scale#

  • WBS: 02C.08.01

  • Level: 3

  • Predecessors: F20F

  • Successors: LSST-1580, CAMM8350

  • Due: 2023-09-05

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Base Data Center and Base Office Building technical closets, and cabling connecting them for the Base Campus and Control Networks. Tested with data volumes and access points equivalent to operational staffing and control servers for full camera observing.

DLP-468: Provide networking for formal Chilean DAC production at full-camera scale#

  • WBS: 02C.08.02

  • Level: 3

  • Predecessors: DLP-36, F20F

  • Successors: LDM-503-11

  • Due: 2023-09-05

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with full camera observing.

DLP-467: Install networking for full-camera-scale Chilean DAC serving on order 80 people#

  • WBS: 02C.08.02

  • Level: 3

  • Predecessors: F20F

  • Successors: LDM-503-11

  • Due: 2023-09-05

  • Completed: 2020-11-30

Installed the Base LAN equipment in the Chilean DAC. Tested with data volumes and access points equivalent to anticipated DAC user workload contemporaneous with commissioning camera observing.

DMTC-6800-1350: Network End-to-End Verification Test#

  • WBS: 02C.08.00

  • Level: 4

  • Predecessors: SITCOM-130

  • Successors: LSST-1580

  • Due: 2025-06-17

  • Completion pending

Warning

No description available

02C.10: Science Quality and Reliability Engineering#

_images/graph_02C.10.png

Fig. 9 Relationships between milestones in WBS 02C.10 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

DLP-346: SQuaSH Prototype using existing architectural components deployed#

  • WBS: 02C.10.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2017-07-25

Warning

No description available

DLP-345: Prototype metrics collection system deployed#

  • WBS: 02C.10.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-05-31

Warning

No description available

DLP-344: Release SQuaSH-in-a-box automated deployment platform#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: F17F

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DLP-342: Publish updated introductory software guide to developing with the stack#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: DLP-340

  • Due: 2023-06-01

  • Completed: 2016-04-08

Warning

No description available

DLP-341: Deploy stack user support and discussion forum#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: DLP-340

  • Successors: DLP-337

  • Due: 2023-06-01

  • Completed: 2016-04-08

Warning

No description available

DLP-340: Operating system testing for public stack releases increased#

  • WBS: 02C.10.02

  • Level: 3

  • Successors: DLP-342, DLP-341

  • Due: 2023-06-01

  • Completed: 2016-04-08

Warning

No description available

DLP-352: Metrics of Interest integrated in running stack or post-tests#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: F17F

  • Successors: S18S

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DLP-351: Intermediate Performance Metrics routinelly calculated and published#

  • WBS: 02C.10.02

  • Level: 3

  • Due: 2023-06-01

  • Completed: 2016-05-31

Warning

No description available

DLP-337: Release new stack documentation infrastructure#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: DLP-341, DLP-334

  • Due: 2023-06-01

  • Completed: 2016-05-31

Warning

No description available

DLP-334: SQuaRE release improved or changed CI system#

  • WBS: 02C.10.02

  • Level: 3

  • Successors: DLP-337

  • Due: 2023-06-01

  • Completed: 2016-04-08

Warning

No description available

DM-SQRE-1: Project internal Jupyter notebook service#

  • WBS: 02C.10.02

  • Level: 3

  • Successors: LDM-503-01, DM-SQRE-2

  • Due: 2023-06-01

  • Completed: 2017-11-01

Warning

No description available

DM-SQRE-2: Notebook service capabilities are suitable for the commissioning team to develop notebooks for its needs#

Members of the integration and commissioning team are able to develop notebooks in nublado that are capable of the interactive data exploration that is suitable to their needs to support ComCam commissioning. Data is available via a Butler to a mounted filesystem. They are able to work effectively, though in some cases their user experience can be improved. They are primarily trained in person by bootcamps and may require extensive support via Slack. The service may be intermittently unavailable due to engineering work.

DM-SQRE-3: Notebook service stable for commissioning and other internal project users#

Members of the integration and commissioning team as well as science pipelines developers use nublado to support their interactive data exploration needs relating to ComCam commissioning. They are able to work effectively though in some cases their user experience can be improved. Access to EFD data is easily available from notebooks. Data is available via Butler to a mounted filesystem. There is a curated library of notebooks modeling common tasks and a way to automatically verify that they are in working condition. Users can refer to documentation for answers to common problems. The service is stable and interruptions to the service are primarily notified well in advance and co-ordinated with the commissioning team to minimize disruption to the extent that the underlying infrastructure permits it.

DM-SQRE-6: IVOA Server-side Operations for Data Access API#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: F20F

  • Successors: LSST-1580, DM-SQRE-7

  • Due: 2023-09-05

  • Completed: 2022-02-28

Warning

No description available

DM-SQRE-7: IVOA-compliant image metadata services#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: F20F, DM-SQRE-6

  • Successors: LSST-1580

  • Due: 2023-09-05

  • Completed: 2022-06-30

Warning

No description available

DM-SQRE-4: Notebook service ready for verification & validation#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: DM-SQRE-3

  • Successors: LDM-503-14

  • Due: 2024-03-04

  • Completion pending

Science pipeline developers and members of the verification and validation science team use nublado to support their interactive data exploration needs relating to ComCam commissioning. They are able to work effectively though in some cases their user experience can be improved. As well as access to EFD, catalogue and pixel data, they have access to any available APIs to the workflow/batch system for non-trivial data processing. There are curated libraries of continuously integrated notebooks modeling common tasks and they is a reasonable workflow to allow Github-based collaboration on those notebooks. Users can refer to documentation for how to get started and for answers to common problems. The service is stable with minimal interruptions, but may not always scale well to large number of users.

DM-SQRE-5: Notebook service ready for general science users#

  • WBS: 02C.10.02

  • Level: 3

  • Predecessors: LSST-1560

  • Successors: LSST-1580

  • Due: 2025-06-17

  • Completion pending

Members of the science collaborations or other non-project staff with access to data use nublado to support their interactive scientific data exploration. They are able to share data with each other using LSP sharing mechanisms and can access data without a direct filesystem mount (eg VO services, remote Butler etc). The service scales well to large number of users provided the infrastructure is sized for them. Users can primarily rely on documentation for support, and use a helpdesk system for questions. There is robust functionality for performing large distributed catalog operations and for visually exploring LSSTCam sized images.

02C.11: Security#

_images/graph_02C.11.png

Fig. 10 Relationships between milestones in WBS 02C.11 and their immediate predecessors and successors. Ellipses correspond to milestones within this WBS element; rectangles to those in other elements. Blue milestones have been completed; orange milestones are overdue.#

A9440: START: Data Security#

  • WBS: 02C.11

  • Level: Undefined

  • Predecessors: LSST-1440

  • Due: 2022-10-03

  • Completed: 2022-10-03

Warning

No description available

DMTC-021102-1175: COMP: Firewalls and Physical Security#

  • WBS: 02C.11.02

  • Level: 4

  • Successors: LSST-1580

  • Due: 2024-03-29

  • Completion pending

Warning

No description available

DMTC-021101-1155: COMP: Encryption#

  • WBS: 02C.11.01

  • Level: 4

  • Predecessors: SUMMIT-2976

  • Successors: LSST-1580

  • Due: 2024-04-23

  • Completion pending

Warning

No description available

DMTC-021103-1085: COMP: Delayed Data Store Installed#

  • WBS: 02C.11.03

  • Level: 4

  • Due: 2024-09-26

  • Completion pending

Warning

No description available

Bibliography#

[1]

[DMTR-302]. Yusra AlSayyad. LDM-503-13a: Science Pipelines Fall 2020 Release Test Plan and Report. 2021. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-302.lsst.io/

[2] (1,2)

[LDM-533]. Eric C. Bellm and John D. Swinbank. LSST Level 1 System Test Specification. 2019. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-533.lsst.io/

[3]

[LDM-534]. Jim Bosch, Hsin-Fang Chiang, Michelle Gower, and John D. Swinbank. LSST Level 2 System Test Specification. 2021. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-534.lsst.io/

[4]

[DMTR-121]. Michelle Butler. LDM-503-8 Spectrograph Data Acquisition Test Plan and Report. 2019. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-121.lsst.io/

[5]

[DMTR-102]. Michelle Butler. LDM-503-8b (Small Scale CCOB Data Access) Test Plan and Report. 2019. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-102.lsst.io/

[6]

[DMTR-181]. Michelle Butler. LDM-503-10: DAQ Validation Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-181.lsst.io/

[7]

[DMTR-182]. Michelle Butler. LDM-503-10b: Large Scale CCOB Data Access Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-182.lsst.io/

[8]

[DMTR-171]. Michelle Butler. LDM-503-6: ComCam Interface Verification Readiness Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-171.lsst.io/

[9] (1,2)

[LDM-538]. Michelle Butler, Jim Parsons, and Michelle Gower. LSST DM Raw Image Archiving Service Test Specification. 2021. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-538.lsst.io/

[10]

[DMTR-271]. Jeffrey Carlin. LDM-GEN3: Gen 3 Butler Acceptance Testing Test Plan and Report. 2022. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-271.lsst.io/

[11]

[LDM-540]. G. P. Dubois-Felsmann, L.P. Guy, J. Carlin, K.S. Krughoff, C. Slater, and M. Wood-Vasey. LSST Science Platform Test Specification. 2020. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-540.lsst.io/

[12]

[DMTR-161]. Gregory Dubois-Felsmann. LDM-503-10a: LSP with Authentication and TAP Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-161.lsst.io/

[13]

[DMTR-301]. Gregory Dubois-Felsmann. LDM-503-14a: RSP redeployed on the Interim Data Facility (IDF), ready for DP0.1 Test Plan and Report. 2021. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-301.lsst.io/

[14]

[DMTR-381]. Gregory Dubois-Felsmann. LDM-503-RSPa: RSP on the Interim Data Facility (IDF) is ready for DP0.2 Test Plan and Report. 2023. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-381.lsst.io/

[15]

[LDM-554]. Gregory Dubois-Felsmann, David Ciardi, Fritz Mueller, and Frossie Economou. Data Management LSST Science Platform Requirements. 2019. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-554.lsst.io/

[16]

[LDM-542]. Gregory Dubois-Felsmann, Frossie Economou, Kian-Tat Lim, Fritz Mueller, Stephen R. Pietrowicz, and Xiuqin Wu. Science Platform Design. 2019. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-542.lsst.io/

[17]

[DMTR-231]. Robert Gruendl. LDM-503-11a: ComCam OPS Readiness Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-231.lsst.io/

[18] (1,2)

[LDM-643]. Margaret Johnson and Robert A. Gruendl. Proposed DM OPS Rehearsals. 2020. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-643.lsst.io/

[19]

[LSE-163]. M. Jurić, T. Axelrod, A.C. Becker, J. Becla, E. Bellm, J.F. Bosch, D. Ciardi, A.J. Connolly, G.P. Dubois-Felsmann, F. Economou, M. Freemon, M. Gelman, R. Gill, M. Graham, L.P. Guy, Ž. Ivezić, T. Jenness, J. Kantor, K.S. Krughoff, K-T Lim, R.H. Lupton, F. Mueller, D. Nidever, W. O'Mullane, M. Patterson, D. Petravick, D. Shaw, C. Slater, M. Strauss, J. Swinbank, J.A. Tyson, M. Wood-Vasey, and X. Wu. Data Products Definition Document. 2023. Vera C. Rubin Observatory . URL: https://lse-163.lsst.io/

[20]

[LSE-319]. M. Jurić, D. Ciardi, G.P. Dubois-Felsmann, and L.P. Guy. LSST Science Platform Vision Document. 2019. Vera C. Rubin Observatory . URL: https://lse-319.lsst.io/

[21]

[LDM-732]. Jeff Kantor. Vera C. Rubin Observatory Network Verification Document. 2020. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-732.lsst.io/

[22]

[DMTR-291]. Simon Krughoff. DM-503-EFDa: EFD on Summit for M1/M3 Test Plan and Report. 2021. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-291.lsst.io/

[23]

[LSE-400]. K-T Lim. Header Service Interface. 2019. URL: https://lse-400.lsst.io

[24] (1,2)

[LDM-148]. K.-T. Lim, J. Bosch, G. Dubois-Felsmann, T. Jenness, J. Kantor, W. O'Mullane, D. Petravick, G. Comoretto, and the DM Leadership Team. Data Management System Design. 2020. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-148.lsst.io/

[25]

[RTN-001]. William O'Mullane. Data Preview 0: Definition and planning. 2024. Vera C. Rubin Observatory Technical Note. URL: https://rtn-001.lsst.io/

[26] (1,2,3,4,5,6)

[LDM-564]. William O'Mullane, Frossie Economou, Tim Jenness, Andrew Loftus, and John D. Swinbank. Data Management Releases for Verification/Integration. 2023. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-564.lsst.io/

[27] (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31)

[LDM-151]. J.D. Swinbank, T. Axelrod, A.C. Becker, J. Becla, E. Bellm, J.F. Bosch, H. Chiang, D.R. Ciardi, A.J. Connolly, G.P. Dubois-Felsmann, F. Economou, M. Fisher-Levine, M. Graham, Ž. Ivezić, M. Jurić, T. Jenness, R.L. Jones, J. Kantor, S. Krughoff, K-T. Lim, R.H. Lupton, F. Mueller, D. Petravick, P.A. Price, D.J. Reiss, D. Shaw, C. Slater, M. Wood-Vasey, X. Wu, P. Yoachim, and _for the LSST Data Management_. Data Management Science Pipelines Design. 2020. Vera C. Rubin Observatory Data Management Controlled Document. URL: https://ldm-151.lsst.io/

[28]

[DMTR-112]. John Swinbank. LDM-503-07 (Camera Data Processing) Test Plan and Report. 2019. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-112.lsst.io/

[29]

[DMTR-192]. John Swinbank. LDM-503-11b: Science Pipelines Fall 2019 Release Test Plan and Report. 2020. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-192.lsst.io/

[30]

[DMTR-111]. John D. Swinbank. LDM-503-09a (Science Pipelines Fall 2018 Release) Test Plan and Report. 2019. Vera C. Rubin Observatory Data Management Test Report. URL: https://dmtr-111.lsst.io/