Repository navigation
DM-56308: updated DP2 NB 306 1 and new 306 2 (extragal transients) #220
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,7 +22,7 @@ | |
| "Data Release: [Data Preview 2](https://dp2.lsst.io/)\\\n", | ||
|
Contributor
There was a problem hiding this comment.
Contributor
Author
There was a problem hiding this comment. Note to self: specify this is not nearest-neighbor, this will return all sources within the radius so multiple matches are possible. It's why the 2" gets used here, to demonstrate this, so make it clearer for the user.
Contributor
There was a problem hiding this comment. Perhaps mention that the cases when flux <=0 can be filtered out from the analysis, if the magnitude is needed (from forced photometry on the science image). Reply via ReviewNB
Contributor
Author
There was a problem hiding this comment. Note to self: mention again that in the future, with no template contamination, transient fluxes should all be >0 in the difference image, and so being careful to retain flux<0 data points is a DP2-specific issue, because of how templates are made. |
||
| "Container Size: Large\\\n", | ||
| "LSST Science Pipelines version: r30.0.11\\\n", | ||
| "Last verified to run: 2026-10-05\\\n", | ||
| "Repository: [github.com/lsst/tutorial-notebooks](https://github.com/lsst/tutorial-notebooks)\\\n", | ||
| "DOI: [10.11578/rubin/dc.20250909.20](https://doi.org/10.11578/rubin/dc.20250909.20)" | ||
| ] | ||
|
|
@@ -36,7 +36,7 @@ | |
| "\n", | ||
| "**LSST data products:** `ForcedSourceOnDiaObject`, `DiaObject`, `Visit`\n", | ||
| "\n", | ||
| "**Packages:** `lsst.rsp`\n", | ||
| "\n", | ||
| "**Credit:**\n", | ||
| "Originally developed by the Rubin Community Science team with feedback from Eric Bellm.\n", | ||
|
|
@@ -58,11 +58,10 @@ | |
| "\n", | ||
| "This notebook demonstrates how to obtain and plot a difference-image light curve of an extragalactic transient.\n", | ||
| "\n", | ||
| "**For light curves of extragalactic transients, it is recommended to use forced flux measured on the difference image.**\n", | ||
| "Using the difference-image photometry ensures that any underlying host galaxy light does not contaminate the flux measurement, as can happen if science-image (or direct- or visit-image) photometry is used.\n", | ||
| "Using the forced photometry ensures that a data point exists for every observation that overlapped the transient's coordinates, not just those in which it was detected at $>5\\sigma$ in the difference image.\n", | ||
| "Difference-image fluxes can be negative and thus should not be converted to magnitudes, as without care this can lead to missing data points (especially observations with non-detections, which can be scientifically very useful for transients).\n", | ||
| "\n", | ||
| "The forced difference-image photometry is stored in the `ForcedSourceOnDiaObject` table, and the relevant columns are:\n", | ||
| "* `diaObjectId`: Unique DiaObject identifier\n", | ||
|
|
@@ -72,7 +71,11 @@ | |
| "\n", | ||
| "Observation dates (MJDs) are stored in the `Visit` table, in column `expMidptMJD` (exposure midpoint MJD), and are obtained with a table join.\n", | ||
| "\n", | ||
| "**Be aware that some template images contain transient flux.**\n", | ||
| "In the future, the template images will be created from the previous year's data, but for DP2 this is not possible.\n", | ||
| "The third (33%) of the visit images with the best seeing were combined to create the template images, and in some cases flux from transient objects \"contaminates\" the template image (e.g., when many images of a given sky region were obtained in a short amount of time). \n", | ||
| "In these cases, transients will appear to have negative difference-image flux in some epochs.\n", | ||
| "This tutorial will demonstrate how to identify such cases.\n", | ||
| "\n", | ||
| "**Related tutorials:** There are 200-level tutorials on `difference_images` as well as the `DiaSource`, `DiaObject` and `ForcedSourceOnDiaObject` catalogs." | ||
| ] | ||
|
|
@@ -95,7 +98,6 @@ | |
| "`matplotlib`, a comprehensive library for data visualization\n", | ||
| "(<a href=\"https://matplotlib.org/\">matplotlib.org</a>; \n", | ||
| "<a href=\"https://matplotlib.org/stable/gallery/index.html\">matplotlib gallery</a>).\n", | ||
| "From the `lsst` package, import modules for accessing the Table Access Protocol (TAP) service and image display functions from the LSST Science Pipelines (<a href=\"https://pipelines.lsst.io/\">pipelines.lsst.io</a>)." | ||
| ] | ||
| }, | ||
|
|
@@ -110,7 +112,8 @@ | |
| "import numpy as np\n", | ||
| "from lsst.rsp import RSPDiscovery\n", | ||
| "from lsst.utils.plotting import (get_multiband_plot_colors,\n", | ||
| " get_multiband_plot_symbols,\n", | ||
| " set_rubin_plotstyle)" | ||
| ] | ||
| }, | ||
| { | ||
|
|
@@ -155,6 +158,7 @@ | |
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "set_rubin_plotstyle()\n", | ||
| "filter_colors = get_multiband_plot_colors()\n", | ||
| "filter_names = filter_colors.keys()\n", | ||
| "filter_symbols = get_multiband_plot_symbols()" | ||
|
|
@@ -167,20 +171,33 @@ | |
| "source": [ | ||
| "## 2. Retrieve the light curve data\n", | ||
| "\n", | ||
| "Light curve data - forced difference-image fluxes - are retrieved from the `ForcedSourceOnDiaObject` table, using the `diaObjectId` column.\n", | ||
| "The `ForcedSourceOnDiaObject` table cannot be queried by coordinate; all spatial queries for time-domain objects must be done on the `DiaObject` table.\n", | ||
| "\n", | ||
| "* Section 2.1 demonstrates the two step process to retrieve first the `diaObjectId` for the extragalactic transient using a spatial query on the `DiaObject` table.\n", | ||
| "* Section 2.2 demonstrates how to then retrieve the light curve data from the `ForcedSourceOnDiaObject` table using a query on the `diaObjectId` column.\n", | ||
| "* Section 2.3 demonstrates how to combine these two steps into a single query using table joins." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "f49ec673-dd38-49a3-9c90-26ef1bf5c537", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### 2.1. Spatial query for the diaObject\n", | ||
| "\n", | ||
| "**Spatial queries should be done on `DiaObject`, not on the source tables.**\n", | ||
| "Spatial queries for time-domain objects are done on the `DiaObject` table, *not* on the tables containing the measured fluxes at the same location in every image.\n", | ||
| "Spatial queries on `ForcedSourceOnDiaObject` (or `DiaSource`) are inefficient.\n", | ||
| "Coordinates are not included in the `ForcedSourceOnDiaObject` table for this reason.\n", | ||
| "\n", | ||
| "**Set the cross-match radius a bit high (2\" instead of 1\") for this demonstration.**\n", | ||
| "Difference-image detections are associated within a 1\" radius, and typically that is a fine search radius to use for point-source cross-match (and is used in Section 2.3).\n", | ||
| "However in the query below a 2\" radius is used to mimic a situation in which, e.g., the transient was detected in a lower-resolution survey, in order to demonstrate how to deal with multiple potential cross-match results.\n", | ||
| "\n", | ||
| "For scenarios in which cross-match for a *set* of coordinates is desired, instead of a single coordinate, see the tutorials on cross-match to user-uploaded tables. \n", | ||
| "\n", | ||
| "Define the coordinates of a known transient ([SN2025mbp](https://www.wis-tns.org/object/2025mbp)) and the search radius. Query the `DiaObject` table and retrieve the `diaObjectId`, the number of detections (`nDiaSources`), and the offset distance between all matches and the search coordinates. Retrieve the results as an Astropy table." | ||
| ] | ||
| }, | ||
| { | ||
|
|
@@ -192,7 +209,9 @@ | |
| "source": [ | ||
| "ra_targ = 151.592579\n", | ||
| "dec_targ = 1.158333\n", | ||
| "\n", | ||
| "search_radius = 2.0/3600.0\n", | ||
| "\n", | ||
| "query = \"\"\"SELECT diaObjectId, nDiaSources,\n", | ||
| " DISTANCE(POINT('ICRS', ra, dec), POINT('ICRS', {}, {}))\n", | ||
| " FROM dp2.DiaObject\n", | ||
|
|
@@ -241,8 +260,9 @@ | |
| "source": [ | ||
| "### 2.2. Query for forced photometry\n", | ||
| "\n", | ||
| "Create a query to retrieve the difference-image forced flux, its error, and the band (filter) from the `ForcedSourceOnDiaObject` table, for all rows associated with the `diaObjectId`.\n", | ||
| "Use a table join to retrieve the MJD of the observation from the `Visit` table.\n", | ||
| "Retrieve the results as an Astropy table, then print the table length (number of measurements)." | ||
| ] | ||
| }, | ||
| { | ||
|
|
@@ -253,6 +273,7 @@ | |
| "outputs": [], | ||
| "source": [ | ||
| "use_diaObjectId = 786352299365629972\n", | ||
| "\n", | ||
| "query = \"\"\"SELECT fsodo.psfDiffFlux, fsodo.psfDiffFluxErr,\n", | ||
| " fsodo.band, v.expMidptMJD\n", | ||
| " FROM dp2.ForcedSourceOnDiaObject AS fsodo\n", | ||
|
|
@@ -266,6 +287,7 @@ | |
| " job.raise_if_error()\n", | ||
| "assert job.phase == 'COMPLETED'\n", | ||
| "flux_table = job.fetch_result().to_table()\n", | ||
| "\n", | ||
| "print(len(flux_table))" | ||
| ] | ||
| }, | ||
|
|
@@ -287,6 +309,64 @@ | |
| "# flux_table" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "b7d44df7-69bd-4063-948d-67d082700577", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Delete the first two query results, as they will be repeated as a single three-table join in the next section." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "694f0280-604d-4ff2-b2c6-426c854fbe51", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "del diao_table, flux_table" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "c3039cb5-ffff-4a86-bc0d-228c28a1c854", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### 2.3. Combine steps into a triple-table query\n", | ||
| "\n", | ||
| "**Lower the cross-match radius to 1\"** so that only one `diaObject` is returned from the spatial query.\n", | ||
| "\n", | ||
| "Combine the two queries above into a single query with a triple-table join. Execute the query job and return the results as an Astropy table, and print the number of rows." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "39efd36a-03a5-4bed-9a00-e1ad623db1ba", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "search_radius = 1.0/3600.0\n", | ||
| "\n", | ||
| "query = \"\"\"SELECT diao.diaObjectId, fsodo.psfDiffFlux, fsodo.psfDiffFluxErr,\n", | ||
| " fsodo.band, v.expMidptMJD\n", | ||
| " FROM dp2.DiaObject AS diao\n", | ||
| " JOIN dp2.ForcedSourceOnDiaObject AS fsodo ON fsodo.diaObjectId = diao.diaObjectId\n", | ||
| " JOIN dp2.Visit AS v ON v.visit = fsodo.visit\n", | ||
| " WHERE CONTAINS(POINT('ICRS', diao.ra, diao.dec), CIRCLE('ICRS', {}, {}, {})) = 1\n", | ||
| " \"\"\".format(ra_targ, dec_targ, search_radius)\n", | ||
| "job = tap_service.submit_job(query)\n", | ||
| "job.run()\n", | ||
| "job.wait(phases=['COMPLETED', 'ERROR'])\n", | ||
| "print('Job phase is', job.phase)\n", | ||
| "if job.phase == 'ERROR':\n", | ||
| " job.raise_if_error()\n", | ||
| "assert job.phase == 'COMPLETED'\n", | ||
| "flux_table = job.fetch_result().to_table()\n", | ||
| "\n", | ||
| "print(len(flux_table))" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "89907542-962f-4dbd-9632-d9f1788c846d", | ||
|
|
@@ -371,7 +451,9 @@ | |
| "\n", | ||
| "Do not convert magnitudes to difference-image fluxes, as any negative fluxes will be lost. Negative difference-image fluxes are not erroneous -- they simply mean the time-varying object was brighter in the template than in the science image.\n", | ||
| "\n", | ||
| "Re-plot the light curve in Figure 2, but with magnitudes, to show the error message and that some data points go missing. \n", | ||
| "\n", | ||
| ">**Warning:** There will be a pink \"RuntimeWarning\" to indicate that negative values were passed to `np.log10`." | ||
| ] | ||
| }, | ||
| { | ||
|
|
@@ -422,6 +504,7 @@ | |
| "outputs": [], | ||
| "source": [ | ||
| "use_diaObjectId = 786352299365629972\n", | ||
| "\n", | ||
| "query = \"\"\"SELECT psfFlux, psfFluxErr, band, midpointMjdTai\n", | ||
| " FROM dp2.DiaSource AS dias\n", | ||
| " WHERE diaObjectId = {}\"\"\".format(use_diaObjectId)\n", | ||
|
|
@@ -433,6 +516,7 @@ | |
| " job.raise_if_error()\n", | ||
| "assert job.phase == 'COMPLETED'\n", | ||
| "dias_table = job.fetch_result().to_table()\n", | ||
| "\n", | ||
| "print(len(dias_table))" | ||
| ] | ||
| }, | ||
|
|
@@ -483,13 +567,17 @@ | |
| "id": "014f78e1-1913-4a8f-8f0e-f17e9a10e6a6", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "> **Figure 4:** The top panel is the same as Figure 2 (forced PSF difference-image fluxes from the `ForceSourceOnDiaObject` table), but the bottom panel is the detected difference-image flux from the `DiaSource` table.\n", | ||
| "\n", | ||
| "Notice that data points for some epochs, especially those in which the difference-image flux was close to 0, are missing from the bottom plot. For forced, difference-image photometry, a flux of zero is still an interesting and useful data point - but these points are missed if only the detections in the `DiaSource` table are used instead of all forced measurements in `ForcedSourceOnDiaObject`.\n", | ||
| "\n", | ||
| "Notice also that the first epochs have difference-image fluxes less than zero, which is indicative of some template contamination in those filters." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "346764ae-c246-48cb-9a81-a3da86209665", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [] | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Forced photometry on the science/visit image for the light curve?
Reply via ReviewNB
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note to self: emphasize that fluxes should be used with difference-image measurements (do it in boldface like the other important points).