Configuration

Configuration#

The configuration file uses the TOML format. There are many options that have defaults, sometimes even sensible ones. In general, if an entire section is missing, the operation will be excluded. Note that sections for the configuration (e.g., [calibration], [polarimetry]) can be in any order, although keeping them in the same order as the pipeline execution may be clearer.

We use semantic versioning, so there are certain guarantees about the backwards compatibility of the pipeline, which means you don’t have to have the exact same version of vampires_dpp as in the configuration- merely a version that is compatible.

File Outputs

We notate where the pipeline ends up saving data files with the “💾” emoji.

Pipeline#

class vampires_dpp.pipeline.config.PipelineConfig(**data)#

Data Processing Pipeline options

The processing configuration is all done through this class, which can easily be converted to and from TOML. The options will set the processing steps in the pipeline. An important paradigm in the processing pipeline is skipping unnecessary operations. That means if a file already exists, the pipeline will only reprocess it if the force flag is set, which will reprocess all files for that step (and subsequent steps), or if the input file or files are newer. You can try this out by deleting one calibrated file from a processed output and re-running the pipeline.

File Outputs

  • Auxilliary files in aux/

    • Copy of config., centroid file, astrometry file, mean PSFs, filter curve(s), synth. PSF(s).

  • Data products (ADI cubes, output file header table) in products/

  • Difference images in diff/

    • diff/single/ and diff/double/

Parameters:

  • name (str) – filename-friendly name used for outputs from this pipeline. For example “20230101_ABAur”

  • dpp_version (str) – The version of vampires_dpp that this configuration file is valid with. Typically not set by user.

  • coronagraphic (bool) – If true will use coronagraphic routines for processing.

  • planetary (bool) – If true will use planetary routines for processing.

  • save_adi_cubes (bool) – If true, will save ADI cubes and derotation angles in product directory.

  • target (TargetConfig | None) – If set, provides options for target object, primarily coordinates. If not set, will use header values.

  • combine (CombineConfig) – Options for frame combinations

  • calibrate (CalibrateConfig) – Options for basic image calibration

  • analysis (AnalysisConfig) – Options for PSF/flux analysis in collapsed data

  • frame_select (FrameSelectConfig) – Options for frame selection

  • align (AlignmentConfig) – Options for frame alignment

  • coadd (CoaddConfig) – Options for coadding image cubes

  • specphot (SpecphotConfig) – If set, provides options for spectrophotometric calibration. If not set, will leave data in units of adu.

  • diff_images (DiffImageConfig) – Diagnostic difference imaging options. Double-differencing requires an FLC.

  • polarimetry (PolarimetryConfig | None) – If set, enables and provides settings for polarimetric differential imaging (PDI).

classmethod from_file(filename)#

Load configuration from TOML file

Parameters:

filename (PathLike) – Path to TOML file with configuration settings.

Raises:

ValueError – If the configuration version is not compatible with the current vampires_dpp version.

Examples

>>> Pipeline.from_file("config.toml")
save(filename)#

Save configuration settings to TOML file

Parameters:

filename (PathLike) – Output filename

to_toml()#

Create serializable TOML string

Return type:

str

Target Information#

class vampires_dpp.pipeline.config.TargetConfig(**data)#

Astronomical coordinate options.

Tip: GAIA

This can be auto-generated wtih GAIA coordinate information through the command line dpp new interface.

Parameters:

  • name (str) – SIMBAD-friendly target name

  • ra (str) – Right ascension in sexagesimal hour angles

  • dec (str) – Declination in sexagesimal degrees

  • parallax (float) – parallax of system in mas

  • pm_ra (float) – Proper motion of RA axis in mas/yr, by default 0.

  • pm_dec (float) – Proper motion of DEC axis in mas/yr, by default 0.

  • frame (str) – Coordinate reference frame, by default “icrs”.

  • obstime (str) – Observation time as a string, by default “J2016” (to coincide with GAIA coordinates)

File combination#

class vampires_dpp.pipeline.config.CombineConfig(**data)#

Frame combination options

VAMPIRES data comes in many shapes and sizes. We like to think the core data product is every individual frame, and therefore the FITS cubes we started with does not have to define the boundaries of our data analysis. There are two methods for reshuffling data, currently:

  1. “cube” – this method will effectively do nothing; the data will be combined by their original FITS cubes

  2. “pdi” – this method will combine all frames from a single HWP angle, and is required for polarimetry

When data is combined it will become a single FITS file

Parameters:

  • method (Literal['cube', 'pdi'])

  • save_intermediate (bool) – If true, will save the combined data cubes into the <output>/combined folder (WARNING can lead to insane data volume)

Calibration#

class vampires_dpp.pipeline.config.CalibrateConfig(**data)#

Config for general image calibration.

The calibration strategy is generally

  1. Load data and fix header values

  2. Calculate precise coordinates if TargetConfig is used in pipeline

  3. Background subtraction

  4. (Optional) flat-field normalization

  5. (Optional) bad pixel correction

  6. Flip camera 1 data along y-axis

We use a file-matching approach for calibrations to try and flexibly use the calibration data you have, even if it’s not the ideal calibration file or is from a different night. The file matching will always require the calibrations to have the same pixel crop (both size and location) and detector read mode. For background files, we’ll try and find files with the same exposure time and detector gain, but will accept others. Flat files will try and match detector gain, filter, and exposure time, in that order. For all files, if there are multiple matches we will select the single file closest in time.

File Outputs

  • If save_intermediate is true, will save calibrated data to calibrated/

Parameters:

  • calib_directory (Path | None) – Path to calibration file directory, if not provided no calibration will be done, regardless of other settings.

  • back_subtract (bool) – If true will look for background files in calib_directory and subtract them if found. If not found, will subtract detector bias value.

  • flat_correct (bool) – If true will look for flat files in calib_directory and perform flat normalization if found.

  • fix_bad_pixels (bool) – If true, will run adaptive sigma-clipping algorithm for one iteration on each frame and correct bad pixels. By default false.

  • reproject – If true, will use custom astrometry solution to warp frame

  • save_intermediate (bool) – If true, will save intermediate calibrated data to calibrated/ folder.

Analysis#

class vampires_dpp.pipeline.config.AnalysisConfig(**data)#

PSF modeling and analysis options.

File Outputs

  • For each file an NPZ <https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html>_ file is created in metrics/

    • Keys are metrics/centroids/statistics

    • Values are arrays with dimensions (nfields, npsfs, nframes)

Parameters:

  • fit_psf_model (bool) – If true, fits a PSF model to each window

  • psf_model (Literal['moffat']) – Only Moffat available right now

  • photometry (bool) – If true, will measure photometric sums in apertures at the centroid (or the DFT centroid if available)

  • phot_aper_rad (float) – Aperture radius in pixels for circular aperture photometry. If “auto”, will use the FWHM from the file header.

  • phot_ann_rad (Sequence[float] | Literal[False]) – If provided, will do local background-subtracted photometry with an annulus with the given inner and outer radius, in pixels.

  • strehl (bool) – If true, will measure the Strehl ratio by comparing the PSF peak to the synthetic PSF peak, normalized by the flux in an aperture 16 pixels wide.

  • window_size (int) – The cutout side length when getting cutouts for each PSF. Cutouts are centered around the file centroid estimate. A size of 21 is a decent size to avoid including too much of the PSF halo around any coronagraph masks.

Frame selection#

class vampires_dpp.pipeline.config.FrameSelectConfig(**data)#

Frame selection options

Parameters:

  • frame_select (bool) – If true, will use the given metric to select frames for inclusions/exclusion from each data cube.

  • metric (Literal['max', 'l2norm', 'normvar', 'strehl']) – Frame selection metric

  • cutoff (Annotated[float, Interval(gt=None, ge=0, lt=None, le=1)]) – If frame_select is provided, this is the cutoff _quantile_ (from 0 to 1), where 0.2 means 20% of the frames from each cube will be discarded according the the selection metric.

  • save_intermediate (bool) – If true, will save the frame-selected files to the <output>frame_select folder (WARNING can lead to insane data volume)

Frame alignment#

class vampires_dpp.pipeline.config.AlignmentConfig(**data)#

Frame alignment options

Parameters:

  • align (bool) – If true, data will be aligned by the give method

  • method (Literal['dft', 'com', 'peak', 'model']) – Alignment method, (if “dft” is not provided, it will not be measured at all)

  • crop_width (int) – Post-alignment crop width, should be roughly equal to FOV. Cropped data can set this lower for reduced memory footprint.

  • reproject (bool) – If true, will reproject cam2 astrometry onto cam1 for better image differences

  • save_intermediate (bool) – If true, will save the registered files to the <output>/registered folder (WARNING can lead to insance data volume)

Cube coadding#

class vampires_dpp.pipeline.config.CoaddConfig(**data)#

Frame combination options

File Outputs

  • Each input file is collapsed and saved into the collapsed/ folder if coadd is true, otherwise will save in the registered/ folder.

Parameters:

  • coadd (bool) – If true, will coadd each cube of data (where the cube is determined from the combination method). If false, the data will be saved as cubes.

  • method (Literal['median', 'mean', 'varmean', 'biweight'])

  • recenter (bool) – If true, will measure the centroid of the PSF in the collapsed frame and realign the data

  • recenter_method (Literal['dft', 'com', 'peak', 'model']) – Only used if recenter is true; method for PSF registration.

Spectrophotometric Calibration#

class vampires_dpp.pipeline.config.SpecphotConfig(**data)#

Spectrophotometric Configuration

Spectrophotometric calibration requires determining the precise conversion from detector data numbers ($adu/s$) to astronomical flux ($Jy$). We enable this through synthetic photometry of calibrated spectra. The synthetic photometry is accomplished with synphot. We offer two input types for the stellar spectrum-

  1. Calibrated spectrum data

    • Requires an absolutely calibrated spectrum

    • Data must be prepared such that calling SourceSpectrum.from_file loads the spectrum. Refer to their documentation for format information.

    • Set source as the path

  2. Stellar Model Library

    • Uses pickles uvk model library

    • Spectral type and reference magnitude required for normalizing model

The synthetic photometry is used to determine the expected flux in Jy, which is used to determine the conversion factor. This conversion factor maps from data flux in adu/s to Jy, with units of Jy s/adu. To determine the factor we take the flux metric from each collapsed frame (and average between any satellite spots) before dividing by the frame exposure time (header["EXPTIME"]). We store the conversion factor and other derivatives, such as the Vega zero-point magnitude in the FITS header of the output file. Lastly, we convert the input data pixel-by-pixel from adu to Jy using the conversion factor and integration time. We lastly convert to surface brightness by dividing each pixel by its solid angle (header["PXSCALE"]^2).

Note: combining data

Because each camera’s data is calibrated independently, when you combine cam1 and cam2 data (such as PDI, ADI post-processing) you should average the two cameras’ data to maintain accurate spectrophotometric calibration.

Parameters:

  • source (Literal['pickles'] | ~pathlib.Path | None) – Spectrum source type. If a path, must be a file that can be loaded by synphot.SourceSpectrum.from_file.

  • sptype (str | None) – Only used if source is “pickles”. Stellar spectral type. Note: must be one of the spectral types available in the pickles model atlas. Refer to the STScI documentation for more information on available spectral types.

  • mag (float | None) – Only used if source is “pickles”. Stellar reference magnitude

  • mag_band (Literal['U', 'B', 'V', 'r', 'i', 'J', 'H', 'K'] | None) – Only used if source is “pickles”. Stellar reference magnitude band

  • unit (Literal['e-/s', 'contrast', 'Jy', 'Jy/arcsec^2']) – Output unit. (Note: e-/s is the default without spectrophotometry, and source calibration will be skipped)

  • flux_metric (Literal['photometry', 'sum']) – Which frame analysis statistic to use for determining flux. “photometry” uses an aperture sum, while “sum” uses the sum in the analysis cutout window.

Polarimetry#

class vampires_dpp.pipeline.config.PolarimetryConfig(**data)#

Polarimetric differential imaging (PDI) options

Warning: experimental

The polarimetric reduction in this pipeline is an active work-in-progress. Do not consider any outputs publication-ready without further vetting and consultation with the SCExAO team.

PDI is processed after all of the individual file processing since it requires sorting the files into complete sets for the triple-differential calibration.

File Outputs

  • All PDI outputs are in pdi/

  • Top-level products include

    • Collapsed stokes cubes (and wavelength-collapsed cubes for MBI data)

    • Header table for Stokes frames, if using a difference method.

  • If using Mueller-matrices (method="leastsq" or mm_correct=True) FITS file with matrices for each input file in pdi/mm/

  • If using a difference method, will form individual Stokes frames and save in pdi/stokes/

Parameters:

  • method (Literal['triplediff', 'doublediff']) – Determines the polarization calibration method, either the double/triple-difference method (difference) or using the inverse least-squares solution from Mueller calculus (leastsq). In both cases, the Mueller matrix calibration is performed, but for the difference method data are organized into distinct HWP sets. This can result in data being discarded, however it is much easier to remove effects from e.g., satellite spots because you can median collapse the data from each HWP set, whereas for the inverse least-squares the data is effectively collapsed with a mean.

  • derotate (bool) – Derotate images to north up east left when forming Stokes images. Required for Mueller-matrix correction

  • mm_correct (bool) – Apply Mueller-matrix correction (only applicable to data reduced using the “difference” method). By default, True.

  • hwp_adi_sync (bool) – If true, will assume the HWP is in pupil-tracking mode. By default, True.

  • use_ideal_mm (bool) – If true and doing Mueller-matrix correction (mm_correct=True) will use only idealized versions for the components in the Mueller-matrix model.

  • ip_correct (bool) – If provided, will do post-hoc IP correction from the photometric sum in the given region.

  • ip_method (Literal['aperture', 'annulus']) – If ip_correct=True this determines the region type for IP measurement.

  • ip_radius (float) – The first radius for IP correction, if set. For “aperture” this is the radius, for “annulus” this is the inner radius.

  • ip_radius2 (float | None) – The second radius for IP correction. This is only used if ip_method="annulus"- this is the outer radius.

  • cyl_stokes (Literal['azimuthal', 'radial']) – If ‘azimuthal’ will calculate (Qphi, Uphi); if ‘radial’ will calculate (Qr, Ur) in final Stokes products.