Configuration

Configuration#

The configuration file uses the TOML format. There are many options that have defaults, sometimes even sensible ones. In general, if an entire section is missing, the operation will be excluded. Note that sections for the configuration (e.g., [calibration], [polarimetry]) can be in any order, although keeping them in the same order as the pipeline execution may be clearer.

We use semantic versioning, so there are certain guarantees about the backwards compatibility of the pipeline, which means you don’t have to have the exact same version of vampires_dpp as in the configuration- merely a version that is compatible.

File Outputs

We notate where the pipeline ends up saving data files with the “💾” emoji.

Pipeline #

pydantic model vampires_dpp.pipeline.config.PipelineConfig#

Data Processing Pipeline options.

The processing configuration is all done through this class, which can easily be converted to and from TOML. The options will set the processing steps in the pipeline. An important paradigm in the processing pipeline is skipping unnecessary operations. That means if a file already exists, the pipeline will only reprocess it if the force flag is set, which will reprocess all files for that step (and subsequent steps), or if the input file or files are newer. You can try this out by deleting one calibrated file from a processed output and re-running the pipeline.

File Outputs

Auxilliary files in aux/
- Copy of config., centroid file, astrometry file, mean PSFs, filter curve(s), synth. PSF(s).
Data products (ADI cubes, output file header table) in products/
Difference images in diff/
- diff/single/ and diff/double/

field name: str = ''#: Filename-friendly name used for outputs from this pipeline (e.g. ‘20230101_ABAur’).

field dpp_version: str [Required]#: Version of vampires_dpp this configuration file was authored against. Required; the loader will reject configs whose version is not SemVer-compatible with the installed package.

field coronagraphic: bool = False#: If true, use coronagraphic routines for processing.

field planetary: bool = False#: If true, use planetary routines for processing.

field save_adi_cubes: bool = False#: If true, save ADI cubes and derotation angles in the product directory.

field target: TargetConfig | None = None#: If set, provides target object options (primarily coordinates). If not set, header values are used.

field combine: CombineConfig [Optional]#: Options for frame combinations.

field calibrate: CalibrateConfig [Optional]#: Options for basic image calibration.

field analysis: AnalysisConfig [Optional]#: Options for PSF/flux analysis in collapsed data.

field frame_select: FrameSelectConfig [Optional]#: Options for frame selection.

field align: AlignmentConfig [Optional]#: Options for frame alignment.

field coadd: CoaddConfig [Optional]#: Options for coadding image cubes.

field specphot: SpecphotConfig [Optional]#: Options for spectrophotometric calibration. If unit is ‘e-/s’, calibration is skipped.

field diff_images: DiffImageConfig [Optional]#: Diagnostic difference imaging options. Double-differencing requires an FLC.

field nrm: NRMConfig | None = None#: If set, enables NRM (non-redundant masking) processing.

field polarimetry: PolarimetryConfig | None = None#: If set, enables and provides settings for polarimetric differential imaging (PDI).

classmethod from_file(filename)#

Load configuration from TOML file.

The version compatibility check runs as part of model validation; a stale or missing dpp_version field will raise during validation.

Return type:: PipelineConfig
Parameters:: filename (PathLike)

to_toml()#

Create a serializable TOML string.

Return type:: str

save(filename)#

Save configuration settings to a TOML file.

Return type:: None
Parameters:: filename (PathLike)

Target Information #

pydantic model vampires_dpp.pipeline.config.TargetConfig#

Astronomical coordinate options.

Tip: GAIA

This can be auto-generated wtih GAIA coordinate information through the command line dpp new interface.

field name: str [Required]#: SIMBAD-friendly target name

field ra: str [Required]#: Right ascension in sexagesimal hour angles

field dec: str [Required]#: Declination in sexagesimal degrees

field parallax: Annotated[float] [Required]#: Parallax of system in mas

field pm_ra: Annotated[float] = 0#: Proper motion of RA axis in mas/yr

field pm_dec: Annotated[float] = 0#: Proper motion of DEC axis in mas/yr

field frame: str = 'icrs'#: Coordinate reference frame

field obstime: str = 'J2016'#: Observation time as a string (default J2016 to coincide with GAIA coordinates)

property ra_ang#

property dec_ang#

get_coord()#

Return SkyCoord from the current parameters.

Return type:: SkyCoord

File combination #

pydantic model vampires_dpp.pipeline.config.CombineConfig#

Frame combination options

VAMPIRES data comes in many shapes and sizes. We like to think the core data product is every individual frame, and therefore the FITS cubes we started with does not have to define the boundaries of our data analysis. There are two methods for reshuffling data, currently:

“cube” – this method will effectively do nothing; the data will be combined by their original FITS cubes
“pdi” – this method will combine all frames from a single HWP angle, and is required for polarimetry

When data is combined it will become a single FITS file.

field method: Literal['cube', 'pdi'] = 'cube'#: Frame combination method. ‘cube’ keeps the original FITS cube boundaries; ‘pdi’ combines all frames from a single HWP angle.

field save_intermediate: bool = False#: If true, save the combined data cubes into the combined/ folder (WARNING: can lead to large data volume).

Calibration #

pydantic model vampires_dpp.pipeline.config.CalibrateConfig#

Config for general image calibration.

The calibration strategy is generally

Load data and fix header values
Calculate precise coordinates if TargetConfig is used in pipeline
Background subtraction
(Optional) flat-field normalization
(Optional) bad pixel correction
Flip camera 1 data along y-axis

We use a file-matching approach for calibrations to try and flexibly use the calibration data you have, even if it’s not the ideal calibration file or is from a different night. The file matching will always require the calibrations to have the same pixel crop (both size and location) and detector read mode. For background files, we’ll try and find files with the same exposure time and detector gain, but will accept others. Flat files will try and match detector gain, filter, and exposure time, in that order. For all files, if there are multiple matches we will select the single file closest in time.

File Outputs

If save_intermediate is true, will save calibrated data to calibrated/

field calib_directory: Path | None = None#: Path to calibration file directory; if not provided, no calibration will be done regardless of other settings.

field back_subtract: bool = True#: If true will look for background files in calib_directory and subtract them if found. If not found, will subtract detector bias value.

field flat_correct: bool = False#: If true will look for flat files in calib_directory and perform flat normalization if found.

field fix_bad_pixels: bool = False#: If true, run an adaptive sigma-clipping algorithm for one iteration on each frame and correct bad pixels.

field save_intermediate: bool = False#: If true, save intermediate calibrated data to the calibrated/ folder.

Analysis #

pydantic model vampires_dpp.pipeline.config.AnalysisConfig#

PSF modeling and analysis options.

File Outputs

For each file an NPZ <https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html>_ file is created in metrics/
- Keys are metrics/centroids/statistics
- Values are arrays with dimensions (nfields, npsfs, nframes)

field fit_psf_model: bool = False#: If true, fits a PSF model to each window.

field psf_model: Literal['moffat'] = 'moffat'#: PSF model type (only Moffat available right now).

field photometry: bool = True#: If true, measure photometric sums in apertures at the centroid (or DFT centroid if available).

field phot_aper_rad: Annotated[float] = 8#: Aperture radius in pixels for circular aperture photometry. If ‘auto’, uses the FWHM from the file header.

field phot_ann_rad: Union[Sequence[Annotated[float]], Literal[False]] = False#: If provided, do local background-subtracted photometry with an annulus given as (inner, outer) radius in pixels.

field strehl: bool = True#: If true, measure the Strehl ratio by comparing the PSF peak to the synthetic PSF peak (normalized by the flux in a 16-pixel aperture).

field window_size: Annotated[int] = 21#: Cutout side length when getting cutouts for each PSF; centered on the file centroid estimate. ~21 avoids including too much halo around any coronagraph masks.

Frame selection #

pydantic model vampires_dpp.pipeline.config.FrameSelectConfig#

Frame selection options.

field frame_select: bool = False#: If true, use the given metric to select frames for inclusion/exclusion from each data cube.

field metric: Literal['max', 'l2norm', 'normvar', 'strehl'] = 'strehl'#: Frame selection metric.

field cutoff: Annotated[float] = 0#: If frame_select is true, this is the cutoff quantile (0 to 1); 0.2 means 20% of frames in each cube are discarded.

field save_intermediate: bool = False#: If true, save the frame-selected files to the frame_select/ folder (WARNING: can lead to large data volume).

Frame alignment #

pydantic model vampires_dpp.pipeline.config.AlignmentConfig#

Frame alignment options.

field align: bool = True#: If true, data will be aligned by the given method.

field pad: bool = False#: If true, data will be padded so the full FOV is retained after rotation.

field method: Literal['dft', 'com', 'peak', 'model'] = 'dft'#: Alignment method (if ‘dft’ is not provided, it will not be measured at all).

field crop_width: Annotated[int] = 536#: Post-alignment crop width; should be roughly equal to FOV. Lower values reduce memory footprint.

field reproject: bool = False#: If true, reproject cam2 astrometry onto cam1 for better image differences.

field save_intermediate: bool = False#: If true, save the registered files to the registered/ folder (WARNING: can lead to large data volume).

Cube coadding #

pydantic model vampires_dpp.pipeline.config.CoaddConfig#

Frame combination options.

File Outputs

Each input file is collapsed and saved into the collapsed/ folder if coadd is true, otherwise will save in the registered/ folder.

field coadd: bool = True#: If true, coadd each cube of data (cube boundaries determined from the combination method). If false, the data is saved as cubes.

field method: Literal['median', 'mean', 'varmean', 'biweight'] = 'median'#: Coadd reduction method.

field recenter: bool = True#: If true, measure the centroid of the PSF in the collapsed frame and realign the data.

field recenter_method: Literal['dft', 'com', 'peak', 'model'] = 'dft'#: Only used if recenter is true; method for PSF registration.

Spectrophotometric Calibration #

pydantic model vampires_dpp.pipeline.config.SpecphotConfig#

Spectrophotometric Configuration

Spectrophotometric calibration requires determining the precise conversion from detector data numbers ($adu/s$) to astronomical flux ($Jy$). We enable this through synthetic photometry of calibrated spectra. The synthetic photometry is accomplished with synphot. We offer two input types for the stellar spectrum-

Calibrated spectrum data
- Requires an absolutely calibrated spectrum
- Data must be prepared such that calling SourceSpectrum.from_file loads the spectrum. Refer to their documentation for format information.
- Set source as the path
Stellar Model Library
- Uses pickles uvk model library
- Spectral type and reference magnitude required for normalizing model

The synthetic photometry is used to determine the expected flux in Jy, which is used to determine the conversion factor. This conversion factor maps from data flux in adu/s to Jy, with units of Jy s/adu. To determine the factor we take the flux metric from each collapsed frame (and average between any satellite spots) before dividing by the frame exposure time (header["EXPTIME"]). We store the conversion factor and other derivatives, such as the Vega zero-point magnitude in the FITS header of the output file. Lastly, we convert the input data pixel-by-pixel from adu to Jy using the conversion factor and integration time. We lastly convert to surface brightness by dividing each pixel by its solid angle (header["PXSCALE"]^2).

Note: combining data

Because each camera’s data is calibrated independently, when you combine cam1 and cam2 data (such as PDI, ADI post-processing) you should average the two cameras’ data to maintain accurate spectrophotometric calibration.

field unit: Literal['e-/s', 'contrast', 'Jy', 'Jy/arcsec^2'] = 'e-/s'#: Output unit. (e-/s is the default without spectrophotometry, and source calibration will be skipped)

field source: Union[Literal['pickles', 'zeropoints'], Path, None] = 'zeropoints'#: Spectrum source type. If a path, must be a file loadable by synphot.SourceSpectrum.from_file. If ‘pickles’, uses the pickles atlas. If ‘zeropoints’, uses coefficients from Lucas+2024.

field sptype: str | None = None#: Only used if source is ‘pickles’. Stellar spectral type (must be one of the spectral types in the pickles model atlas).

field mag: Optional[Annotated[float]] = None#: Only used if source is ‘pickles’. Stellar reference magnitude.

field mag_band: Optional[Literal['U', 'B', 'V', 'r', 'i', 'J', 'H', 'K']] = None#: Only used if source is ‘pickles’. Stellar reference magnitude band.

field flux_metric: Literal['photometry', 'sum'] = 'photometry'#: Which frame analysis statistic to use for determining flux. ‘photometry’ uses an aperture sum, while ‘sum’ uses the sum in the analysis cutout window.

Polarimetry #

pydantic model vampires_dpp.pipeline.config.PolarimetryConfig#

Polarimetric differential imaging (PDI) options.

Warning: experimental

The polarimetric reduction in this pipeline is an active work-in-progress. Do not consider any outputs publication-ready without further vetting and consultation with the SCExAO team.

PDI is processed after all of the individual file processing since it requires sorting the files into complete sets for the triple-differential calibration.

File Outputs

All PDI outputs are in pdi/
Top-level products include
- Collapsed stokes cubes (and wavelength-collapsed cubes for MBI data)
- Header table for Stokes frames, if using a difference method.
If using Mueller-matrices (method="leastsq" or mm_correct=True) FITS file with matrices for each input file in pdi/mm/
If using a difference method, will form individual Stokes frames and save in pdi/stokes/

field method: Literal['triplediff', 'doublediff'] = 'triplediff'#: Polarization calibration method. Difference methods organize data into HWP sets (some data may be discarded but median collapse can remove e.g. satellite spot effects). ‘leastsq’ (Mueller calculus) uses all data via mean.

field derotate: bool = True#: Derotate images to north up east left when forming Stokes images. Required for Mueller-matrix correction.

field mm_correct: bool = True#: Apply Mueller-matrix correction (only applicable to data reduced using a difference method).

field hwp_adi_sync: bool = True#: If true, assume the HWP is in pupil-progress.tracking mode.

field use_ideal_mm: bool = False#: If true and doing Mueller-matrix correction, use only idealized versions for the components in the Mueller-matrix model.

field ip_correct: bool = True#: If true, do post-hoc instrumental polarization (IP) correction from the photometric sum in the given region.

field ip_method: Literal['aperture', 'annulus'] = 'aperture'#: If ip_correct=True, this determines the region type for IP measurement.

field ip_radius: Annotated[float] = 15#: First radius for IP correction. For ‘aperture’ this is the radius; for ‘annulus’ this is the inner radius.

field ip_radius2: Optional[Annotated[float]] = None#: Second radius for IP correction (only used if ip_method=’annulus’); the outer radius.

field cyl_stokes: Literal['azimuthal', 'radial'] = 'azimuthal'#: If ‘azimuthal’ will calculate (Qphi, Uphi); if ‘radial’ will calculate (Qr, Ur) in final Stokes products.

field optimize_uphi: bool = False#: If true, choose the Qphi/Uphi offset angle for each wavelength by minimizing the mean of Uphi^2 over the given region. This empirically corrects residual cross-talk.

field uphi_method: Literal['aperture', 'annulus'] = 'annulus'#: If optimize_uphi=True, this determines the region type used to minimize Uphi.

field uphi_radius: Annotated[float] = 10#: First radius for Uphi optimization. For ‘aperture’ this is the radius; for ‘annulus’ this is the inner radius.

field uphi_radius2: Optional[Annotated[float]] = None#: Second radius for Uphi optimization (only used if uphi_method=’annulus’); the outer radius.

field uphi_max_angle: Annotated[float] = 10#: If optimize_uphi=True, the offset angle is searched within +/- this many degrees.

field mask_satspots: bool = False#: If true, mask satellite spots when forming Stokes images.

Configuration

Contents

Configuration#