OGIP data format support

The pyspextools module offers to import OGIP spectral files and convert them to SPEX format. There are basicly two types of OGIP files: regular type I files, which are distributed with most X-ray missions, and type II files distributed with the Chandra transmission grating (TG) spectra. Pyspextools can handle both types.

pyspextools.io.ogip: Importing OGIP files

The pyspextools OGIPRegion class contains methods to read a source spectrum, background spectrum, response file and effective area file, and save them as a SPEX region, which can be used by the other methods in the pyspextools module. Please note that the module does a direct translation of the OGIP files. No filtering of bad channels or other optimizations are performed. Bad channel filtering and optimizations can be found in pyspextools.data.

The main method of the OGIPRegion class is read_region. We recommend to use this method to read a combination of a spectrum and response matrix. This method will do the necessary checks to make sure that the final region object is consistent. The OGIP class can be imported separately if needed:

import pyspextools.io.ogip as ogip
oregion = ogip.OGIPRegion()

The oregion instance above will be an extended version of the parent region class of pyspextools and will have the same functionality.

The ‘read_region’ method can be called to read the OGIP spectra and responses and return a SPEX region object, which can be written to spo and res files.

For some missions, the provided PHA file contains counts, but also a STAT_ERR column with non-Poisson errors. It is usually a good idea to use Poisson statistics in these cases. The read_region routine and other methods have an optional force_poisson flag to make sure that the error on the counts is the square root of the number of observed counts.

A convenient script called OGIP2spex - Convert OGIP files to SPEX format is available to convert OGIP spectra and responses to SPEX format.

pyspextools.io.tg: Importing Chandra grating files

Spectra from the Chandra gratings are stored in a special format, called PHA2 format, and contain the spectra of the specific orders of the spectrum. The TGRegion class allows a user to import a Chandra grating spectrum into a SPEX region.

Similar to the OGIP files, there is also a region class for grating spectra called TGRegion:

import pyspextools.io.tg as tg
tgregion = tg.TGRegion()

The ‘read_region’ method can be called to read the grating spectra and responses and return a SPEX region object, which can be written to spo and res files.

A convenient script called TG2Spex - Convert Chandra grating data is available to convert one Chandra grating observation to SPEX format.

Notes about converting PHA to SPEX format

The methods in this class perform an almost direct copy of the PHA spectrum files into SPEX format. The conversion of source and background spectra into (background subtracted) source and background rates with errors is relatively straightforward. The areascal and backscal values from the PHA file are used to scale the spectral rates and errors as they are stored in SPEX format.

Note

The method of subtracting the background from a spectrum provides wrong results in case of Poisson distributed data and C-statistics. It would be better to use a model for the background instead. Such alternatives will be implemented from SPEX version >=3.05.00.

This method does not yet ignore bad channels. In pyspextools this is regarded as additional filtering of the data and is implemented in a different method (ref). Therefore, this method alone could give different results from the SPEX trafo program.

Notes about converting RMF to SPEX format

While the conversion of PHA format spectra to SPEX format is relatively trivial, the conversion of response matrices is more interesting. The RMF format chooses to store response groups, which describe a particular redistribution function, per model energy bin. Since there can be multiple physical effects causing their own characteristic response function, there can be multiple groups associated with a particular model energy, each describing the redistribution function for a particular physical effect.

In SPEX, the groups are not organised per model energy bin, but per physical effect, which is called a response component. This allows more freedom in the binning that is needed for the group and thus the storage space required. The optimal range and binning needed for a sharp Gaussian response feature is obviously different from a broad-band power-law redistribution function. The latter can be binned in a much coarser way than the Gaussian case.

However, the RMF files do not contain information about the physical response components. There are methods to detect similar groups in an RMF file and organise them in a component, but we did not add this method to this class. This class does a direct copy of all the groups of the RMF to one SPEX response component. If a more optimal re-arrangement of groups in components is desired, that should be done with other methods in the pyspextools module (ref).

In some rare cases (e.g. ROSAT responses), the energy grid in the response matrix contains bins with a zero width. The Rmf class has a method called fix_energy_grid, which just moves the bin boundary by a tiny amount. If this case is recognized by the check method, it will return 2 with a warning. The user can decide what to do with the warning. In ogip2spex, the bins are fixed automatically while issueing a warning at the same time.

Update (version 0.4.0): There is a new type of RMF file that contains two MATRIX extensions in the fits file that both model different components of the instrument response. Pyspextools can convert these spectra now and in the resulting SPEX format the separate MATRIX extensions are translated into SPEX response components.

The OGIPRegion class description

class pyspextools.io.ogip.OGIPRegion

The OGIPRegion class contains methods to read OGIP data into the pyspextools module and convert these to spo and res format objects.

Variables

• spec (pyspextools.io.pha.Pha) – Input PHA source spectrum.

• back (pyspextools.io.pha.Pha) – Input PHA background spectrum (optional).

• resp (pyspextools.io.rmf.Rmf) – Input RMF response matrix.

• area (pyspextools.io.arf.Arf) – Input ARF effective area (optional).

• corr (pyspextools.io.pha.Pha) – Input CORR correction file (optional).

• input_spec (bool) – Is a source spectrum read in?

• input_back (bool) – Is a background spectrum specified?

• input_resp (bool) – Is a response matrix read in?

• input_area (bool) – Is an effective area file specified?

• input_corr (bool) – Is a correction file specified?

• first_channel_zero (bool) – If the first channel is 0, then matrix data needs shift since SPEX starts at 1.

• save_grouping (bool) – Save the grouping of the spectra (Default: no).

read_region(phafile, rmffile, bkgfile=None, arffile=None, corrfile=None, grouping=False, force_poisson=False)

Add an OGIP spectrum and response to a SPEX region. The pha and rmf file names are mandatory. If needed, a background file and effective area file can be added.

Parameters

• phafile (str) – Name of the PHA file to read.

• rmffile (str) – Name of the RMF file to read.

• bkgfile (str) – Name of the background PHA file to read (optional).

• arffile (str) – Name of the ARF file to read (optional).

• corrfile (str) – Name of the Correction file to read (optional).

• grouping (bool) – Keep the grouping information?

• force_poisson (bool) – Force the calculation of Poisson errors (default: False)

add_region(spec, resp, back=None, corr=None, area=None)

Add OGIP objects to an OGIP region. This is useful when the OGIP files are already read in, but they need to be added to an OGIP region and converted to spo and res objects.

Parameters

• spec (pyspextools.io.pha.Pha) – PHA object to add.

• resp (pyspextools.io.rmf.Rmf) – RMF object to add.

• back (pyspextools.io.pha.Pha) – PHA background object to add.

• corr (pyspextools.io.pha.Pha) – PHA correction spectrum to add.

• area (pyspextools.io.arf.Arf) – ARF effective area object to add.

ogip_to_spex()

Convert the OGIP part of the OGIP region to spo and res objects.

read_source_pha(phafile, force_poisson=False)

Open a pha file containing the source spectrum.

Parameters

• phafile (str) – PHA file name of source spectrum to read.

• force_poisson (bool) – Force the calculation of Poisson errors (default: False)

read_background_pha(bkgfile, force_poisson=False)

Open a pha file containing the background spectrum (if specified).

Parameters

• bkgfile (str) – PHA file name of background spectrum to read.

• force_poisson (bool) – Force the calculation of Poisson errors (default: False)

read_rmf(rmffile)

Open rmf file containing the response matrix.

Parameters

rmffile (str) – RMF file name to read.

read_arf(arffile)

Read arf file containing the effective area.

Parameters

arffile (str) – ARF file name to read.

read_corr(corrfile)

Read correction file if specified.

Parameters

corrfile (str) – PHA correction file name to read.

correct_possible_shift(ext)

See if there is a shift in the response array. When the spectral channels start at 0 in OGIP responses, then there is a possibility that the channel indexing needs to be shifted by 1. The SPEX format first channel should always be 1. Run this function after the conversion of OGIP to SPEX had taken place and if the first channel in the OGIP spectrum is 0. The ogip_to_spex method calls this function by default.

Parameters

ext (int) – Matrix extension number (Most RMFs have only one extension (ext=0), but could be more)

check_ogip()

Check consistency of the OGIP files in this class.

show_ogip()

Shows a summary of the loaded OGIP data.

The TGRegion class description

class pyspextools.io.tg.TGRegion

The TGRegion class contains methods to read Chandra grating data into the pyspextools module and convert these to spo and res format objects.

Variables

grating (str) – Grating name.

read_region(pha2file, rmflist, arflist, grating, bkgsubtract=True)

Add a Chandra spectrum and response to a SPEX region. The pha2 file and the rmf and arf file lists are mandatory. The grating option can be either HETG, METG or LETG.

Parameters

• pha2file (str) – PHA2 file name to read.

• rmflist (list) – List of RMF response files.

• arflist (list) – List of ARF effective area files.

• grating (str) – Grating name.

• bkgsubtract (bool) – Subtract the background?

The OGIP file classes

Pha spectra

class pyspextools.io.pha.Pha

Class to read OGIP PHA files. The variable naming is made consistent with the HEASOFT HEASP module by Keith Arnaud.

Variables

• FirstChannel (int) – First valid spectral channel.

• DetChans (int) – Total number of legal channels.

• Channel (numpy.ndarray) – Spectrum channels.

• Rate (numpy.ndarray) – Spectrum spectrum count rate.

• StatError (numpy.ndarray) – Spectrum error rate (if exists).

• SysError (numpy.ndarray) – Spectrum systematic error.

• Quality (numpy.ndarray) – Quality flag.

• Grouping (numpy.ndarray) – Grouping information.

• AreaScaling (numpy.ndarray) – Areascal keyword/array.

• BackScaling (numpy.ndarray) – Backscal keyword/array.

• CorrScal (float) – Correction spectrum scaling.

• Exposure (float) – Exposure time of the spectrum.

• Poisserr (bool) – Are the errors Poissonian.

• Spectrumtype – Spectrumtype (TOTAL, NET or BKG).

• PhaType (str) – Whether the spectrum is in COUNTS or RATE.

• rmffile (str) – Associated Response matrix file.

• arffile (str) – Associated Effective area file.

• bkgfile (str) – Associated Background file.

• corfile (str) – Associated Correction spectrum file.

• Pha2Back (bool) – Is there a PHA2 background available?

• BackRate (numpy.ndarray) – PHA2 Background Rate.

• BackStatError (numpy.ndarray) – PHA2 Background Error.

• Pha2BackScal (float) – Backscale value for background.

read(filename, force_poisson=False)

Read a spectrum from a PHA file.

Parameters

• filename (str) – PHA file name to be read.

• force_poisson (bool) – Force the calculation of Poisson errors (default: False)

read_header(header)

Utility function to read the header from a SPECTRUM extension for both PHA and PHAII files.

Parameters

header (astropy.io.fits.Header) – Header of the SPECTRUM extension.

check()

Check if the object contains the minimum required data.

create_dummy(resp)

Generate dummy spectrum based on rmf channel information.

Parameters

resp (pyspextools.io.rmf.Rmf) – Input RMF response object.

disp()

Display a summary of the PHA file.

check_compatibility(pha)

Check if another PHA object is compatible with the current one in terms of number of channels.

Parameters

pha (pyspextools.io.pha.Pha) – PHA object to check compatibility for.

Pha2 grating spectra

class pyspextools.io.pha2.Pha2

Class to read PHA2 type OGIP spectra.

Variables

• NumberSpectra (int) – Number of spectra in PHAII file.

• phalist (list) – List of PHA spectra.

• tg_m (numpy.ndarray) – Array of order numbers.

• tg_part (numpy.ndarray) – Array of grating numbers.

• instrument (str) – Instrument name.

• telescope (str) – Telescope name.

• grating (str) – Grating name

• gratings (dict) – Dictionary of grating names.

read(phafile, force_poisson=True, background=False)

Read a type II pha file. Many time Gehrels errors are provided, but we prefer Poisson. Therefore, the optional ‘force_poisson’ flag is True by default. Set force_poisson to false to obtain the errors from the file. If the user wants to subtract the background, the flag ‘background’ should be set to True.

Parameters

• phafile (str) – Name of the type II PHA file.

• force_poisson (bool) – Flag to set the enforcement of Poisson errors.

• background (bool) – Subtract the background (True/False)?

combine_orders(grating)

Combine the orders for spectra from the same grating (1 = HETG, 2 = METG, 3 = LETG).

Parameters

grating (int) – Grating number to combine the orders for.

Rmf instrument response matrix

class pyspextools.io.rmf.Rmf

Class to read OGIP RMF files. The response is given in two parts: an EBOUNDS extension, containing the energy boundries of the instrument channels, and one or more MATRIX extensions, which contain components of the response matrix.

Variables

• ebounds (pyspextools.io.rmf.RmfEbounds) – Represents the EBOUNDS extension in the RMF file, which contains the channel energy scale.

• matrix (list) – List containing the matrix extensions (type pyspextools.io.rmf.RmfMatrix)

• NumberMatrixExt (int) – The number of matrix extensions.

• MatrixExt (numpy.ndarray) – Array containing the FITS extension numbers that contain a response matrix.

read(rmffile)

Method to read OGIP RMF files. The variable naming is made consistent with the HEASOFT HEASP module by Keith Arnaud.

Parameters

rmffile (str) – RMF file name to read.

write(rmffile, telescop=None, instrume=None, filterkey=None, overwrite=False)

Method to write an OGIP format RMF file.

Parameters

• rmffile (str) – RMF file name to write.

• telescop (str) – Name of the telescope to be put in the TELESCOP keyword.

• instrume (str) – Name of the instrument to be put in the INSTRUME keyword.

• filterkey (str) – Name of the filter to be put in the FILTER keyword.

• overwrite (bool) – Overwrite existing file names? (True/False)

check()

Check the RMF for internal consistency.

disp()

Display a summary of the RMF object.

check_compatibility(arf)

Check whether the input arf object is really an ARF object with data and consistent with this RMF file.

Parameters

arf (pyspextools.io.Arf) – Input arf object to check.

fix_energy_grid()

In some exceptional cases, like with ROSAT response matrices, the matrix contains energy bins with zero width, which is not physical. This method changes them to a small finite width to allow them to be processed with ogip2spex and trafo.

The Rmf consists of two parts. The first is an EBOUNDS extension, which is defined in the class below:

class pyspextools.io.rmf.RmfEbounds

Class to read the EBOUNDS extension from an RMF or RSP file.

Variables

• FirstChannel (int) – First channel number.

• Channel (numpy.ndarray) – Channel numbers.

• ChannelLowEnergy (numpy.ndarray) – Start energy of channel.

• ChannelHighEnergy (numpy.ndarray) – End energy of channel.

• NumberChannels (int) – Number of data channels.

• EnergyUnits (string) – Unit of the energy scale

And one or more response matrix extensions. The matrices are of type RmfMatrix and are kept in the list called matrix.

class pyspextools.io.rmf.RmfMatrix

Class to read a MATRIX extension from an OGIP RMF or RSP file.

Variables

• NumberGroups (numpy.ndarray) – Number of response groups.

• FirstGroup (numpy.ndarray) – First response group for this energy bin.

• FirstChannelGroup (numpy.ndarray) – First channel number in this group.

• NumberChannelsGroup (numpy.ndarray) – Number of channels in this group.

• FirstElement (numpy.ndarray) – First response element for this group.

• LowEnergy (numpy.ndarray) – Start energy of bin.

• HighEnergy (numpy.ndarray) – End energy of bin.

• Matrix (numpy.ndarray) – Response matrix elements.

• NumberEnergyBins (int) – Number of energy bins.

• NumberTotalGroups (int) – Total number of groups.

• NumberTotalElements (int) – Total number of response elements.

• AreaScaling (float) – Value of EFFAREA keyword.

• ResponseThreshold (float) – Minimum value in response.

• EnergyUnits (str) – Units of the energy scale.

• RMFUnits (str) – Units for RMF values.

• AreaIncluded (bool) – Is the effective area included in the response?

• Order (int) – Order of the matrix.

Arf mirror effective area

class pyspextools.io.arf.Arf

Class to read OGIP ARF files. The variable naming is made consistent with the HEASOFT HEASP module by Keith Arnaud.

Variables

• LowEnergy (numpy.ndarray) – Low Energy boundary of bin

• HighEnergy (numpy.ndarray) – High Energy boundary of bin

• EffArea (numpy.ndarray) – Effective area of bin

• EnergyUnits (str) – Energy units

• ARFUnits (str) – Unit of effective area

• Order (int) – Grating order (for grating arrays, else 0)

• Grating (int) – Grating instrument (if available, 1 = HEG, 2 = MEG, 3 = LEG)

read(arffile)

Read the effective area from an OGIP ARF file.

Parameters

arffile (str) – Effective area file name (FITS/OGIP format)

write(arffile, telescop=None, instrume=None, filter=None, overwrite=False)

Write an OGIP compatible ARF file (Non-grating format).

Parameters

• arffile (str) – Effective area file name to write (FITS/OGIP format)

• telescop (str) – Telescope name (optional)

• instrume (str) – Instrument name (optional)

• filter (str) – Filter setting (optional)

• overwrite (bool) – Overwrite existing file? (True/False)

check()

Check if the basic information is read in.

disp()

Display a summary of the ARF object.