Burst Fitting example using BurstFit class¶

Using a multiple gaussian spectra function¶

In [1]:

from burstfit.fit import BurstFit
from burstfit.data import BurstData
from burstfit.model import Model, SgramModel
from burstfit.utils.plotter import plot_me
from burstfit.utils.functions import gauss_norm2, pulse_fn_vec, sgram_fn_vec, gauss_norm
from burstfit.io import BurstIO
import logging

In [2]:

logging_format = "%(asctime)s - %(funcName)s -%(name)s - %(levelname)s - %(message)s"
logging.basicConfig(
    level=logging.INFO,
    format=logging_format,
)

Loading the data¶

In [3]:

import tempfile
from urllib.request import urlretrieve

In [4]:

temp_dir = tempfile.TemporaryDirectory()
download_path = str(temp_dir.name) + "/FRB180417.fil"
url = "https://zenodo.org/record/3905426/files/FRB180417.fil"
urlretrieve(
    url, download_path,
)
fil_file = download_path

Get candidate cutout and preprocess¶

We will use BurstData class for this. It will crop the relevant time window. Dedisperse the data. Normalise the data to zero mean and unit standard deviation. Apply RFI masks (if any) and RFI filtering (if activated).

In [5]:

bd = BurstData(
    fp=fil_file,
    dm=475.28400,
    tcand=2.0288800,
    width=2,
    snr=16.8128,
    min_samp=256,
)

In [6]:

bd.prepare_data()

2021-03-18 19:49:23,922 - prepare_data -burstfit.data - INFO - Preparing data for burst fitting.
2021-03-18 19:49:23,929 - crop_dedispersed_data -burstfit.data - INFO - Cropping data with time_window: 0.2s.
2021-03-18 19:49:23,930 - normalise_data -burstfit.data - INFO - Normalising data using off pulse mean and std.
2021-03-18 19:49:23,934 - normalise_data -burstfit.data - INFO - Off pulse mean and std are: (127.98158986433457, 18.365628141947383)

Out[6]:

Using <class 'str'>:
/tmp/tmpz0fetxnj/FRB180417.fil

This is how the data looks

In [7]:

plot_me(bd.sgram)

Fitting using BurstFit¶

In BurstFit, the fitting procedure for each component is as follows:

Fit the profile using curve_fit and a profile model
Find the spectra using profile fit parameters
Fit the spectra using curve_fit and spectra model
Use the profile and spectra fit parameters as initial guess and fit the 2D spectrogram using curve_fit and spectrogram model

Therefore, BurstFit requires three functions: profile function, spectra function and spectrogram function. Use the Model class to make profile and spectra models. You can optionally give names of the input parameters. We then create a spectrogram model using SgramModel class, with the above two models and a spectrogram function (sgram_fn_vec).

We will try with a single gaussian spectra function first, eventhough there are two components in the spectra.

In [8]:

pnames = ['S', 'mu_t', 'sigma_t', 'tau']
pulseModel = Model(pulse_fn_vec, param_names=pnames)

snames = ['mu_f', 'sigma_f']
spectraModel = Model(gauss_norm, param_names=snames)

In [9]:

sgramModel = SgramModel(pulseModel, spectraModel, sgram_fn_vec, 
                        mask=bd.mask, clip_fac=bd.clip_fac)

Provide basic candidate information to BurstFit: sgram, model, DM, width, etc

In [10]:

bf = BurstFit(
    sgram_model=sgramModel,
    sgram=bd.sgram,
    width=bd.width,
    dm=bd.dm,
    foff=bd.foff,
    fch1=bd.fch1,
    tsamp=bd.tsamp,
    clip_fac=bd.clip_fac,
    mask=bd.mask)

Fitting the data can be done by fitall()

In [11]:

bf.fitall(plot=True)

2021-03-18 19:49:24,531 - run_tests -burstfit.fit - INFO - Running statistical tests on the residual.
2021-03-18 19:49:24,533 - run_tests -burstfit.fit - INFO - Running off pulse - off pulse test
2021-03-18 19:49:24,540 - tests -root - INFO - P values: T-test (0.65766), Kruskal (0.71932), KS (0.59128), F-test (0.29584)
2021-03-18 19:49:24,541 - run_tests -burstfit.fit - INFO - Running on pulse - off pulse (L) test
2021-03-18 19:49:24,548 - tests -root - INFO - P values: T-test (0.00000), Kruskal (0.00000), KS (0.00000), F-test (0.00053)
2021-03-18 19:49:24,549 - run_tests -burstfit.fit - INFO - Running on pulse - off pulse (R) test
2021-03-18 19:49:24,556 - tests -root - INFO - P values: T-test (0.00000), Kruskal (0.00000), KS (0.00000), F-test (0.00309)
2021-03-18 19:49:24,557 - fitcycle -burstfit.fit - INFO - Fitting component 1.
2021-03-18 19:49:24,558 - initial_profilefit -burstfit.fit - INFO - Running initial profile fit for component: 1
2021-03-18 19:49:24,603 - initial_profilefit -burstfit.fit - INFO - Converged parameters (profile fit) are:
2021-03-18 19:49:24,604 - initial_profilefit -burstfit.fit - INFO - S: 512.8418742176256 +- 46.638907434560494
2021-03-18 19:49:24,605 - initial_profilefit -burstfit.fit - INFO - mu_t: 78.73232431022271 +- 0.08341666472383193
2021-03-18 19:49:24,606 - initial_profilefit -burstfit.fit - INFO - sigma_t: 0.7947937226184538 +- 0.08461197577615237
2021-03-18 19:49:24,606 - initial_profilefit -burstfit.fit - INFO - tau: 0.07023980173530475 +- 0.0

2021-03-18 19:49:24,945 - make_spectra -burstfit.fit - INFO - Making spectra using profile fit parameters.
2021-03-18 19:49:24,947 - initial_spectrafit -burstfit.fit - INFO - Running spectra profile fit for component: 1
2021-03-18 19:49:24,959 - initial_spectrafit -burstfit.fit - INFO - Converged parameters (spectra fit) are:
2021-03-18 19:49:24,960 - initial_spectrafit -burstfit.fit - INFO - mu_f: 299.8622159664497 +- 15.64622961586148
2021-03-18 19:49:24,961 - initial_spectrafit -burstfit.fit - INFO - sigma_f: 68.27149144964595 +- 10.524321434257567

2021-03-18 19:49:25,226 - sgram_fit -burstfit.fit - INFO - Running sgram profile fit for component: 1
2021-03-18 19:49:25,227 - sgram_fit -burstfit.fit - INFO - initial estimate for parameters: [299.8622159664497, 68.27149144964595, 512.8418742176256, 78.73232431022271, 0.7947937226184538, 0.07023980173530475, 475.284]
2021-03-18 19:49:25,559 - sgram_fit -burstfit.fit - INFO - Converged parameters are:
2021-03-18 19:49:25,560 - sgram_fit -burstfit.fit - INFO - mu_f: 1534.7218167022465 +- 15581.01205834671
2021-03-18 19:49:25,561 - sgram_fit -burstfit.fit - INFO - sigma_f: 877.130695501071 +- 5043.481360702697
2021-03-18 19:49:25,561 - sgram_fit -burstfit.fit - INFO - S: 11238.494797163012 +- 218108.20711029993
2021-03-18 19:49:25,562 - sgram_fit -burstfit.fit - INFO - mu_t: 79.07907758603417 +- 0.06085757421707285
2021-03-18 19:49:25,562 - sgram_fit -burstfit.fit - INFO - sigma_t: 0.7788034353914925 +- 0.0031230343762619446
2021-03-18 19:49:25,563 - sgram_fit -burstfit.fit - INFO - tau: 0.06843028885537503 +- 0.00039534528397907074
2021-03-18 19:49:25,564 - sgram_fit -burstfit.fit - INFO - DM: 474.43859616615305 +- 0.008781363329783682

2021-03-18 19:49:26,245 - model -burstfit.fit - INFO - Making model.
2021-03-18 19:49:26,246 - model -burstfit.fit - INFO - Found 1 components.
2021-03-18 19:49:26,250 - get_off_pulse_region -burstfit.fit - INFO - mu_t and sigma_t found in params. Using those to estimate off pulse region.
2021-03-18 19:49:26,251 - get_off_pulse_region -burstfit.fit - INFO - Using sgram fit parameters.
2021-03-18 19:49:26,253 - model -burstfit.fit - INFO - Making model.
2021-03-18 19:49:26,253 - model -burstfit.fit - INFO - Found 1 components.
2021-03-18 19:49:26,258 - calc_redchisq -burstfit.fit - INFO - Reduced chi-square value of fit is: 1.0048251789841864
2021-03-18 19:49:26,259 - run_tests -burstfit.fit - INFO - Running statistical tests on the residual.
2021-03-18 19:49:26,260 - run_tests -burstfit.fit - INFO - Running off pulse - off pulse test
2021-03-18 19:49:26,265 - tests -root - INFO - P values: T-test (0.65766), Kruskal (0.71932), KS (0.59128), F-test (0.29584)
2021-03-18 19:49:26,265 - run_tests -burstfit.fit - INFO - Running on pulse - off pulse (L) test
2021-03-18 19:49:26,270 - tests -root - INFO - P values: T-test (0.53052), Kruskal (0.56784), KS (0.46749), F-test (0.03168)
2021-03-18 19:49:26,270 - run_tests -burstfit.fit - INFO - On pulse residual is similar to left off pulse region.
2021-03-18 19:49:26,271 - run_tests -burstfit.fit - INFO - Running on pulse - off pulse (R) test
2021-03-18 19:49:26,276 - tests -root - INFO - P values: T-test (0.29156), Kruskal (0.32086), KS (0.04698), F-test (0.09333)
2021-03-18 19:49:26,276 - run_tests -burstfit.fit - INFO - On pulse residual is similar to right off pulse region.
2021-03-18 19:49:26,277 - fitall -burstfit.fit - INFO - On pulse residual looks like noise. Terminating individual component fitting.
2021-03-18 19:49:26,278 - fitall -burstfit.fit - INFO - Final number of components = 1. Terminating fitting.
2021-03-18 19:49:26,279 - get_off_pulse_region -burstfit.fit - INFO - mu_t and sigma_t found in params. Using those to estimate off pulse region.
2021-03-18 19:49:26,279 - get_off_pulse_region -burstfit.fit - INFO - Using sgram all-component-fit parameters.

Changing spectra function¶

There clearly are two components in the spectra. The fit results donot look very good, and the signal is still present in the residual. Let's modify our spectra model and use two gaussians to model this. We can use gauss_norm2 function for that.

In [12]:

pnames = ['S', 'mu_t', 'sigma_t', 'tau']
pulseModel = Model(pulse_fn_vec, param_names=pnames)

snames = ['mu_f1', 'sigma_f1', 'mu_f2', 'sigma_f2', 'amp']
spectraModel2 = Model(gauss_norm2, param_names=snames)

I am also giving it some spectra bounds so it get easier for curve_fit to identify region of interest for the two components.

In [13]:

spectra_bounds = ([0, 0, 200, 0, 0], [100, 50, 300, 50, 1])
sgramModel2 = SgramModel(pulseModel, spectraModel2, sgram_fn_vec, 
                        mask=bd.mask, clip_fac=bd.clip_fac)

In [14]:

bf2 = BurstFit(
    sgram_model=sgramModel2,
    sgram=bd.sgram,
    width=bd.width,
    dm=bd.dm,
    foff=bd.foff,
    fch1=bd.fch1,
    tsamp=bd.tsamp,
    clip_fac=bd.clip_fac,
    mask=bd.mask)

We can similarly give profile bounds if needed. For this one we don't need to use that.

In [15]:

bf2.fitall(plot=True, spectra_bounds=spectra_bounds, profile_bounds=[])

2021-03-18 19:49:26,475 - run_tests -burstfit.fit - INFO - Running statistical tests on the residual.
2021-03-18 19:49:26,476 - run_tests -burstfit.fit - INFO - Running off pulse - off pulse test
2021-03-18 19:49:26,481 - tests -root - INFO - P values: T-test (0.65766), Kruskal (0.71932), KS (0.59128), F-test (0.29584)
2021-03-18 19:49:26,482 - run_tests -burstfit.fit - INFO - Running on pulse - off pulse (L) test
2021-03-18 19:49:26,486 - tests -root - INFO - P values: T-test (0.00000), Kruskal (0.00000), KS (0.00000), F-test (0.00053)
2021-03-18 19:49:26,487 - run_tests -burstfit.fit - INFO - Running on pulse - off pulse (R) test
2021-03-18 19:49:26,491 - tests -root - INFO - P values: T-test (0.00000), Kruskal (0.00000), KS (0.00000), F-test (0.00309)
2021-03-18 19:49:26,492 - fitall -burstfit.fit - WARNING - Input spectra bounds detected. Using them for component 1
2021-03-18 19:49:26,493 - fitcycle -burstfit.fit - INFO - Fitting component 1.
2021-03-18 19:49:26,494 - initial_profilefit -burstfit.fit - INFO - Running initial profile fit for component: 1
2021-03-18 19:49:26,519 - initial_profilefit -burstfit.fit - INFO - Converged parameters (profile fit) are:
2021-03-18 19:49:26,520 - initial_profilefit -burstfit.fit - INFO - S: 512.8418742176256 +- 46.638907434560494
2021-03-18 19:49:26,520 - initial_profilefit -burstfit.fit - INFO - mu_t: 78.73232431022271 +- 0.08341666472383193
2021-03-18 19:49:26,521 - initial_profilefit -burstfit.fit - INFO - sigma_t: 0.7947937226184538 +- 0.08461197577615237
2021-03-18 19:49:26,522 - initial_profilefit -burstfit.fit - INFO - tau: 0.07023980173530475 +- 0.0

2021-03-18 19:49:26,825 - make_spectra -burstfit.fit - INFO - Making spectra using profile fit parameters.
2021-03-18 19:49:26,827 - initial_spectrafit -burstfit.fit - INFO - Running spectra profile fit for component: 1
2021-03-18 19:49:26,842 - initial_spectrafit -burstfit.fit - INFO - Converged parameters (spectra fit) are:
2021-03-18 19:49:26,843 - initial_spectrafit -burstfit.fit - INFO - mu_f1: 87.72708851306162 +- 14.657344144023687
2021-03-18 19:49:26,844 - initial_spectrafit -burstfit.fit - INFO - sigma_f1: 40.38070079339924 +- 14.56794385561647
2021-03-18 19:49:26,844 - initial_spectrafit -burstfit.fit - INFO - mu_f2: 284.84651913292134 +- 9.334720624356743
2021-03-18 19:49:26,845 - initial_spectrafit -burstfit.fit - INFO - sigma_f2: 41.23788107637332 +- 9.291859540498173
2021-03-18 19:49:26,846 - initial_spectrafit -burstfit.fit - INFO - amp: 0.3497407333338407 +- 0.09115810205812795

2021-03-18 19:49:27,232 - sgram_fit -burstfit.fit - INFO - Running sgram profile fit for component: 1
2021-03-18 19:49:27,233 - sgram_fit -burstfit.fit - INFO - initial estimate for parameters: [87.72708851306162, 40.38070079339924, 284.84651913292134, 41.23788107637332, 0.3497407333338407, 512.8418742176256, 78.73232431022271, 0.7947937226184538, 0.07023980173530475, 475.284]
2021-03-18 19:49:27,585 - sgram_fit -burstfit.fit - INFO - Converged parameters are:
2021-03-18 19:49:27,586 - sgram_fit -burstfit.fit - INFO - mu_f1: 74.6600995552445 +- 3.148966015198683
2021-03-18 19:49:27,587 - sgram_fit -burstfit.fit - INFO - sigma_f1: 29.08997449280042 +- 3.1735652932670435
2021-03-18 19:49:27,588 - sgram_fit -burstfit.fit - INFO - mu_f2: 282.0987558671014 +- 6.1593569158810695
2021-03-18 19:49:27,588 - sgram_fit -burstfit.fit - INFO - sigma_f2: 48.04757054335587 +- 6.826533560258564
2021-03-18 19:49:27,589 - sgram_fit -burstfit.fit - INFO - amp: 0.4123476364901036 +- 0.036921986314662335
2021-03-18 19:49:27,589 - sgram_fit -burstfit.fit - INFO - S: 555.8960256802608 +- 42.5662741419663
2021-03-18 19:49:27,590 - sgram_fit -burstfit.fit - INFO - mu_t: 79.02660306937426 +- 0.05278628950931549
2021-03-18 19:49:27,591 - sgram_fit -burstfit.fit - INFO - sigma_t: 0.7844582964946152 +- 0.006745505400432026
2021-03-18 19:49:27,591 - sgram_fit -burstfit.fit - INFO - tau: 0.07026178702689054 +- 0.0006220715236890621
2021-03-18 19:49:27,592 - sgram_fit -burstfit.fit - INFO - DM: 474.5754998470141 +- 0.12095473392329054

2021-03-18 19:49:28,229 - model -burstfit.fit - INFO - Making model.
2021-03-18 19:49:28,230 - model -burstfit.fit - INFO - Found 1 components.
2021-03-18 19:49:28,234 - get_off_pulse_region -burstfit.fit - INFO - mu_t and sigma_t found in params. Using those to estimate off pulse region.
2021-03-18 19:49:28,235 - get_off_pulse_region -burstfit.fit - INFO - Using sgram fit parameters.
2021-03-18 19:49:28,237 - model -burstfit.fit - INFO - Making model.
2021-03-18 19:49:28,237 - model -burstfit.fit - INFO - Found 1 components.
2021-03-18 19:49:28,242 - calc_redchisq -burstfit.fit - INFO - Reduced chi-square value of fit is: 1.003115392773371
2021-03-18 19:49:28,243 - run_tests -burstfit.fit - INFO - Running statistical tests on the residual.
2021-03-18 19:49:28,244 - run_tests -burstfit.fit - INFO - Running off pulse - off pulse test
2021-03-18 19:49:28,249 - tests -root - INFO - P values: T-test (0.65766), Kruskal (0.71932), KS (0.59128), F-test (0.29584)
2021-03-18 19:49:28,249 - run_tests -burstfit.fit - INFO - Running on pulse - off pulse (L) test
2021-03-18 19:49:28,254 - tests -root - INFO - P values: T-test (0.43787), Kruskal (0.50214), KS (0.35819), F-test (0.21593)
2021-03-18 19:49:28,254 - run_tests -burstfit.fit - INFO - On pulse residual is similar to left off pulse region.
2021-03-18 19:49:28,255 - run_tests -burstfit.fit - INFO - Running on pulse - off pulse (R) test
2021-03-18 19:49:28,260 - tests -root - INFO - P values: T-test (0.22684), Kruskal (0.27991), KS (0.10434), F-test (0.40143)
2021-03-18 19:49:28,260 - run_tests -burstfit.fit - INFO - On pulse residual is similar to right off pulse region.
2021-03-18 19:49:28,261 - fitall -burstfit.fit - INFO - On pulse residual looks like noise. Terminating individual component fitting.
2021-03-18 19:49:28,261 - fitall -burstfit.fit - INFO - Final number of components = 1. Terminating fitting.
2021-03-18 19:49:28,263 - get_off_pulse_region -burstfit.fit - INFO - mu_t and sigma_t found in params. Using those to estimate off pulse region.
2021-03-18 19:49:28,263 - get_off_pulse_region -burstfit.fit - INFO - Using sgram all-component-fit parameters.

here are the parameters:

In [16]:

bf2.sgram_params['all']

Out[16]:

{1: {'popt': [74.6600995552445,
   29.08997449280042,
   282.0987558671014,
   48.04757054335587,
   0.4123476364901036,
   555.8960256802608,
   79.02660306937426,
   0.7844582964946152,
   0.07026178702689054,
   474.5754998470141],
  'perr': array([3.14896602e+00, 3.17356529e+00, 6.15935692e+00, 6.82653356e+00,
         3.69219863e-02, 4.25662741e+01, 5.27862895e-02, 6.74550540e-03,
         6.22071524e-04, 1.20954734e-01])}}

Voila! Both the components are fitted nicely and the residual sgram looks just like noise. The tests show that on-pulse residual looked like off-pulse region. So the fitting was terminated.