This module includes classes to create the datasets.

class VIIRS750_download[source]

VIIRS750_download(region, tstart, tend) :: Ladsweb

Utility for downloading VIIRS 750m data to create the dataset.

class BandsFilter[source]

BandsFilter(to_keep:list)

Remove bands not in to_keep list from the dictionary.

class BandsRename[source]

BandsRename(input_names:list, output_names:list)

class MergeTiles[source]

MergeTiles(band:str)

class BandsAssertShape[source]

BandsAssertShape()

class ActiveFires[source]

ActiveFires(file)

Get active fires and interpolate to grid.

class MirCalc[source]

MirCalc(solar_zenith_angle:str, mir_radiance:str, tir_radiance:str, output_name:str='MIR')

class BaseDataset[source]

BaseDataset(name:str, paths:InOutPath, region:Region, times:DatetimeIndex=None, bands:list=None)

The BaseDataset class is a template to be used for the several types of datasets we are going to use. The methods BaseDataset.list_files, BaseDataset.find_dates and BaseDataset.open are just placeholders to be defined individually for each data source as they will depend on the filenames and file types.

BaseDataset.list_files[source]

BaseDataset.list_files(time:Timestamp)

This method should return a list of filenames corresponding to the given Timestamp.

BaseDataset.find_dates[source]

BaseDataset.find_dates()

This method should return a pd.DatetimeIndex with list of dates present in the data available in the input path.

BaseDataset.open[source]

BaseDataset.open(files:list)

This method is used to open a file or list of files for a given time period and returns a dictionary with the data ready to be passed to the processing functions.

The remaining methods of BaseDataset listed below are already defined but some may need to be redifined for particular datasets with different characteristics.

BaseDataset.match_times[source]

BaseDataset.match_times(other, on='month')

Set the times attribute to match the times of other dataset.

BaseDataset.filter_times[source]

BaseDataset.filter_times(year)

To select only a specific year. This can be usefull for testing and for adding more new years and avoid reprocessing all the dataset.

BaseDataset.save[source]

BaseDataset.save(time:Timestamp, data:dict, do_compression=True)

Saves data in a single file for a specific timestamp in .mat format.

BaseDataset.process_one[source]

BaseDataset.process_one(time:Timestamp, proc_funcs:list=[], save=True, **proc_funcs_kwargs)

This method defines a processing pipeline consisting of opening the file using the open method, applying each of the proc_funcs to the output of the previous and save the processed data using save method.

BaseDataset.process_all[source]

BaseDataset.process_all(proc_funcs=[], max_workers=1, **proc_funcs_kwargs)

process_all runs process_one in parallel using the number of workers defined by max_workers and passes the proc_funcs list to process_one method

class Viirs750Dataset[source]

Viirs750Dataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands:list=None) :: BaseDataset

Subclass of BaseDataset to process VIIRS 750-meter bands.

Example:

iop = InOutPath('../data', '../data')
R = Region('PI', [-10, 36, 5, 44], 0.01)
bands =  ['Reflectance_M5', 'Reflectance_M7', 'Reflectance_M10', 'Radiance_M12',
           'Radiance_M15', 'SolarZenithAngle', 'SatelliteZenithAngle']

merge_tiles = MergeTiles('SatelliteZenithAngle')
mir_calc = MirCalc('SolarZenithAngle', 'Radiance_M12', 'Radiance_M15')
rename = BandsRename(['Reflectance_M5', 'Reflectance_M7'], ['Red', 'NIR'])
bfilter = BandsFilter(['Red', 'NIR', 'MIR'])

data = Viirs750Dataset(iop, R, bands=bands)

100%|██████████| 1/1 [00:00<00:00, 1782.53it/s]

test_eq(str(data.times[0]), '2017-10-27 00:00:00')

out = data.process_one(time=data.times[0], 
                 proc_funcs=[merge_tiles, mir_calc, rename, bfilter], 
                 save=False)

/home/mnpinto/anaconda3/envs/fastai_dev/lib/python3.7/site-packages/ipykernel_launcher.py:70: RuntimeWarning: invalid value encountered in greater_equal
/home/mnpinto/anaconda3/envs/fastai_dev/lib/python3.7/site-packages/ipykernel_launcher.py:50: RuntimeWarning: invalid value encountered in less_equal
/home/mnpinto/anaconda3/envs/fastai_dev/lib/python3.7/site-packages/ipykernel_launcher.py:51: RuntimeWarning: invalid value encountered in less_equal
/home/mnpinto/anaconda3/envs/fastai_dev/lib/python3.7/site-packages/ipykernel_launcher.py:73: RuntimeWarning: invalid value encountered in less_equal

test_eq([k for k in out], ['MIR', 'Red', 'NIR'])
test_eq(np.nanmean(out['NIR']).round(4), 0.0474)

class Viirs750Dataset2(Viirs750Dataset):
    _use_netcdf4 = False
data2 = Viirs750Dataset2(iop, R, bands=bands)
out2 = data2.process_one(time=data.times[0], 
                 proc_funcs=[merge_tiles, mir_calc, rename, bfilter], 
                 save=False)

100%|██████████| 1/1 [00:00<00:00, 2119.41it/s]
/home/mnpinto/anaconda3/envs/fastai_dev/lib/python3.7/site-packages/ipykernel_launcher.py:53: RuntimeWarning: invalid value encountered in greater_equal
/home/mnpinto/anaconda3/envs/fastai_dev/lib/python3.7/site-packages/ipykernel_launcher.py:50: RuntimeWarning: invalid value encountered in less_equal
/home/mnpinto/anaconda3/envs/fastai_dev/lib/python3.7/site-packages/ipykernel_launcher.py:51: RuntimeWarning: invalid value encountered in less_equal
/home/mnpinto/anaconda3/envs/fastai_dev/lib/python3.7/site-packages/ipykernel_launcher.py:73: RuntimeWarning: invalid value encountered in less_equal

# Comparing open with netCDF4 and pyhdf
test_eq(np.nanmean(out['NIR']).round(7), np.nanmean(out2['NIR']).round(7))
test_eq(np.nanmean(out['Red']).round(7), np.nanmean(out2['Red']).round(7))
test_eq(np.nanmean(out['MIR']).round(7), np.nanmean(out2['MIR']).round(7))

class MCD64Dataset[source]

MCD64Dataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process MCD64A1 Collection 6 burned areas product.

class FireCCI51Dataset[source]

FireCCI51Dataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process FireCCI51 burned areas product.

class AusCoverDataset[source]

AusCoverDataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process AusCover burned areas product.

class MTBSDataset[source]

MTBSDataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process MTBS burned areas product.

class ICNFDataset[source]

ICNFDataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process ICNF burned areas product.

class Region2Tiles[source]

Region2Tiles(paths:InOutPath, input_name:str, target_name:str, regions:list=None, bands:list=None, size=128, step=100)

Crops processed dataset into 128x128 tiles to be used to train the model

Region2Tiles.open[source]

Region2Tiles.open(file, bands:list)

Open .mat file and select bands.

Region2Tiles.save[source]

Region2Tiles.save(data, file, r, c, folder, bands)

Save data to file on folder selecting only the given bands. The row and column index are included in the filename so that the large scene can be reconstructed afterwards after generating the model predictions.

Region2Tiles.crop[source]

Region2Tiles.crop(im, r, c)

crop image into a square of size sz

Region2Tiles.process_one[source]

Region2Tiles.process_one(file, bands, folder)

Create tiles for a file saving the results in folder for each crop using save method

Region2Tiles.process_all[source]

Region2Tiles.process_all(max_workers=8, include=[])

Run process_one in parallel using the number of workers given by max_workers