class VIIRS375_download[source]

VIIRS375_download(region, tstart, tend, bands=None) :: Ladsweb

Utility for downloading VIIRS 375m data to create the dataset.

class VIIRS750_download[source]

VIIRS750_download(region, tstart, tend, bands=None) :: Ladsweb

Utility for downloading VIIRS 750m data to create the dataset.

class BandsFilter[source]

BandsFilter(to_keep:list)

Remove bands not in to_keep list from the dictionary.

class BandsRename[source]

BandsRename(input_names:list, output_names:list)

class MergeTiles[source]

MergeTiles(band:str, ignore:list=[])

class BandsAssertShape[source]

BandsAssertShape()

class ActiveFiresLog[source]

ActiveFiresLog(file)

Get active fires, interpolate to grid and apply log1p.

class InterpolateAngles[source]

InterpolateAngles(R_base:Region, R_interp:Region, bands:list)

Interpolated Angles to working resolution.

class MirCalc[source]

MirCalc(solar_zenith_angle:str, mir_radiance:str, tir_radiance:str, output_name:str='MIR')

class BaseDataset[source]

BaseDataset(name:str, paths:InOutPath, region:Region, times:DatetimeIndex=None, bands:list=None)

The BaseDataset class is a template to be used for the several types of datasets we are going to use. The methods BaseDataset.list_files, BaseDataset.find_dates and BaseDataset.open are just placeholders to be defined individually for each data source as they will depend on the filenames and file types.

BaseDataset.list_files[source]

BaseDataset.list_files(time:Timestamp)

This method should return a list of filenames corresponding to the given Timestamp.

BaseDataset.find_dates[source]

BaseDataset.find_dates()

This method should return a pd.DatetimeIndex with list of dates present in the data available in the input path.

BaseDataset.open[source]

BaseDataset.open(files:list, crop=None)

This method is used to open a file or list of files for a given time period and returns a dictionary with the data ready to be passed to the processing functions.

The remaining methods of BaseDataset listed below are already defined but some may need to be redifined for particular datasets with different characteristics.

BaseDataset.match_times[source]

BaseDataset.match_times(other, on='month')

Set the times attribute to match the times of other dataset.

BaseDataset.filter_times[source]

BaseDataset.filter_times(year)

To select only a specific year. This can be usefull for testing and for adding more new years and avoid reprocessing all the dataset.

BaseDataset.save[source]

BaseDataset.save(time:Timestamp, data:dict, crop=None)

Saves data in a single file for a specific timestamp in netcdf4 format.

BaseDataset.process_one[source]

BaseDataset.process_one(time:Timestamp, proc_funcs:list=[], save=True, max_size=None, **proc_funcs_kwargs)

This method defines a processing pipeline consisting of opening the file using the open method, applying each of the proc_funcs to the output of the previous and save the processed data using save method.

BaseDataset.process_all[source]

BaseDataset.process_all(proc_funcs=[], max_workers=1, max_size=None, **proc_funcs_kwargs)

process_all runs process_one in parallel using the number of workers defined by max_workers and passes the proc_funcs list to process_one method

class ViirsDataset[source]

ViirsDataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands:list=None) :: BaseDataset

Subclass of BaseDataset to process VIIRS 750-meter bands.

class Viirs375Dataset[source]

Viirs375Dataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands:list=None) :: ViirsDataset

Subclass of BaseDataset to process VIIRS 750-meter bands.

class Viirs750Dataset[source]

Viirs750Dataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands:list=None) :: ViirsDataset

Subclass of BaseDataset to process VIIRS 750-meter bands.

Example:

iop = InOutPath('../data', '../data')
R = Region('PI', [-10, 36, 5, 44], 0.01)
bands =  ['Reflectance_M5', 'Reflectance_M7', 'Reflectance_M10', 'Radiance_M12',
           'Radiance_M15', 'SolarZenithAngle', 'SatelliteZenithAngle']

merge_tiles = MergeTiles('SatelliteZenithAngle', ignore=['R'])
mir_calc = MirCalc('SolarZenithAngle', 'Radiance_M12', 'Radiance_M15')
rename = BandsRename(['Reflectance_M5', 'Reflectance_M7'], ['Red', 'NIR'])
bfilter = BandsFilter(['Red', 'NIR', 'MIR'])

data = Viirs750Dataset(iop, R, bands=bands)

test_eq(data.name, 'VIIRS750')

100%|██████████| 1/1 [00:00<00:00, 1268.69it/s]

data.times

DatetimeIndex(['2017-10-27'], dtype='datetime64[ns]', freq=None)

test_eq(str(data.times[0]), '2017-10-27 00:00:00')

out = data.process_one(time=data.times[0], max_size=2000, 
                 proc_funcs=[merge_tiles, mir_calc, rename, bfilter], 
                 save=False)

test_eq([k for k in out], ['MIR', 'Red', 'NIR'])
test_eq(np.nanmean(out['NIR']).astype(np.float64).round(4), 0.0474)

class Viirs750Dataset2(Viirs750Dataset):
    _use_netcdf4 = True
data2 = Viirs750Dataset2(iop, R, bands=bands)
out2 = data2.process_one(time=data.times[0], max_size=2000,
                 proc_funcs=[merge_tiles, mir_calc, rename, bfilter], 
                 save=False)

100%|██████████| 1/1 [00:00<00:00, 2516.08it/s]
<ipython-input-22-027d8babb8c7>:115: UserWarning: Failed to open file with netCDF4 library, trying pyhdf instead. To use pyhdf by default set _use_netcdf4 to False.
  warn('Failed to open file with netCDF4 library, trying pyhdf instead. To use pyhdf by default set _use_netcdf4 to False.')

test_eq(np.nanmean(out['NIR']).round(7), np.nanmean(out2['NIR']).round(7))
test_eq(np.nanmean(out['Red']).round(7), np.nanmean(out2['Red']).round(7))
test_eq(np.nanmean(out['MIR']).round(7), np.nanmean(out2['MIR']).round(7))

class MCD64Dataset[source]

MCD64Dataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process MCD64A1 Collection 6 burned areas product.

class FireCCI51Dataset[source]

FireCCI51Dataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process FireCCI51 burned areas product.

class AusCoverDataset[source]

AusCoverDataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process AusCover burned areas product.

class MTBSDataset[source]

MTBSDataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process MTBS burned areas product.

class ICNFDataset[source]

ICNFDataset(paths:InOutPath, region:Region, times:DatetimeIndex=None, bands=['bafrac']) :: BaseDataset

Subclass of BaseDataset to process ICNF burned areas product.

class Region2Tiles[source]

Region2Tiles(paths:InOutPath, input_name:str, target_name:str, regions:list=None, bands:list=None, size=128, step=100)

Crops processed dataset into 128x128 tiles to be used to train the model

Region2Tiles.open[source]

Region2Tiles.open(file, bands:list)

Open .mat file and select bands.

Region2Tiles.save[source]

Region2Tiles.save(data, file, r, c, folder, bands)

Save data to file on folder selecting only the given bands. The row and column index are included in the filename so that the large scene can be reconstructed afterwards after generating the model predictions.

Region2Tiles.crop[source]

Region2Tiles.crop(im, r, c)

crop image into a square of size sz

Region2Tiles.process_one[source]

Region2Tiles.process_one(file, bands, folder)

Create tiles for a file saving the results in folder for each crop using save method

Region2Tiles.process_all[source]

Region2Tiles.process_all(max_workers=8, include=[])

Run process_one in parallel using the number of workers given by max_workers