Cloud Storage Input/Output¶
- cloudreg.scripts.download_raw_data.download_raw_data(in_bucket_path, channel, outdir)[source]¶
Download COLM raw data from S3 to local storage
- Parameters:
in_bucket_path (str) – Name of S3 bucket where raw dadta live at
channel (int) – Channel number to process
outdir (str) – Local path to store raw data
- cloudreg.scripts.download_raw_data.download_tile(s3, raw_tile_bucket, raw_tile_path, outdir, bias=None)[source]¶
Download single raw data image file from S3 to local directory
- Parameters:
s3 (S3.Resource) – A Boto3 S3 resource
raw_tile_bucket (str) – Name of bucket with raw data
raw_tile_path (str) – Path to raw data file in S3 bucket
outdir (str) – Local path to store raw data
bias (np.ndarray, optional) – Bias correction multiplied by image before saving. Must be same size as image Defaults to None.
- cloudreg.scripts.download_raw_data.download_tiles(tiles, raw_tile_bucket, outdir)[source]¶
Download a chunk of tiles from S3 to local storage
- Parameters:
tiles (list of str) – S3 paths to raw data files to download
raw_tile_bucket (str) – Name of bucket where raw data live
outdir (str) – Local path to store raw data at
- cloudreg.scripts.download_raw_data.get_all_s3_objects(s3, **base_kwargs)[source]¶
Get all s3 objects with base_kwargs
- Parameters:
s3 (boto3.S3.client) – an active S3 Client.
- Yields:
dict – Response object with keys to objects if there are any.
- cloudreg.scripts.download_raw_data.get_list_of_files_to_process(in_bucket_name, prefix, channel)[source]¶
Get paths of all raw data files for a given channel.
- Parameters:
in_bucket_name (str) – S3 bucket in which raw data live
prefix (str) – Prefix for the S3 path at which raw data live
channel (int) – Channel number to process
- Returns:
List of S3 paths for all raw data files
- Return type:
list of str
- cloudreg.scripts.download_raw_data.get_out_path(in_path, outdir)[source]¶
Get output path for given tile, maintaining folder structure for Terastitcher
- Parameters:
in_path (str) – S3 key to raw tile
outdir (str) – Path to local directory to store raw data
- Returns:
Path to store raw tile at.
- Return type:
str
- cloudreg.scripts.download_data.download_data(s3_path, outfile, desired_resolution, resample_isotropic=False, return_size=False)[source]¶
Download whole precomputed volume from S3 at desired resolution and optionally resample data to be isotropic
- Parameters:
s3_path (str) – S3 path to precomputed volume
outfile (str) – Path to output file
desired_resolution (int) – Lowest resolution (in nanometers) at which to download data if desired_resolution isnt available.
resample_isotropic (bool, optional) – If true, resample data to be isotropic at desired_resolution.
- Returns:
Resoluton of downloaded data in microns
- Return type:
resolution
- cloudreg.scripts.download_data.get_mip_at_res(vol, resolution)[source]¶
Find the mip that is at least a given resolution
- Parameters:
vol (cloudvolume.CloudVoluem) – CloudVolume object for desired precomputed volume
resolution (int) – Desired resolution in nanometers
- Returns:
mip and resolution at that mip
- Return type:
tuple