preprocessor.transfer package

Submodules

preprocessor.transfer.abc module

class preprocessor.transfer.abc.Downloader[source]

Bases: ABC

ABC for file downloaders. Implementing classes shall download files from a given storage location to be preprocessed.

abstract download(remote_path: str, local_path: str) → str[source]

class preprocessor.transfer.abc.Uploader[source]

Bases: ABC

abstract product_exists(remote_dir: str, expected_files: int) → Tuple[bool, List[str]][source]

abstract upload(local_path: str | List[str], remote_dir: str) → Dict[str, str][source]

preprocessor.transfer.local module

class preprocessor.transfer.local.Base(root_directory: str)[source]: Bases: object

class preprocessor.transfer.local.LocalDownloader(root_directory: str)[source]

Bases: Base, Downloader

Downloader for Local storage

download(remote_path: str, local_dir: str) → str[source]

class preprocessor.transfer.local.LocalUploader(root_directory: str)[source]

Bases: Base, Uploader

Uploader for Local storage

product_exists(remote_dir: str, expected_files: int = 2) → Tuple[bool, List[str]][source]: Returns a tuple containing: a boolean if remote directory exists and has at least expected_files_count files a list of files on the remote_dir

upload(local_path: str | List[str], remote_dir: str) → Dict[str, str][source]

preprocessor.transfer.swift module

class preprocessor.transfer.swift.Base(username=None, password=None, project_name=None, project_id=None, region_name=None, user_domain_id=None, user_domain_name=None, project_domain_id=None, project_domain_name=None, auth_url=None, auth_version=None, container=None)[source]

Bases: object

export_env_vars()[source]: exports current openstack env vars to os.environ for GDAL

get_service()[source]

reset_env_vars()[source]: nulls all the openstack env vars from os.environ

validate_container(remote_path)[source]

class preprocessor.transfer.swift.SwiftDownloader(username=None, password=None, project_name=None, project_id=None, region_name=None, user_domain_id=None, user_domain_name=None, project_domain_id=None, project_domain_name=None, auth_url=None, auth_version=None, container=None)[source]

Bases: Base, Downloader

Downloader for OpenStack swift object storages

download(remote_path: str, local_path: str) → str[source]

class preprocessor.transfer.swift.SwiftUploader(username=None, password=None, project_name=None, project_id=None, region_name=None, user_domain_id=None, user_domain_name=None, project_domain_id=None, project_domain_name=None, auth_url=None, auth_version=None, container=None)[source]

Bases: Base, Uploader

Uploader for OpenStack swift object storages

product_exists(remote_dir: str, expected_files_count: int = 2) → Tuple[bool, List[str]][source]: Returns a tuple containing: a boolean if remote directory exists and has at least expected_files_count files a list of files on the remote_dir

upload(local_path: str | List[str], remote_dir: str) → Dict[str, str][source]

preprocessor.transfer.util module

preprocessor.transfer.util.get_downloader(type_name, args, kwargs) → Downloader[source]

preprocessor.transfer.util.get_uploader(type_name, args, kwargs) → Uploader[source]

Module contents

preprocessor.transfer.get_downloader(type_name, args, kwargs) → Downloader[source]

preprocessor.transfer.get_uploader(type_name, args, kwargs) → Uploader[source]