preprocessor package

Subpackages

• preprocessor.steps package
  • Submodules
  • preprocessor.steps.browse_report module
    • apply_browse_report_georeference()
    • browse_georeference()
    • frange()
    • generate_gsc()
  • preprocessor.steps.calc module
    • calc_formula()
    • calc_step()
    • evaluate_formula()
    • set_default_no_data()
  • preprocessor.steps.georeference module
    • GetOgrEnvelope()
    • corner_georef()
    • correct_geo_transform()
    • fix_geotrans()
    • gcp_georef()
    • gcps_from_borders()
    • georeference_step()
    • gsc_footprint_georef()
    • no_op()
    • rpc_georef()
    • set_gcps_from_gsc_footprint()
    • world_georef()
  • preprocessor.steps.output module
    • output_step()
  • preprocessor.steps.stack module
    • create_groups()
    • handle_group_sort()
    • intermediate_warp()
    • remove_rotated_geotransform()
    • stack_bands_step()
  • preprocessor.steps.subdataset module
    • extract_subdataset_step()
    • extract_subdatasets()
  • Module contents
    • calc_step()
    • extract_subdataset_step()
    • georeference_step()
    • output_step()
    • stack_bands_step()
• preprocessor.transfer package
  • Submodules
  • preprocessor.transfer.abc module
    • Downloader
      • Downloader.download()
    • Uploader
      • Uploader.product_exists()
      • Uploader.upload()
  • preprocessor.transfer.local module
    • Base
    • LocalDownloader
      • LocalDownloader.download()
    • LocalUploader
      • LocalUploader.product_exists()
      • LocalUploader.upload()
  • preprocessor.transfer.swift module
    • Base
      • Base.export_env_vars()
      • Base.get_service()
      • Base.reset_env_vars()
      • Base.validate_container()
    • SwiftDownloader
      • SwiftDownloader.download()
    • SwiftUploader
      • SwiftUploader.product_exists()
      • SwiftUploader.upload()
  • preprocessor.transfer.util module
    • get_downloader()
    • get_uploader()
  • Module contents
    • get_downloader()
    • get_uploader()

Submodules

preprocessor.archive module

preprocessor.archive.filter_filenames(filenames: List[str], glob: str, case: bool = False) → List[str]

preprocessor.archive.is_tarfile(archive_file) → bool

Helper to detect whether a path or a file object is referencing a valid TAR file.

preprocessor.archive.open_tarfile(archive_file: str | bytes) → TarFile

Open a TAR file from either a path or a file object.

preprocessor.archive.unpack_files(archive_path, target_dir: str, glob=None, case=None, filenames=None, recursive=False) → List[str]

Unpacks the contents of the specified ZIP or TAR archive to the given target directory. Optionally, only a given list of filenames will be extracted. When a glob is passed, all filenames (either given or from the archive) will be filtered and only the matching files will be extracted.

preprocessor.cli module

preprocessor.cli.setup_logging(debug=False)

preprocessor.cli.validate_config(config)

preprocessor.config module

preprocessor.config.constructor_env_variables(loader, node)

Extracts the environment variable from the node’s value :param yaml.Loader loader: the yaml loader :param node: the current node in the yaml :return: the parsed string that contains the value of the environment variable

preprocessor.config.load_config(input_file: TextIO)

preprocessor.daemon module

preprocessor.daemon.run_daemon(config, host, port, listen_queue, write_queue)

Run the preprocessing daemon, listening on a redis queue for files to be preprocessed. After preprocessing the filename of the preprocessed files will be pushed to the output queue.

preprocessor.metadata module

preprocessor.metadata.evaluate_xpath(root, xpath)

preprocessor.metadata.extract_footprint(metadata_files, footprint_extractor: str = '//gml:target/eop:Footprint/gml:multiExtentOf\n/gml:MultiSurface/gml:surfaceMembers/gml:Polygon')

preprocessor.metadata.extract_metadata_for_stac(metadata_files: Dict[str, str], product_type: str, product_level: str | None)

Temporary function extracting necessary metadata to create a minimal STAC item. For now the xpaths are hardcoded here

preprocessor.metadata.extract_product_types_and_levels(metadata_files: List, config: dict)

Extracts product_types and product_levels found in the metadata based on configured XML xpath extractors.

preprocessor.metadata.parse_polygons_gsc(elem)

preprocessor.metadata.parse_ring(string)

preprocessor.metadata.serialize_coord_list(coords)

preprocessor.metadata.update_config_by_product_types_and_levels(metadata_files: List, config: dict)

Extracts product_type and product_level based on config, updates the config dict by type based config.

preprocessor.preprocess module

preprocessor.preprocess.copy_files(source, target, move=False)

preprocessor.preprocess.custom_postprocessor(source_dir, target_dir, preprocess_config, path, data_file_globs: List[str] = [], args=None, kwargs=None)

Preprocessing step for a custom preprocessing.

preprocessor.preprocess.custom_preprocessor(source_dir, target_dir, preprocess_config, path, data_file_globs: List[str] = [], args=None, kwargs=None)

Preprocessing step for a custom preprocessing.

preprocessor.preprocess.preprocess_browse(config: dict, browse_type: str, browse_report: dict, browse: dict, use_dir: str | None = None)

preprocessor.preprocess.preprocess_file(config: dict, file_path: str, use_dir: str | None = None)

Runs the preprocessing of a single file.

preprocessor.preprocess.preprocess_internal(preprocess_config, previous_step='unpack')

preprocessor.preprocessor_fix_core_dimap_image_ref module

preprocessor.preprocessor_fix_core_dimap_image_ref.attempt_to_create_hdr(source_dir, target_dir, preprocess_config, output: str = 'imagery.hdr')

preprocessor.preprocessor_fix_core_dimap_image_ref.attempt_to_set_projection(source_dir, target_dir, preprocess_config)

preprocessor.preprocessor_fix_core_dimap_image_ref.rename_reference_dimap(source_dir: PathLike, target_dir: PathLike, preprocess_config: dict, search: str, replace: str, output_file_name: str = 'imagery.dim')

preprocessor.preprocessor_iceye_helpers module

preprocessor.preprocessor_iceye_helpers.extract_rpc_metadata(root: Element)

preprocessor.preprocessor_iceye_helpers.extract_rpc_to_text_file(source_dir: PathLike, target_dir: PathLike, preprocess_config: dict, glob: str = '*ICEYE.xml', output_file_name: str = 'ICEYE.rpc')

preprocessor.preprocessor_iceye_helpers.get_rpc_value(tag, root)

preprocessor.stac module

class preprocessor.stac.HrefSortableAsset(href: str, title: str | None = None, description: str | None = None, media_type: str | None = None, roles: List[str] | None = None, extra_fields: Dict[str, Any] | None = None)

Bases: Asset

Helper function enabling sorting Assets by href.

description: str | None

A description of the Asset providing additional details, such as how it was processed or created. CommonMark 0.29 syntax MAY be used for rich text representation.

extra_fields: Dict[str, Any]

Optional, additional fields for this asset. This is used by extensions as a way to serialize and deserialize properties on asset object JSON.

href: str

Link to the asset object. Relative and absolute links are both allowed.

media_type: str | None

Optional description of the media type. Registered Media Types are preferred. See MediaType for common media types.

owner: Item | Collection | None

The Item or Collection that this asset belongs to, or None if it has no owner.

roles: List[str] | None

Optional, Semantic roles (i.e. thumbnail, overview, data, metadata) of the asset.

title: str | None

Optional displayed title for clients and users.

class preprocessor.stac.STATS_APPROX(value)

Bases: IntEnum

An enumeration.

APPROX_OK = 1

APPROX_OVERVIEW = 2

NO_APROX = 0

preprocessor.stac.create_simple_stac_item(preprocessor_config: dict, root_config: dict, upload_files: Dict[str, str], extra_files: Dict[str, str], product_type: str, product_level: str | None)

Temporary method creating a minimal STAC item from information about products uploaded and metadata files uploaded. Accepts: ‘upload_files’ dictionary of upload_files (images), where key is local path and value is remote path. ‘extra_files’ dictionary of extra_files (sidecar or metadata), where key is local path and value is remote path. Assuming metadata file to read and create a STAC info from is first to pick by iterator.

preprocessor.stac.create_stac_asset(local_path: str, remote_path: str, root_config: Dict[str, Any], asset_config: Dict[str, Any], name: str = '', aggregator: Dict = {}, is_image: bool = False, compute_statistics: bool = False, approx: STATS_APPROX = STATS_APPROX.APPROX_OK, force_histogram_min_value: float | None = None, force_histogram_max_value: float | None = None)

Helper function creating a STAC asset and filling it with image/metadata properties based on config.

preprocessor.stac.extract_asset_config_by_glob(local_path: str, stac_item_structure: Dict[str, Any], config: Dict[str, Any])

preprocessor.util module

class preprocessor.util.Timer

Bases: object

Helper timer class to allow logging of timing values

property elapsed

preprocessor.util.apply_gdal_config_options(preprocessor_config)

Applies config specific gdal configuration options for a given preprocessing step Returning original values to allow switching them back after preprocessing done.

preprocessor.util.convert_unit(size_in_bytes, unit='B')

Convert the size from bytes to other units like KB, MB, GB, TB

preprocessor.util.flatten(llist)

preprocessor.util.get_all_data_files(source_dir, preprocessor_config, data_file_globs=[])

Based on ‘data_file_globs’ configuration, gets all unique data file paths from folder matching any of the globs

preprocessor.util.get_size_in_bytes(file_path, unit)

Get size of file at given path in bytes

preprocessor.util.pairwise(col)

preprocessor.util.replace_ext(filename: str, new_ext: str, force_dot: bool = True) → str

preprocessor.util.set_gdal_options(config_options)

Sets a key, value dictionary of config options to gdal

preprocessor.util.workdir(config: dict, use_dir: str = None)

Module contents