Skip to content

Add trailing slashes if we're copying directories to S3 with fsspec

David Joy requested to merge dev/shared-directory-fix-for-incremental into main

Merge request template for patch (non-feature) branches. [click to expand]

See https://semver.org for versioning information:

Given a version number MAJOR.MINOR.PATCH, increment the:

   MAJOR version when you make incompatible API changes,
   MINOR version when you add functionality in a backwards compatible manner, and
   PATCH version when you make backwards compatible bug fixes.

Major version number is 0 for no expectation of backwards compatibility.

"feature" template should be used for significant features that increment GEOWATCH's minor version number. (Ex. 0.5.0 -> 0.6.0)

"patch" template should be used for bugfixes or minor features that increment GEOWATCH's patch version number. (Ex. 0.5.0 -> 0.5.1)

Short Description

Fix (?) issue where copying a directory up to an S3 "directory" that was not empty would place the source directory into the destination directory, rather than copying the contents of the source directory into the destination directory. There's a world where this isn't the desired default behavior, but it fixes the shared directory issue I was having with incremental mode.

Related MRs, Issues, and Other Links

Types of Changes

  • Documentation/tests
  • Bug fix
  • New feature [consider using the feature template instead]
  • Breaking change [use the feature template instead]

Changelog [optional]

Fixed

Added

Changed

  • Now when copying a directory up to a non-empty S3 "directory" with fsspec the contents of the source directory will be copied into the destination directory (rather than copying the source directory itself)

Merge request reports