Feature Type

  • [x] Adding new functionality to pandas

  • [x] Changing existing functionality in pandas

  • [ ] Removing existing functionality in pandas

Problem Description

Python 3.14 will introduce compression.zstd as part of the standard library (PEP 784 and library documentation).

Currently, pandas uses the third-party zstandard library for Zstandard compression. Would it be possible to also support the standard library implementation? If not, we will need to keep using an external dependency even though an equivalent standard library module exists.

Feature Description

  • From Python 3.14 onwards, use compression.zstd internally.
  • For Python 3.13 and earlier, use the backports.zstd library, which provides the same API as compression.zstd, so that users can write code in the same way as with the standard library.
    Add backports.zstd as an install dependency when [compression] is specified.
  • Keep the existing zstandard library available during a migration period.
  • From a user perspective, the way parameters are passed in the options dictionary may change.
    While the functionality is equivalent, the API differs, so existing dictionaries may not work without modification.
    • If only the file extension or the compression level is specified, no changes are needed (e.g., compression='zstd', compression={'method': 'zstd', 'level': 1}).

Suggested migration steps

  1. Add an 'engine' key to the options dictionary, allowing 'zstandard' or 'compression.zstd' to be specified.
  2. Use the specified library.
  3. Default to 'zstandard' if not specified.
  4. Emit a warning that the default will change to compression.zstd in the future.
  5. On Python 3.13 and earlier, interpret 'compression.zstd' as backports.zstd.
  6. Change the default to compression.zstd when 'engine' is not specified.
  7. Remove the 'engine' key and always use compression.zstd.

Alternative Solutions

  • Continue supporting both compression.zstd and zstandard. It would leave the external dependency in place.
  • Convert the options dictionary internally without using an 'engine' key.
    Some mapping can be inferred from keyword names, but the maintenance cost may not be worth it.

Additional Context

No response

Comment From: vinegh4

take