Pandas version checks
-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[x] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
import warnings
warnings.simplefilter('error')
df = pd.DataFrame(
{'year': [2018, 2018, 2018],
'month': [1, 1, 1],
'day': [1, 2, 3],
'value': [1, 2, 3]})
df['date'] = pd.to_datetime(df[['year', 'month', 'day']])
Issue Description
With python 3.14 and the Pandas main branch (or 2.2.3 with pd.options.mode.copy_on_write = "warn") the above fails with:
Python 3.14.0a7+ (heads/main:276252565cc, Apr 27 2025, 16:05:04) [Clang 19.1.7 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.3.0.dev -- An enhanced Interactive Python. Type '?' for help.
Tip: You can use LaTeX or Unicode completion, `\alpha<tab>` will insert the α symbol.
In [1]: import pandas as pd
In [2]: df = pd.DataFrame(
...: {'year': [2018, 2018, 2018],
...: 'month': [1, 1, 1],
...: 'day': [1, 2, 3],
...: 'value': [1, 2, 3]})
...: df['date'] = pd.to_datetime(df[['year', 'month', 'day']])
<ipython-input-2-a8566e79621c>:6: ChainedAssignmentError: A value is trying to be set on a copy of a DataFrame or Series through chained assignment.
When using the Copy-on-Write mode, such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy.
Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html
df['date'] = pd.to_datetime(df[['year', 'month', 'day']])
In [3]: import warnings
In [4]: warnings.simplefilter('error')
In [5]: df = pd.DataFrame(
...: {'year': [2018, 2018, 2018],
...: 'month': [1, 1, 1],
...: 'day': [1, 2, 3],
...: 'value': [1, 2, 3]})
...: df['date'] = pd.to_datetime(df[['year', 'month', 'day']])
---------------------------------------------------------------------------
ChainedAssignmentError Traceback (most recent call last)
<ipython-input-5-a8566e79621c> in ?()
2 {'year': [2018, 2018, 2018],
3 'month': [1, 1, 1],
4 'day': [1, 2, 3],
5 'value': [1, 2, 3]})
----> 6 df['date'] = pd.to_datetime(df[['year', 'month', 'day']])
~/.virtualenvs/cp314-clang/lib/python3.14/site-packages/pandas/core/frame.py in ?(self, key, value)
4156 def __setitem__(self, key, value) -> None:
4157 if not PYPY:
4158 if sys.getrefcount(self) <= 3:
-> 4159 warnings.warn(
4160 _chained_assignment_msg, ChainedAssignmentError, stacklevel=2
4161 )
4162
ChainedAssignmentError: A value is trying to be set on a copy of a DataFrame or Series through chained assignment.
When using the Copy-on-Write mode, such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy.
Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html
In [6]: pd.__version__
Out[6]: '3.0.0.dev0+2080.g44c5613568'
With Python 3.14 there will be an optimization where the reference count is not incremented if Python can be sure that something above the calling scope will hold a reference for the life time of a scope. This is causing a number of failures in test suites when reference counts are checked. In this case I think it erroneously triggering the logic that the object is a intermediary.
Found this because it is failing the mpl test suite (this snippet is extracted from one of our tests).
With py313 I do not get this failure.
Expected Behavior
no warning
Installed Versions
It is mostly development versions of things, this same env with pd main also fails.
Comment From: rhshadrach
Thanks for the report! It sounds like we may need to disable these warnings for Python 3.14+ if the refcount cannot be relied upon.
cc @jorisvandenbossche @phofl
Comment From: rhshadrach
Since CoW is implemented using refcount, could there also be cases where we believe data is not being shared but it really is?
Comment From: jorisvandenbossche
Since CoW is implemented using refcount
The actual Copy-on-Write mechanism itself is implement using weakrefs, and does not rely on refcounting, I think.
The refcounts are used for the warning about chained assignments. While not essential for ensure correct behaviour (correctly copying when needed), those warnings are quite important towards the users for migrating / generally avoiding mistakes in the future (giving how widely spread chained assignment is).
So ideally we would be able to keep this warning working.
With Python 3.14 there will be an optimization where the reference count is not incremented if Python can be sure that something above the calling scope will hold a reference for the life time of a scope.
Do you know if there is a technical explanation of this somewhere? (or the PR implementing it? Didn't directly find anything mentioned in the 3.14 whatsnew page) I'll have to look a bit more into this change and the specific example if there is anything on our side that we can do detect when this happens or to otherwise deal with it.
Comment From: mpage
Hi! Sorry for the random comment, but @ngoldbaum pointed out this issue to me. I'm the author of the optimization. Happy to answer any questions or help brainstorm a solution with you.
Comment From: ngoldbaum
I just tried with both 3.14.0 RC1 and 3.14.0t RC1 and this is still an issue on current main.
Let me try to dig in to understand under exactly what circumstances this happens - maybe we can just change the check to < 3 instead of <= 3, because the stackref is always going to be missing on 3.14 and newer.
Comment From: ngoldbaum
Unfortunately no, it's not that easy, there are cases where none of the three references are stackrefs.