Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
from datetime import datetime
import pandas as pd
from pandas import Timestamp
df = pd.DataFrame(
{'Betrag': {0: '-30,60', 1: '-45,00', 2: '-582,00'},
'Buchungsart': {0: 'Überweisung', 1: 'Überweisung', 2: 'Überweisung'},
'Dokumentdatum': {0: Timestamp('2024-01-25 00:00:00'),
1: Timestamp('2024-01-01 00:00:00'),
2: Timestamp('2024-01-08 00:00:00')},
'Fälligkeitsdatum': {0: Timestamp('2024-01-25 00:00:00'),
1: Timestamp('2024-01-31 00:00:00'),
2: Timestamp('2024-02-01 00:00:00')},
'Genauigkeit Betrag': {0: 'fix', 1: 'fix', 2: 'fix'},
'Genauigkeit Datum': {0: 'fix', 1: 'fix', 2: 'fix'},
'Kategorie': {0: 'Kategorie1',
1: 'Kategorie2',
2: 'Kategorie3'},
'Kontrahent': {0: 'Kontrahent1', 1: 'Kontrahent2', 2: 'Kontrahent3'},
'Kontrahent BIC': {0: '', 1: '', 2: ''},
'Kontrahent IBAN': {0: 'XX000000000000000001',
1: 'XX000000000000000002',
2: 'XX000000000000000003'},
'Kostenträger': {0: 'Ort',
1: 'Person',
2: 'Gegenstand'},
'Mandatsreferenz': {0: '', 1: '', 2: ''},
'Produkt': {0: 'Produkt1',
1: 'Produkt2',
2: 'Produkt3'},
'Zahlungsreferenz/SEPA Creditor Id': {0: '3276842762347',
1: '327684276234712',
2: '32768427'},
'erledigt': {0: '', 1: '', 2: ''},
'gesendet am': {0: Timestamp('2024-01-14 00:00:00'), 1: None, 2: None},
'laufende Summe': {0: '1.659,09', 1: '1.614,09', 2: '1.032,09'}}
)
aktuelles_Jahr = datetime.today().year
aktueller_Monat = datetime.today().month
df_nicht_erfasst = df.query("Fälligkeitsdatum.dt.year == @aktuelles_Jahr and "
"Fälligkeitsdatum.dt.month == @aktueller_Monat and "
"Buchungsart == 'Überweisung' and "
"`gesendet am` == ''")
Issue Description
After updating to pandas 2.2, when I run a program with the last line above, there is now the following warning displayed:
FutureWarning: The behavior of 'isin' with dtype=datetime64[ns] and castable values (e.g. strings) is deprecated. In a future version, these will not be considered matching by isin. Explicitly cast to the appropriate dtype before calling isin instead.
However, the line referenced in the warning does not contain any isin function.
Also, I am unclear what I am now supposed to do to remedy this warning.
Expected Behavior
Clarification about why this is about isinand pointer to explanation + example about how to change the code
Installed Versions
Comment From: phofl
Can you please provide a reproducer? The DataFrame is missing
Comment From: workflowsguy
@phofl, thank you. I have updated the code example.
Comment From: lithomas1
Thanks for the reproducer - I can reproduce on my laptop.
Tentatively milestoning for 2.2.1.
Comment From: lopof
The problem is "'gesendet am' == ''". Where you want to filter on "None" values, which are stored as NaT.
Could be fixed with
df_nicht_erfasst = df.query("Fälligkeitsdatum.dt.year == @aktuelles_Jahr and "
"Fälligkeitsdatum.dt.month == @aktueller_Monat and "
"Buchungsart == 'Überweisung' and "
"`gesendet am` == 'NaT'")
or
df_nicht_erfasst = df.query("Fälligkeitsdatum.dt.year == @aktuelles_Jahr and "
"Fälligkeitsdatum.dt.month == @aktueller_Monat and "
"Buchungsart == 'Überweisung' and "
"`gesendet am` != `gesendet am`")
Comment From: workflowsguy
@lopof, Thank you for looking into this.
Your first solution still gives the warning, while the second one does not.
As I have gained a little bit better understanding of the workings of pandas, I have changed the last line of the query statement to
"`gesendet am`.isnull()")
as, I think, this most clearly reflects the intention, namely filtering for scheduled transactions with a missing "sent on" date (and this also does not trigger the FutureWarning).
Comment From: lopof
@workflowsguy ah even better solution, nice. Didn't see that
Comment From: jbrockmendel
No warning on main. Can patch for 2.3.3 or close.