Pandas version checks

  • [X] I have checked that this issue has not already been reported.

  • [X] I have confirmed this bug exists on the latest version of pandas.

  • [ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

timestamp = pd.to_datetime("2023-01-01", utc=True)
df = pd.DataFrame(
    {
        "index": pd.Series([pd.NaT, timestamp]),
        "value": pd.Series([0, 1]),
    }
).set_index("index")


assert df.loc[pd.Series([pd.NaT, timestamp, timestamp], dtype=df.index.dtype),][
    "value"
].tolist() == [0, 1, 1]


# The line bellow throws KeyError
# None of [DatetimeIndex(['NaT'], dtype='datetime64[ns]', name='index', freq=None)] are in the [index]
# But you'd expect [0]
assert df.loc[pd.Series([pd.NaT], dtype=df.index.dtype),]["value"].tolist() == [0]

Issue Description

If pd.NaT is in my df index, I should be able to get df.loc[pa.NaT, ].

This works in most cases, for example tz-naive, or if the argument of loc is a mix of NaT and valid values.

But it stops working if the index is tz-aware and the argument to loc is all NaT.

Expected Behavior

df.loc[pd.Series([pd.NaT], dtype=df.index.dtype),]["value"].tolist() should return [0]

Installed Versions

INSTALLED VERSIONS

commit : 0f437949513225922d851e9581723d82120684a6 python : 3.10.11.final.0 python-bits : 64 OS : Darwin OS-release : 22.6.0 Version : Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : None LOCALE : None.UTF-8

pandas : 2.0.3 numpy : 1.24.2 pytz : 2023.3 dateutil : 2.8.2 setuptools : 67.3.1 pip : 23.2.1 Cython : None pytest : 7.4.0 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.3 html5lib : None pymysql : None psycopg2 : 2.9.5 jinja2 : 3.1.2 IPython : 8.14.0 pandas_datareader: 0.9.0 bs4 : 4.12.2 bottleneck : None brotli : None fastparquet : None fsspec : 2023.6.0 gcsfs : None matplotlib : 3.6.0 numba : None numexpr : 2.8.4 odfpy : None openpyxl : None pandas_gbq : None pyarrow : 12.0.0 pyreadstat : None pyxlsb : None s3fs : 2023.6.0 scipy : 1.10.1 snappy : None sqlalchemy : 2.0.9 tables : None tabulate : 0.8.10 xarray : 2022.6.0 xlrd : None zstandard : None tzdata : 2023.3 qtpy : 2.2.0 pyqt5 : None

Comment From: jbrockmendel

This works on main. Could use a test.

Comment From: AparnaJayaraman

Hi, I’d like to work on this issue as my first contribution. From the description, it seems the problem is with handling NaT in tz-aware DateTimeIndex inside .loc. I’ll start by adding a test case that reproduces the failure, and then look into possible fixes in the indexing logic. Please let me know if anyone is already working on it.