Pandas version checks
-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[x] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
for do_parq_roundtrip in [False, True]:
for do_fillna in [False, True]:
df = df_comparison.copy().reset_index(drop=True)
print("=" * 100)
parq_str = "with" if do_parq_roundtrip else "without"
fillna_str = "with" if do_fillna else "without"
print(f"{parq_str} parquet roundtrip, {fillna_str} fillna")
print("=" * 100)
# Do parquet roundtrip
if do_parq_roundtrip:
df.to_parquet("df.parquet", index=True)
df = pd.read_parquet("df.parquet")
print("Column datatypes:")
print(df.dtypes)
print()
# Detect mismatch
if do_fillna:
is_mismatch = df.col_a.fillna(2) != df.col_b.fillna(2)
else:
is_mismatch = df.col_a != df.col_b
print("Mismatch detected @:")
print(np.argwhere(is_mismatch.fillna(False)))
print()
df["is_mismatch"] = is_mismatch
# Print rows where there is a detected mismatch
print("Detected mismatch rows:")
print(df[df.is_mismatch])
print("=" * 100)
print()
Issue Description
I get inconsistent results when comparing two int64[pyarrow] columns, depending on whether I use fillna or first store the dataframe in parquet and read it again.
The input data is the result of merging two dataframes resulting from df = pd.read_sql(f"select * from some.Table", some_connection, dtype_backend="pyarrow"). I've redacted the results by selecting only certain columns and renaming them. The dataframe contains two columns (col_a and col_b) that are compared. These columns are of dtype int64[pyarrow] and contain either 0, 1, or NA. There is a third column ManualIndex which I've added to make sure nothing cheeky is happening with the index, but which is probably useless.
I've attached a .parquet and .feather export of the data. But keep in mind storing and then re-loading the data apparently has an effect on the output of the comparison, so loading this data and running the script probably gives a different output then the one I've posted below.
The output of the script gives me the following results:
====================================================================================================
without parquet roundtrip, without fillna
====================================================================================================
Column datatypes:
ManualIndex int64
col_a int64[pyarrow]
col_b int64[pyarrow]
dtype: object
Mismatch detected @:
[[252518]
[252519]]
Detected mismatch rows:
ManualIndex col_a col_b is_mismatch
252518 252518 1 <NA> <NA>
252519 252519 1 <NA> <NA>
====================================================================================================
====================================================================================================
without parquet roundtrip, with fillna
====================================================================================================
Column datatypes:
ManualIndex int64
col_a int64[pyarrow]
col_b int64[pyarrow]
dtype: object
Mismatch detected @:
[[252512]
[252513]
[252518]
[252519]]
Detected mismatch rows:
ManualIndex col_a col_b is_mismatch
252512 252512 1 1 True
252513 252513 1 1 True
252518 252518 1 <NA> True
252519 252519 1 <NA> True
====================================================================================================
====================================================================================================
with parquet roundtrip, without fillna
====================================================================================================
Column datatypes:
ManualIndex int64
col_a int64[pyarrow]
col_b int64[pyarrow]
dtype: object
Mismatch detected @:
[]
Detected mismatch rows:
Empty DataFrame
Columns: [ManualIndex, col_a, col_b, is_mismatch]
Index: []
====================================================================================================
====================================================================================================
with parquet roundtrip, with fillna
====================================================================================================
Column datatypes:
ManualIndex int64
col_a int64[pyarrow]
col_b int64[pyarrow]
dtype: object
Mismatch detected @:
[[252505]
[252506]]
Detected mismatch rows:
ManualIndex col_a col_b is_mismatch
252505 252505 1 1 True
252506 252506 1 1 True
====================================================================================================
As you can see the results are different for each combination of settings.
Expected Behavior
My expectations:
- I expect the roundtrip to a parquet file and back to have no bearing on the comparison at all.
- I expect is_mismatch to be True only at ManualIndex 252518 and 252519 when using fillna(2) before comparing.
- I expect is_mismatch to be NA at ManualIndex 252518 and 252519 when not using fillna before comparing.
- I expect is_mismatch to be False everywhere else.
- I expect that when is_mismatch is NA that it does not get found by np.argwhere or result in returned rows when using it to index into a dataframe.
I don't understand how the rows where col_a and col_b are both 1 can ever result in them not being equal according to the comparison. When I filter on these rows and do the comparison manually suddenly the comparison evalues to them being equal.
In short I am very confused, am I doing something wrong here?
Installed Versions
Comment From: rhshadrach
In short I am very confused, am I doing something wrong here?
I don't believe so. However I cannot reproduce your issues, as you indicated. Unfortunately there is nothing we can do to debug without a reproducible example and for this reason do not allow issues to remain open without a reproducer. Closing for now, happy to reopen if one is provided.