Pandas version checks
-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[ ] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
import numpy as np
empty = pd.DataFrame([[None, None, None, None], [None, None, None, None], [None, None, None, None], [None, None, None, None]], columns=list("ABCD"), dtype=np.float64)
print(empty.dtypes)
# A float64
# B float64
# C float64
# D float64
# dtype: object
full_a = pd.DataFrame([[1.0, 2.0, "3.0", 4.0],[5.0,6.0,"7.0",8.0], [9.0,10.0,"11.0",12.0], [13.0,14.0,"15.0",16.0]], columns=list("ABCD"))
print(full_a.dtypes)
# A float64
# B float64
# C object
# D float64
# dtype: object
full_b = pd.DataFrame([[1.5, 2.0, "3.0", 4.0], [5.0,6.5,"7.0",8.0], [9.0,10.0,"11.0",12.0], [13.0,14.0,"15.0",16.5]], columns=list("ABCD"))
print(full_b.dtypes)
# A float64
# B float64
# C object
# D float64
# dtype: object
combined_1 = empty.fillna(full_a)
print(combined_1.dtypes)
# A int64
# B int64
# C object
# D int64
# dtype: object
combined_2 = empty.fillna(full_b)
print(combined_2.dtypes)
# A object
# B object
# C object
# D object
# dtype: object
Issue Description
The returned types of pandas dataframe fillna method gives inconsistent resulting types between a column that contains integral float values, and ones that don't. This leads to very confusing behavior, where the exact values of the input data (even if it was correctly starting as float64s in both dataframes) can affect the output types. In particular, if both the starting column and the merging column have the float64 dtype, as a user I would expect the output column to have a float64 dtype, but instead I get an int64 if all the values happen to be integral, otherwise I get an object dtype?! This behavior is further only observed if one of the other columns happen to be (correctly) an object dtype, when again, I expected the types of unrelated columns not to affect each other.
I know there are currently changes undergoing surrounding casting of types, but here as all types are being inputted correctly I didn't expect any casting to be being performed as part of this operation?
Expected Behavior
In the above example, I expected both combined_1 and combined_2 to have the same dtypes as each other.
I also expected both of them to actually have dtypes of float64 for cols A, B and D, given the input types are float64. The object type for those columns of combined_2 is particularly confusing in this case
Installed Versions
Also tested on 2.3.0 (sorry, website still says 2.2.3 is latest):
Comment From: iabhi4
Thanks for raising this, I investigated the dtype inconsistency and traced it to how fillna(DataFrame) calls where(self.notna(), other). When one column is object, it triggers coercion of all columns to object, even if others are float64. Replacing this with a column-wise np.where(notna, lhs, rhs) preserves expected dtypes.
Behavior aligns with what users intuitively expect, float columns stay float, object stays object. Same overall complexity (O(n × m)), but avoids full mask allocation and dtype promotion. Can submit a PR if this approach looks good to maintainers
Comment From: jbrockmendel
Replacing this with a column-wise np.where(notna, lhs, rhs) preserves expected dtypes
Operating column-wise is definitely the solution here, but we probably want to keep using the pandas where implementation rather than np.where since it will be robust for non-numpy dtypes.
Comment From: iabhi4
Thanks for the guidance @jbrockmendel! I’ve updated the logic to use Series.where as suggested and added a regression test to ensure dtype preservation