Pandas version checks

  • [x] I have checked that this issue has not already been reported.

  • [x] I have confirmed this bug exists on the latest version of pandas.

  • [x] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

  • Create a series with Index and a MultiIndex to use for reindexing later
>>> series = pd.Series(
...   [26.7300, 24.2550],
...   index=pd.Index([81, 82], name='a')
... )
>>> series
a
81    26.730
82    24.255
dtype: float64
>>> series.index
Index([81, 82], dtype='int64', name='a')
>>> other_index = pd.MultiIndex(
...   levels=[
...     pd.Index([81, 82], name='a'),
...     pd.Index([np.nan], name='b'),
...     pd.Index([
...       '2018-06-01', '2018-07-01'
...     ], name='c')
...   ],
...   codes=[
...     [0, 0, 1, 1],
...     [0, 0, 0, 0],
...     [0, 1, 0, 1]
...   ],
...   names=['a', 'b', 'c']
... )
>>> other_index
MultiIndex([(81, nan, '2018-06-01'),
            (81, nan, '2018-07-01'),
            (82, nan, '2018-06-01'),
            (82, nan, '2018-07-01')],
           names=['a', 'b', 'c'])
  • reindex to MultiIndex (other_index) which expands series.index by two more levels.
  • unfortunately the reindex sets all values of the original series to NaN which can be fixed by turning series.index into a 1-level MultiIndex first

>>> series.reindex(other_index) # this removes all values of the series
a   b    c         
81  NaN  2018-06-01   NaN
         2018-07-01   NaN
82  NaN  2018-06-01   NaN
         2018-07-01   NaN
dtype: float64
  • apply to_mi(...) to turn the series.index into a 1-level MultiIndex
  • rerun reindex on the new series with MultiIndex and the values are maintained/filled as expected
>>> def to_mi(series):
...   if isinstance(series.index, pd.MultiIndex):
...     series_mi = series.index
...   else:
...     level_names = [series.index.name]
...     level_values = [series.index]
...     series_mi = pd.MultiIndex.from_arrays(level_values, names=level_names)
...   series_with_mi = pd.Series(series.values, index=series_mi, name=series.name)
...   return series_with_mi
... 
>>> series_mi = to_mi(series)
>>> series_mi
a 
81    26.730
82    24.255
dtype: float64
>>> series_mi.index
MultiIndex([(81,),
            (82,)],
           names=['a'])
>>> series_mi.reindex(other_index)
a   b    c         
81  NaN  2018-06-01    26.730
         2018-07-01    26.730
82  NaN  2018-06-01    24.255
         2018-07-01    24.255
dtype: float64

Issue Description

In the above case, series.reindex(multi_index) will turn the series values to NaN when the series has a single Index. However when the series index is converted to a 1-level MultiIndex prior to the reindex, the values are maintained and filled as expected.

In my opinion it shouldn't matter if a 1-level MultiIndex or an Index is used for a reindex - the outcomes should be the same.

As a further discussion point (here or elsewhere), this issue (and others) also begs the question why a distinction between Index and MultiIndex is necessary (I suspect there are historic reasons). I would imagine that many issues (and code) would go away if MultiIndex was used exclusively (even for 1-dimensional indices).

Expected Behavior

The missing levels in series_mi (compared to other_index) are added and the values of the partial index from the original series are used to fill the places of the added indices.

>>> series_mi.reindex(other_index)
a   b    c         
81  NaN  2018-06-01    26.730 # from index <81> of `series` (`series_mi`)
         2018-07-01    26.730 # from index <81> of `series` (`series_mi`)
82  NaN  2018-06-01    24.255 # from index <82> of `series` (`series_mi`)
         2018-07-01    24.255 # from index <82> of `series` (`series_mi`)
dtype: float64

Installed Versions

INSTALLED VERSIONS ------------------ commit : 3979e954a339db9fc5e99b72ccb5ceda081c33e5 python : 3.11.11 python-bits : 64 OS : Linux OS-release : 6.12.11-200.fc41.x86_64 Version : #1 SMP PREEMPT_DYNAMIC Fri Jan 24 04:59:58 UTC 2025 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_AU.UTF-8 LOCALE : en_AU.UTF-8 pandas : 3.0.0.dev0+1909.g3979e954a3.dirty numpy : 1.26.4 dateutil : 2.9.0.post0 pip : 24.2 Cython : 3.0.11 sphinx : 8.1.3 IPython : 8.32.0 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.13.3 blosc : None bottleneck : 1.4.2 fastparquet : 2024.11.0 fsspec : 2025.2.0 html5lib : 1.1 hypothesis : 6.125.2 gcsfs : 2025.2.0 jinja2 : 3.1.5 lxml.etree : 5.3.0 matplotlib : 3.10.0 numba : 0.61.0 numexpr : 2.10.2 odfpy : None openpyxl : 3.1.5 psycopg2 : 2.9.10 pymysql : 1.4.6 pyarrow : 19.0.0 pyreadstat : 1.2.8 pytest : 8.3.4 python-calamine : None pytz : 2025.1 pyxlsb : 1.0.10 s3fs : 2025.2.0 scipy : 1.15.1 sqlalchemy : 2.0.38 tables : 3.10.2 tabulate : 0.9.0 xarray : 2024.9.0 xlrd : 2.0.1 xlsxwriter : 3.2.2 zstandard : 0.23.0 tzdata : 2025.1 qtpy : None pyqt5 : None

Comment From: rhshadrach

Thanks for the report! Is it possible to shorten your example? Making it as short as possible to demonstrate the behavior would certainly be appreciated.

Comment From: ssche

Thanks for the feedback. I managed to shorten the example and provided additional comments.

Comment From: micheleuap

Adding to this because I've seen this situation before.

This is about the fact when calling reindex, it makes a difference if the calling frame has a regular index, or an "identical" single-level multi-index. 

series = pd.Series([1, 2], index=pd.Index(["x", "y"], name="lvl1"))

# the same series as above, with the index converted to multi-index
series_mi = series.copy()
series_mi.index = pd.MultiIndex.from_frame(series.index.to_frame())

# a multi-index with a level not present in the index of the series
idx = pd.MultiIndex.from_product([["x", "y"], ["m", "n"]], names=["lvl1", "lvl2"])

# this returns nans
series.reindex(idx)

# this returns a series with values [1,1,2,2]
series_mi.reindex(idx)

Comment From: rhshadrach

Thanks @ssche - finally got some time to take a look into this. I believe this can currently be accomplished with the level argument.

series = pd.Series(
   [26.7300, 24.2550],
   index=pd.Index([81, 82], name='a')
)
other_index = pd.MultiIndex(
   levels=[
     pd.Index([81, 82], name='a'),
     pd.Index([np.nan], name='b'),
     pd.Index([
       '2018-06-01', '2018-07-01'
     ], name='c')
   ],
   codes=[
     [0, 0, 1, 1],
     [0, 0, 0, 0],
     [0, 1, 0, 1]
   ],
   names=['a', 'b', 'c']
 )
series.reindex(other_index, level="a")
# a   b    c         
# 81  NaN  2018-06-01    26.730
#          2018-07-01    26.730
# 82  NaN  2018-06-01    24.255
#          2018-07-01    24.255
# dtype: float64

But it makes sense to me to do this automatically if it's feasible. PRs are welcome!

Comment From: ssche

From https://pandas.pydata.org/docs/reference/api/pandas.Series.reindex.html#pandas.Series.reindex, it seems that level=None (the default) is currently undefined as input:

level int or name Broadcast across a level, matching Index values on the passed MultiIndex level.

Could we repurpose level=None as kind of "autodetect" to find overlapping levels? Is level=None used for other use-cases already?

Comment From: rhshadrach

I think so. I do not see the current behavior as being a useful op, so I think we can do this without deprecation (but in a major release only).

Comment From: Roline-Stapny

take I am planning to add a check in series.reindex(..) method to set level value automatically if series index name is present in the MultiIndex level