Pandas version checks

  • [x] I have checked that this issue has not already been reported.

  • [x] I have confirmed this bug exists on the latest version of pandas.

  • [ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

index1_without_name = pd.Index([1, 2])
index1_with_name = pd.Index([1, 2], name="x")
index2_without_duplicates = pd.Index([2, 3])
index2_with_duplicates = pd.Index([2, 3, 2])

multi_index1_without_name = pd.MultiIndex.from_tuples([(1, "a"), (2, "b")])
multi_index1_with_name = pd.MultiIndex.from_tuples([(1, "a"), (2, "b")], names=["x", "y"])
multi_index2_without_duplicates = pd.MultiIndex.from_tuples([(2, "b"), (3, "c")])
multi_index2_with_duplicates = pd.MultiIndex.from_tuples([(2, "b"), (3, "c"), (2, "b")])

# These work
print(index1_without_name.union(index2_without_duplicates))
print(index1_without_name.union(index2_with_duplicates))
print(index1_with_name.union(index2_without_duplicates))
print(index1_with_name.union(index2_with_duplicates))
print(multi_index1_without_name.union(multi_index2_without_duplicates))
print(multi_index1_without_name.union(multi_index2_with_duplicates))
print(multi_index1_with_name.union(multi_index2_without_duplicates))

# This one raises
print(multi_index1_with_name.union(multi_index2_with_duplicates))

Issue Description

For 2 MultiIndex instances i1 and i2, i1.union(i2) behaves inconsistently depending on whether i1 has names and whether i2 has duplicates:

  • If i1 has no names or i2 has no duplicates then i1.union(i2) works as expected
  • If i1 has names and i2 has duplicates then i1.union(i2) raises ValueError: cannot join with no overlapping index names

In addition, if i1 and i2 are plain Index instances, then the case that is problematic for MultiIndex (names and duplicates) works as expected.

Expected Behavior

I expect no exception to be raised. The result should contain the duplicate values of the second MultiIndex as duplicates, just as in the other cases for consistency (although personally this did surprise me, but that's a different topic).

Installed Versions

INSTALLED VERSIONS ------------------ commit : c888af6d0bb674932007623c0867e1fbd4bdc2c6 python : 3.12.7 python-bits : 64 OS : Linux OS-release : 6.14.0-27-generic Version : #27~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 17:38:49 UTC 2 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 2.3.1 numpy : 1.26.4 pytz : 2024.2 dateutil : 2.9.0.post0 pip : 25.1.1 Cython : None sphinx : None IPython : 8.32.0 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : None blosc : None bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : None html5lib : None hypothesis : None gcsfs : None jinja2 : 3.1.4 lxml.etree : None matplotlib : None numba : None numexpr : 2.10.2 odfpy : None openpyxl : None pandas_gbq : None psycopg2 : 2.9.10 pymysql : None pyarrow : 18.0.0 pyreadstat : None pytest : 8.3.4 python-calamine : None pyxlsb : None s3fs : None scipy : None sqlalchemy : None tables : 3.10.1 tabulate : None xarray : None xlrd : None xlsxwriter : None zstandard : None tzdata : 2024.2 qtpy : None pyqt5 : None

Comment From: WillAyd

Thanks for the find - there is something strange going on here.

The result should contain the duplicate values of the second MultiIndex as duplicates, just as in the other cases for consistency (although personally this did surprise me, but that's a different topic).

This sounds like a separate issue with Index.union not returning unique values. Could you check for an existing issue on that, and if nothing open a new one with a minimal reproducer?

Comment From: torfsen

This sounds like a separate issue with Index.union not returning unique values.

@WillAyd I'm not sure, but #56137 sounds like this is the intended (but undocumented) behavior.

Comment From: WillAyd

Ah nice find! In that case, it does seem intentional