Pandas version checks
-
[X] I have checked that this issue has not already been reported.
-
[X] I have confirmed this bug exists on the latest version of pandas.
-
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
sukjun@galaxy4:~$ /usr/bin/python3
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'2.1.1'
>>> s = pd.Series([1, 2, 3, 4, 5])
>>> x = s.mean()
>>> x, type(x)
(3.0, <class 'numpy.float64'>)
>>> x = s.min()
>>> x, type(x)
(1, <class 'numpy.int64'>)
>>>
sukjun@galaxy4:~$ anaconda3/envs/myenv/bin/python3
Python 3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'2.1.1'
>>> s = pd.Series([1, 2, 3, 4, 5])
>>> x = s.mean()
>>> x, type(x)
(3.0, <class 'numpy.float64'>)
>>> x = s.min()
>>> x, type(x)
(1, <class 'int'>)
>>>
Issue Description
As I expected, the Series.mean() function returns a consistent data type, numpy.float64. However, I found that Series.min() function does not guarantee the same data type. Even though I use the same Python (3.10.12) and pandas (2.1.1) versions, the data type of the return value of the Series.min() or Series.max() function is returned differently depending on the Python interpreters. In some cases, it returns numpy.int64 or python's built-in integer type int. Is there anything I missed in the environment settings?
Expected Behavior
I expect that it returns the value as a consistent data type like numpy.int64.
Installed Versions
sukjun@galaxy4:~$ /usr/bin/python3
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.show_versions()
INSTALLED VERSIONS
------------------
commit : e86ed377639948c64c429059127bcf5b359ab6be
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-84-generic
Version : #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.1.1
numpy : 1.26.1
pytz : 2022.1
dateutil : 2.8.2
setuptools : 59.6.0
pip : 22.0.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None
sukjun@galaxy4:~$ anaconda3/envs/myenv/bin/python3
Python 3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.show_versions()
/lunit/home/sukjun/anaconda3/envs/myenv/lib/python3.10/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
INSTALLED VERSIONS
------------------
commit : e86ed377639948c64c429059127bcf5b359ab6be
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-84-generic
Version : #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.1.1
numpy : 1.26.1
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.3
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : 1.3.5
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.8.7
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None
Comment From: twoertwein
Are you using the same numpy version in both cases?
Comment From: MaruthiKo
Are you using the same numpy version in both cases?
As per the given details it seems he is using the same numpy version(1.26.1) in both the cases
Comment From: dandyrilla
@twoertwein Yes, I’m using the same NumPy version in both cases. @MaruthiKo Thank you for the reply!
Comment From: rhshadrach
The difference in dtype depends on whether you have bottleneck installed or not.
Comment From: rhshadrach
What's the expected result here? I was thinking we always returned Python objects for numerical scalars, but apparently not.
Comment From: dandyrilla
@rhshadrach Thank you for your comment! You're right. As you mentioned, I looked at how the returned data type differs depending on whether the bottleneck package is installed or not. If the bottleneck package is installed, it is returned as int format. Otherwise (if the bottleneck package does not exist), it is returned as numpy.int64 format.
With the bottleneck package installed:
sukjun@galaxy4:~$ anaconda3/envs/myenv/bin/python3
Python 3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> s = pd.Series([1, 2, 3, 4, 5])
>>> x = s.min()
>>> x, type(x)
(1, <class 'int'>)
>>>
After I uninstall the bottleneck package:
(myenv) sukjun@galaxy4:~$ pip3 uninstall bottleneck
Found existing installation: Bottleneck 1.3.5
Uninstalling Bottleneck-1.3.5:
Would remove:
/home/sukjun/anaconda3/envs/myenv/lib/python3.10/site-packages/Bottleneck-1.3.5.dist-info/*
/home/sukjun/anaconda3/envs/myenv/lib/python3.10/site-packages/bottleneck/*
Proceed (Y/n)? y
Successfully uninstalled Bottleneck-1.3.5
sukjun@galaxy4:~$ anaconda3/envs/myenv/bin/python3
Python 3.10.12 (main, Jul 5 2023, 18:54:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> s = pd.Series([1, 2, 3, 4, 5])
>>> x = s.min()
>>> x, type(x)
(1, <class 'numpy.int64'>)
>>>
However, it is difficult to understand why pandas' behavior varies depending on whether the other package 'bottleneck' is installed or not. Should I call this a bug, or not a bug? In my case, I am using the return values for JSON serialization, but sometimes the NumPy data type (e.g. np.int64) is not serializable, so I use the .item() method to obtain the python built-in data type int.
Comment From: rhshadrach
I think this is a bug.
Comment From: twoertwein
I think there are many places where pandas might return a numpy type instead of a builtin type. For np.float64, that seems okay'ish (it is a subclass of float) but np.int*, np.float32, np.bool_ are not subclasses of the builtin types.
so I use the
.item()method to obtain the python built-in data typeint.
int(...) should work whether it is a builtin int or numpy int.
Comment From: atetali
take
Comment From: lukemanley
Just noting there appears to be a fair amount of inconsistency across reduction operations in terms of whether a python scalar or a numpy scalar is returned. It might be nice to make this a bit more consistent:
(these are all without bottleneck installed)
import pandas as pd
ser = pd.Series([1.0, 2.0, 3.0])
for op in [
"any",
"all",
"count",
"nunique",
"min",
"max",
"mean",
"median",
"sum",
"prod",
"std",
"var",
"kurt",
"sem",
"skew",
]:
print(op, type(getattr(ser, op)()))
any <class 'numpy.bool_'>
all <class 'numpy.bool_'>
count <class 'numpy.int64'>
nunique <class 'int'>
min <class 'float'>
max <class 'float'>
mean <class 'numpy.float64'>
median <class 'float'>
sum <class 'numpy.float64'>
prod <class 'numpy.float64'>
std <class 'float'>
var <class 'float'>
kurt <class 'float'>
sem <class 'numpy.float64'>
skew <class 'numpy.float64'>