-
[x] I have checked that this issue has not already been reported.
-
[x] I have confirmed this bug exists on the latest version of pandas.
-
[x] (optional) I have confirmed this bug exists on the master branch of pandas.
Code Sample, a copy-pastable example
In [11]: pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/London', nonexistent='shift_forward')
Out[11]: Timestamp('2021-03-28 03:00:00+0100', tz='Europe/London')
In [12]: pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/London', nonexistent='shift_forward') - pd.Timedelta(60, 'm')
Out[12]: Timestamp('2021-03-28 02:00:00+0100', tz='Europe/London')
In [13]: pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/London', nonexistent='shift_forward') - pd.Timedelta(61, 'm')
Out[13]: Timestamp('2021-03-28 00:59:00+0000', tz='Europe/London')
Problem description
When attempting to localize a non-existent time in the Europe/London timezone using the nonexistent='shift_forward', parameter, the resulting time is 03:00 summer time-- even though the time transition skips from 01:00 AM UTC to 02:00 AM summer time.
The issue appears to occur in European timezones that are in UTC when the time is not summer time (i.e. London, Dublin, and Lisbon). Timezones that start at UTC+1 and have their summer time conversions from 2 AM-3 AM work properly.
In [23]: pd.to_datetime('2021-03-28 02:30').tz_localize('Europe/Paris', nonexistent='shift_forward')
Out[23]: Timestamp('2021-03-28 03:00:00+0200', tz='Europe/Paris')
In [24]: pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/Dublin', nonexistent='shift_forward')
Out[24]: Timestamp('2021-03-28 03:00:00+0100', tz='Europe/Dublin')
In [25]: pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/Lisbon', nonexistent='shift_forward')
Out[25]: Timestamp('2021-03-28 03:00:00+0100', tz='Europe/Lisbon')
In [26]: pd.to_datetime('2021-03-28 02:30').tz_localize('Europe/Paris', nonexistent='shift_forward')
Out[26]: Timestamp('2021-03-28 03:00:00+0200', tz='Europe/Paris')
In [27]: pd.to_datetime('2021-03-28 02:30').tz_localize('Europe/Stockholm', nonexistent='shift_forward')
Out[27]: Timestamp('2021-03-28 03:00:00+0200', tz='Europe/Stockholm')
Expected Output
Per the documentation localizing with nonexistent='shift_forward' should "shift the nonexistent time forward to the closest existing time". In this case the closest existing time in Europe/London to 2021-03-28 01:30 would be 2021-03-28 02:00+0100.
In [11]: pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/London', nonexistent='shift_forward')
Out[11]: Timestamp('2021-03-28 02:00:00+0100', tz='Europe/London')
Output of pd.show_versions()
Comment From: mroeschke
Looks like there's Timestamp constructor differences as shown here:
In [2]: ts = pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/London', nonexistent='shift_forward')
In [3]: ts
Out[3]: Timestamp('2021-03-28 03:00:00+0100', tz='Europe/London')
In [4]: pd.Timestamp(ts.value, tz=ts.tz)
Out[4]: Timestamp('2021-03-28 03:00:00+0100', tz='Europe/London')
In [5]: pd.Timestamp(ts.value).tz_localize(ts.tz)
Out[5]: Timestamp('2021-03-28 02:00:00+0100', tz='Europe/London')
In [6]: pd.__version__
Out[6]: '1.3.0.dev0+1297.g7d4757b4de'
Comment From: mroeschke
Tracked down the issue to this line in _localize_tso
https://github.com/pandas-dev/pandas/blob/7d4757b4deb851bb44ab6bb20cdc404fa13fffcf/pandas/_libs/tslibs/conversion.pyx#L737
in which for the 'shift_forward'
we actually want the right edge index
# debugging git diff
+ print(obj.dts.hour)
dt64_to_dtstruct(obj.value + deltas[pos], &obj.dts)
+ print(obj.dts.hour)
In [1]: ts = pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/London', nonexistent='shift_forward')
2
3
Comment From: thomie
This issue seems to be fixed (version 2.3.1).
>>> pd.to_datetime('2021-03-28 01:30').tz_localize('Europe/London', nonexistent='shift_forward')
Timestamp('2021-03-28 02:00:00+0100', tz='Europe/London')