http://stackoverflow.com/questions/22670904/python-pandas-turn-absolute-periods-into-relative-periods/22671904?noredirect=1#comment34537319_22671904
this works, but I think could be more efficient on a whole frame basis
In [37]: def f(x):
....: y = x.dropna()
....: return Series(y.values,x.index[len(x)-len(y):])
....:
In [40]: roller = pd.rolling_sum(df,3).reset_index(drop=True)
In [41]: roller
Out[41]:
1 2 3
0 NaN NaN NaN
1 NaN NaN NaN
2 61 56 56
3 9 9 9
4 12 12 12
5 NaN 15 15
6 NaN NaN 11
7 NaN NaN NaN
[8 rows x 3 columns]
In [43]: roller.apply(f).reindex_like(roller)
Out[43]:
1 2 3
0 NaN NaN NaN
1 NaN NaN NaN
2 NaN NaN NaN
3 NaN NaN 56
4 NaN 56 9
5 61 9 12
6 9 12 15
7 12 15 11
[8 rows x 3 columns]
Comment From: dsm054
I think we'd want a call pattern which would support pulling the non-nan values up to the head -- which is actually what I usually need -- as well as pushing them down to the tail, so we'd need both an axis and a direction (a way to position "down").
Comment From: dsm054
I'm also not sure that dropna is the right method to use, because this will never actually return a smaller frame. It's more like a shift + fillna.
Comment From: jreback
yep.....its more like movena; not sure that really want another method name though.....
Comment From: dsm054
If we don't want to add a new method name, then maybe we should use fillna. It already accepts a method argument, and we could simply add broll/froll or something (bfall/ffall)? Because fillna can already return series/frames which still have nans, it's really like a "do something with some nans, maybe" in practice anyway.
Comment From: jreback
good idea! and I like those method names. you could use the limit argument to put in spacers (e.g. offsets from the end) as well.
Comment From: jreback
maybe df.shift(method='broll') or df.shift(3,method='froll')
Comment From: dsm054
Either would work for me: it's sort of a shift (because it repositions) and sort of a fillna (because it changes non-nan values based on the nan values -- or maybe I should say that the other way around.)
I'm not sure what df.shift(3, method='froll') would do, though. Group all the non-nan values and assign them to .iloc[3:]? To .iloc[:-3]?
Comment From: jreback
how about head/tail instead of froll/broll then it's clear which end u r talking about
so df.shift(3,method='head') would give u 3 rows of nan followed by squashed data
this might be tricky as what if u can't fit the data in (after squashing) will it expand the frame?
Comment From: dsm054
If we're viewing falling as a special shift, I don't think it should expand the frame because shift wouldn't. But I wonder if we should make it explicit that it's different from a normal shift in that it's value-dependent, not merely index-dependent, and call it valid_to_head or valid_to_tail or notnull_to_head or notnull_to_tail.
[Aside: I don't care much about the behaviour of the df.shift(3, method='somename') case because when I need this, I only ever need the non-nan values pushed to one of the edges.]
Comment From: jreback
what about df.shift('head|tail') then? makes it clear what its doing
Comment From: dsm054
Syntactic question: so you mean to bypass the method kwarg entirely and have us branch on periods instead?
Semantic question: just to make sure we're breaking the symmetry the same way :^), do you mean that head would push all the valid data to the start, and tail to the end? (That's how I read it.)
Comment From: jreback
well, could branch on periods or use a method kw (and make periods optional),
maybe this is the best, and just raise if periods is None and method is None
Series.shift(periods=None,freq=None,method=None)
It think your meaning of head/tail is right (at least that's what I would expect)
Comment From: jbrockmendel
I think shift now takes a list. You’d need to figure out what to pass it so it would be a 2 liner instead of a single call.