Python bisect and Pandas dataframe: KeyError

Question

I am using bisect for finding the first index that is greater than my value:

bisect.bisect_left(X['x'].iloc[116:231], x=0.15)

X is in ascending order and the first value X['x'].iloc[116] is 0.275. The bisect function above gives me KeyError: 57. I have understand that it cannot find the corresponding key but cannot understand why it tries to find the key '57'. My expectation is that the first greater value than 0.15 is 0.275 and it should give me the index 116.

I have used bisect because my list is already sorted, but if there is more efficient way to find the first index greater than a specific value I would be pleasure if you let me know.

---

Edit:

I have tried with other indexes and x values and results are getting strange. The smallest number is 0.125:

bisect.bisect_left(X[0:10]['x'], x=0.125)  # works and gives index 0
bisect.bisect_left(X[1:10]['x'], x=0.125)  # KeyError: 0
bisect.bisect_left(X[2:10]['x'], x=0.125)  # KeyError: 1
bisect.bisect_left(X[3:10]['x'], x=0.125)  # KeyError: 1
bisect.bisect_left(X[4:10]['x'], x=0.125)  # KeyError: 3
bisect.bisect_left(X[5:10]['x'], x=0.125)  # KeyError: 2
bisect.bisect_left(X[6:10]['x'], x=0.125)  # KeyError: 2
bisect.bisect_left(X[7:10]['x'], x=0.125)  # KeyError: 1

I have tried to debug but could not manage to track the code giving this error.

Answer 1

The bisect library is designed to be used with arrays not with pandas.Series.

The problem in this case arises from the fact that bisect_left(a, x, lo=0, hi=len(a)) may try to access all elements between a[lo] and a[hi]. As a pandas.Series overwrites the use of the [] bracket operator when the bisect_left function internally calls a[0] it will not call the first element of the passed array but try to acces the element of the series with index 0 which is not present in X['x'].iloc[116:231].

In order to solve your problem you thus have to either cast the pandas.Series into for example a list and add the index displacement

bisect.bisect_left(list(X['x'].iloc[116:231]), x=0.15)+116

or specify the limits for lo and hi explicitly.

bisect.bisect_left(X['x'].iloc[116:231], x=0.15, lo=116, hi=231)