how to convert multiindex to datetimeindex? (without reindex())

Question

I need to use "modin" dataframes, which dont not work with multindexes (at one point i do df.reindex(idx), where idx is a multilevel index), so: how can I convert a multi index to a single index? (merging both levels together)

minimal sample:

import pandas as pd
idx = pd.DatetimeIndex(['2019-07-17 22:43:00',
            '2019-07-17 22:44:00',
            '2019-07-17 22:45:00',
            '2019-07-17 22:46:00',
            '2019-07-17 22:47:00',
            '2019-07-17 22:48:00',
            '2019-07-17 22:49:00',
            '2019-07-17 22:50:00',
            '2019-07-17 22:51:00',
            '2019-07-17 22:52:00', 
            '2019-07-23 22:33:00',
            '2019-07-23 22:34:00',
            '2019-07-23 22:35:00',
            '2019-07-23 22:36:00',
            '2019-07-23 22:37:00',
            '2019-07-23 22:38:00',
            '2019-07-23 22:39:00',
            '2019-07-23 22:40:00',
            '2019-07-23 22:41:00',
            '2019-07-23 22:42:00'] ) 

idx = pd.MultiIndex.from_tuples(zip( idx.date, idx.time))

dates_new   =  idx.get_level_values(0).unique()  
times_new =  idx.get_level_values(1).unique()

idx = pd.MultiIndex.from_product([dates_new,times_new]) 
idx = pd.DatetimeIndex(idx)
print(idx)

the following works, but is there any way to speed it up (on large datasets) ?

[datetime.datetime.combine(date,time) for date,time in idx.values]

Answer 1

Your problem is starting from a DateTimeIndex, you want to find all combination of Date and Time and convert that to a new DateTimeIndex.

I would not use .time access since that gives a datetime object, which doesn't play along nicely with Pandas. Instead, let's try:

dates_new = set(idx.normalize())
times_new = set(idx - idx.normalize())

from itertools import product
new_idx = pd.DatetimeIndex([x+y for x,y in product(dates_new, times_new)])

Output:

DatetimeIndex(['2019-07-23 22:36:00', '2019-07-23 22:41:00',
               '2019-07-23 22:50:00', '2019-07-23 22:40:00',
               '2019-07-23 22:45:00', '2019-07-23 22:51:00',
               '2019-07-23 22:33:00', '2019-07-23 22:42:00',
               '2019-07-23 22:37:00', '2019-07-23 22:46:00',
               '2019-07-23 22:43:00', '2019-07-23 22:52:00',
               '2019-07-23 22:34:00', '2019-07-23 22:47:00',
               '2019-07-23 22:38:00', '2019-07-23 22:35:00',
               '2019-07-23 22:44:00', '2019-07-23 22:49:00',
               '2019-07-23 22:39:00', '2019-07-23 22:48:00',
               '2019-07-17 22:36:00', '2019-07-17 22:41:00',
               '2019-07-17 22:50:00', '2019-07-17 22:40:00',
               '2019-07-17 22:45:00', '2019-07-17 22:51:00',
               '2019-07-17 22:33:00', '2019-07-17 22:42:00',
               '2019-07-17 22:37:00', '2019-07-17 22:46:00',
               '2019-07-17 22:43:00', '2019-07-17 22:52:00',
               '2019-07-17 22:34:00', '2019-07-17 22:47:00',
               '2019-07-17 22:38:00', '2019-07-17 22:35:00',
               '2019-07-17 22:44:00', '2019-07-17 22:49:00',
               '2019-07-17 22:39:00', '2019-07-17 22:48:00'],
              dtype='datetime64[ns]', freq=None)