How to nest multiple regression functions of Sklearn?

Question

I am trying to implement a nested regression model separately which I got as an output from TPOT. The output of TPOT is:

RandomForestRegressor(XGBRegressor(XGBRegressor(**args1), **args2), **args3)

My code is so far:

from xgboost import XGBRegressor
from sklearn.ensemble import RandomForestRegressor

xgb1 = XGBRegressor(**args1)
xgb2 = XGBRegressor(**args2)
rf = RandomForestRegressor(**args3)

I am not sure how I can combine them properly in the order of TPOT's answer.

Answer 1

TPOT Classifier and Regressor make available a scikit-learn Pipeline object that already does that for you.

If you look at the TPOT API both TPOTClassifier and TPOTRegressor expose an attribute fitted_pipeline_ which will hold the best scikit-learn Pipeline TPOT could find. An example of a scikit-learn Pipeline:

PolynomialFeatures(degree=2, include_bias=False, interaction_only=False),
    XGBRegressor(learning_rate=0.1, max_depth=4, min_child_weight=14, n_estimators=100, n_jobs=1, objective="reg:squarederror", subsample=1.0, verbosity=0)

You can either dump it for later load, so you don't have to retrain your model, or you can simply export the best pipeline using TPOT Classifier and Regressor built-in function to export your optimized Pipeline as Python Code so you can re-fit your model:

tpot.export('tpot_digits_pipeline.py')

If for some reason you only have that output posted in the question, you can recreate the scikit-learn Pipeline like this:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import make_pipeline

tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)
features = tpot_data.drop('target', axis=1)
training_features, testing_features, training_target, testing_target = \
            train_test_split(features, tpot_data['target'], random_state=42)

exported_pipeline = make_pipeline(
  RandomForestRegressor(XGBRegressor(XGBRegressor(<replace with actual arg list>), <replace with actual arg list>), <replace with actual arg list>)
)

exported_pipeline.fit(training_features, training_target)