In Huggingface is it possible to add files to an existing dataset instead of overwrite it each time

108 Views Asked by At

I am creating a dataset from a generator and then I want to save it to the hub. However I actually want to append to the dataset in the hub and not overwrite it. Is this possible? My code is below. After running 10 times I would like each row duplicated 10 times. Each time I run it it overwrites the previous data instead of duplicating the rows.

def gen2():
        yield {"pokemon": "bulbasaur", "type": "grass"}
        yield {"pokemon": "squirtle", "type": "water"}

Dataset.from_generator(gen2)
        .push_to_hub("mine/pokemon", private=True)
0

There are 0 best solutions below