How do i use Cassandra/ScyllaDB/MongoDB to store pandas which has dynamic columns and updating it?

Question

I have a timeseries pandas dataframe which dynamically increases the columns every minute:

Initial:

timestamp                100     200     300
2020-11-01 12:00:00       4       3       5

Next minute:

timestamp                100     200     300   500
2020-11-01 12:00:00       4       3       5     0
2020-11-01 12:01:00       4       3       5     25

The dataframe has these updated values and so on every minute.

I found that Cassandara and ScyllaDB are wide column databases but we have to define the columns while creating the structure of the tables. MongoDB could also be useful.

In my case it would update/change every minute. How do I store this on a database like Cassandra, ScyllaDB or even mongo to store this data?

Answer 1

Here's an example to get started with time series modeling and ScyllaDB: https://scylladb.github.io/care-pet/master/getting_started.html There are Spark examples in our blog too