Have written an application in Python that handles a large amount of data used to generate graphics. I need help finding a technique to allow the user to scroll through the data, in both directions, while displaying a segment of the data in a graphic. Ideally only data which would be visible would be read from the database.
I am currently using a combination of Pandas, Sqlalchemy and bqplot, but if there are better packages for implementing the desired functionality I am willing to change.
Sqlalchemy seems like a good bet for handling the database but I have found the documentation hard to understand and need a nudge in the right direction. Any help or advice will be appreciated.
Thanks, Steve
You can store your data in a grid whose each rectangle is the size of the user's display:
This way, you need to load at most 4 rectangles.
As to the database, sqlalchemy is great, but it does fancy ORM stuff you don't need. In fact, as a rule of thumb, if you're not going to do table joins (which, by your description you're not), you don't need an SQL database at all, but rather a large key-value store. Here is a
pymongotutorial, for example, but just choose any key-value store that works for you.