Plot huge amount of data with dates in x-axis

Question

I have a large database containing about 1 million entries. In one column there are dates in this form: '%Y-%m-%d %H:%M:%S. There is one entry every second.

I can select the period I want to plot from the database, e.g

date1 = '2015-04-22 20:28:50'

date2 = '2015-04-23 21:42:09'

and the other column I want to plot in the Y axis. As you can see in the specific example, from date1 to date2 it's about 86000 entries - or - points to plot.

Is there a way to plot efficiently these data using matplotlib, with the dates to show in the x axis? Of course not all dates can be shown, but as the plotting period is dynamic (I insert into a web form the dates I want), is there a way to program it so that the plot will be the best possible every time?

So far, I can put all the dates in a list, and all the Y data in another list.

Below is my code so far, which plots the data but the X-axis labels are nothing near what I want.

from buzhug import Base
import datetime
import data_calculations as pd
import matplotlib.pyplot as plt
import matplotlib
import time

date1 = '2015-04-22 20:28:50'
date2 = '2015-04-24 19:42:09'

db = Base('monitor').open()
result_set = db.select(['MeanVoltage','time'],"time>=start and time<=stop",  start=date1, stop=date2)

V = [float(record.MeanVoltage) for record in result_set]
Date = [str(record.time) for record in result_set]


dates = [datetime.datetime.strptime(record, '%Y-%m-%d %H:%M:%S') for record in Date]
dates = matplotlib.dates.date2num(dates)

fig, ax = plt.subplots()
ax.plot_date(dates, V)
plt.grid(True)
plt.show()

And the result is Plot

Thank you in advance

Edit:

I have fixed the issue by adding these lines:

plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y %H:%M:%S' ))

plt.gcf().autofmt_xdate()

However, now I want to pass the plot to a web server using the mpld3 plugin:

mpld3.plugins.get_plugins(fig)
mpld3.fig_to_html(fig)
mpld3.show()

While, without the plugin, the plot appears just fine, with the dates in the x axis, with the plugin it seems like it can't parse this line

plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y %H:%M:%S' ))

into the html code and as a result the x axis label appears in unix time. Anyone knows what's wrong with the plugin?

Answer 1

The problem is the large number of points (One every second is a bundle!). If you try to plot each point as a circle you will have these problems. But it is easily solved by changing it to a line graph, changing:

ax.plot_date(dates, V, '-') # Where '-' means a line plot

For example:

# some sample data
x = np.linspace(0.1, np.pi, 86000)
y = np.cos(x)**2 * np.log(x)
plt.plot(x, y, 'o')

plt.plot(x, y, '-')