How to get all message history from Hipchat for a room via the API?

Question

I was using the Hipchat API (v2) a bit today and ran into an odd issue where I was not able to really pull up all of the history for a room. It seemed as though when I queried a specific date, for example, it would only retrieve a fraction of the history for that date given. I had had plans to simply iterate across all of the dates for a Room to extract the history in a format that I could use, but ended up hitting this and am now unsure if it is really possible to pull out the history fully.

I realize that this is a bit clunky. It is pulling the JSON as a string and then I have to form it into a hash so I know I'm not doing this as good as it could be done, but here is roughly what I quickly did just to test out the history method for the API:

api_token = "MY_TOKEN"

client = HipChat::Client.new(api_token, :api_version => 'v2')
history = client['ROOM_NAME'].history

history = JSON.parse(history)

history.each do |key, history|
  if history.is_a? Array
    history.each do |message|
      if message.is_a? Hash
        puts "#{message['from']['name']}: #{message['message']}"
      end
    end
  end
end

Obviously then the extension to that was to just curse through the dates in the desired range (using: client['ROOM_NAME'].history(:date => '2010-11-19', :timezone => 'PST')), but again, I was only getting a fraction of the history for the room. Are there some additional parameters that I'm missing for this to make it work as expected?

Answer 1

I got this working but it was a big pain.

Start by sending a query with the current time, in UTC, but without including the time zone, as the start date:

https://internal-hipchat-server/v2/room/2/history?reverse=false&date=2015-06-25T20:42:18.658439&max-results=1000&auth_token=XXX

This is very fiddly:

• If you specify just the current date, without a timezone, as documented in the API, it is interpreted as midnight last night and you only get messages from yesterday or older.
• If you try specifying tomorrow’s date instead, the response is 400 Bad Request This day has not yet come to pass.
• If you specify the time as 2015-06-25T20:42:18.658439+00:00, which is the format that times come in HipChat API responses, HipChat’s parser seems to fail and interpret it as midnight last night.

When you get the response back, take the oldest items.date property, strip the timezone, and resubmit the above URL with an updated date parameter:

https://internal-hipchat-server/v2/room/2/history?reverse=false&date=2015-06-17T19:56:34.533182&max-results=1000&auth_token=XXX

Be sure to include the microseconds, in case a notification posted multiple messages to the same room in the same second.

This will get you the next page of messages. Keep doing this until you get fewer than max-results messages back.

There is a start-index parameter I tried passing before I got the above working, and it will give you a few pages of results, with responses lacking a links.next property, but it won’t give you the full history. On a chatroom with 9166 messages in the history according to statistics.messages_sent, it only returned 3217 messages. So don’t use it. You can use statistics.messages_sent as a sanity check for whether you get all messages.

Oh yeah, and the last_active property in the /v2/room call cannot be trusted because it doesn’t update when notification messages are posted to the room.