How to stream a (big) log file?

Question

When automation testing is running I cannot connect to the testing computer and check the log file to check the progress of the testing because I would interrupt the testing. I can map the hdd and check the file this way but I want to add new feature to my sinatra application.

The application runs the tests and displays the test results so I want to add realtime stream of the log file via sintra. The log file could be even 2MB big so I guess it won't be a good idea to send the whole file every time an update to the log file was made although the server client communication will be done in 99% over LAN only. I would also like to have the latest last line from the log file at the top of a web browser.

Could someone suggest how to do that?

I can think of an ajax call made on regular basis that would pass to sinatra a number of the line that was received as a last one. And sinatra would return any update if available.

update

• Windows 7 64 bit
• ruby 1.9.3p194 (2012-04-20) [i386-mingw32]
• sinatra (1.3.3)
• sinatra-contrib (1.3.1)
• sinatra-reloader (1.0)

Answer 1

You don't say what type of OS your testing system uses, but if it's Linux or Mac OS you're ready to go. If not, and it's Windows, I'd really recommend installing a telnetd or ssh server, and a tail-type app.

SSH and/or Telnet are a lot more lightweight because they're basically just sending text, so they'll impact your testing system less than trying to stream a file via HTTP, especially with the solution you mentioned. Just open a session, tail -f the file, then start the test.

---

To implement a solution using Sinatra, I'd start with a little piece of code like:

#!/bin/env ruby

filepath = ARGV.shift
start_line, num_lines = ARGV.map(&:to_i)

File.foreach(filepath) do |li|
  case 
  when $. < start_line
    next
  when (start_line .. (start_line + num_lines)) === $.
    puts li 
  when $. > (start_line + num_lines)
    break 
  end
end

Save that to the disk as display_file_block.rb, and call it with the parameters:

path/to/file start_line lines_to_display

Where:

• path/to/file is obvious.
• start_line is the starting line in the file to display.
• lines_to_display is the number of lines to display.

Using those you can open a file to display, send a number of lines, starting at an offset.

In Sinatra, set up a request handler for a GET:

get '/tail' do
  path = params['path']
  start = params['start']
  count = params['count']
  `/path/to/display_file_block.rb #{ path } #{ start } #{ count }`
end

You'll possibly want to set the content-type for the response to 'text/plain'. The Sinatra site can show you how to do that.

Answer 2

Here is an example using "push", i.e. WebSockets:

Minimum Websocket Nodejs Tail Example

In my opinion AJAX polling is a duct-tape type of solution to this kind of problem. WebSockets is the way to go, and done correctly will have very little impact on system performance.

By googling "websockets sinatra" you can search for similar solutions for Sinatra, like the first hit for example:

https://github.com/simulacre/sinatra-websocket