Artillery SocketIO Stream Throughput Test

Question

I want to run an artillery test that spins up n socket.io clients that all listen for periodic packets coming from the server. The idea is that the server is broadcasting json "frames" at a certain interval and I would like to see how quickly the client receives these "frames."

I generate the frame on the server and then broadcast it to all clients. The clients then have an identical local copy of the frame to check to see if they are equal when received. I'm doing this to ensure packet integrity so I can confidently use the size of the packet and the latency to measure throughput.

However, I'm not getting the behavior I'm looking for. Here is my server code, my artillery script, and the corresponding helping function.

server.js

// Dependencies
const express = require('express');
const http = require('http');
// note: socketIO uses JSON.stringify() natively to send and receive any objects
const socketIO = require('socket.io');
const app = express();
const server = http.Server(app);
const io = socketIO(server);
const port = 6969
app.set('port', port);

// keep time of session
let t0 = null
let command_sequence_number = 0

// Starts the server.
server.listen(port, function() {
    console.log('Starting server on port ', port);
  });

let num_items = 1000;
items = {}
for (let i = 0; i < num_items; i++) {
  items[`item${i}`] = [0, 0, 0, 0, 0];
}

/**
 * WebSocket handlers (every time a connection is made)
 * @param {Object} socket
 */
io.on('connection', function(socket) {
  console.log('new connection');

});

/**
* simulate data stream
*/
let frame_delay = 30; // milliseconds
setInterval(function() {
  io.sockets.emit('FRAME', items);
}, frame_delay);

artillery_test.yml

config:
    target: "http://localhost:6969"
    processor: "build_frame.js"
    phases:
      - duration: 10
        arrivalRate: 5
        name: Simple Test Phase
scenarios:
  - name: Simple Throughput Test Scenario
    engine: socketio # Enable the Socket.io engine
    flow:
        - function: "build_frame"      
        - loop:
          - emit:
              channel: "dummy"
              data: ""
              response:
                channel: "FRAME"
                data: "{{ items }}"
          count: 3600 #30frames/s * 120s = 3600 frames

build_frame.js

module.exports = { build_frame };

function build_frame (userContext, events, done) {
  let num_items = 1000;
  let items = {}
  for (let i = 0; i < num_items; i++) {
    itesm[`item${i}`] = [0, 0, 0, 0, 0];
  }
  // set the "items" variable for the virtual user to use in the subsequent action
  userContext.vars.items = items;
  return done();
  }

I know the way I'm "listening" for frames is pretty hacky. I don't think emitting a dummy message just to listen for an incoming frame is the way to go, but I would love to know how to just park the client in listening mode and then check the integrity of the incoming data every time a new packet arives.

Here's an example of my output well after the clients have been spun up: Report @ 01:40:08(-0700) 2020-09-02 Elapsed time: 30 minutes, 37 seconds Scenarios launched: 0 Scenarios completed: 0 Requests completed: 746 Mean response/sec: 37.49 Response time (msec): min: 1161.8 max: 1468.1 median: 1317.1 p95: 1388.5 p99: 1444.4 Codes: 0: 746

Warning: High CPU usage warning (pids: 2056). See https://artillery.io/docs/faq/#high-cpu-warnings for details.

Would love any tips!