AMI js segmentation label map overlay class color is not getting displayed

Question

I am referring this example. It is on vanilla javascript.

Imported everything to as an angular provider service in angular 7.3.8 with AMI version 0.32.0 (three 0.99.0).

Using the same test examples as in the link above. Original scan with segmentation map is getting loaded but the class color is not loaded.

Please refer the code below to understand the changes:

loadAMIFile()

Not loading files using loader.load() as in original version but only parsing as loader.load() is not handling non-Dicom files properly. Hence, getting the required data to pass to loader.parse() from files using the below script.

  loadAMIFile(files) {  
    const ext = files[0].name.split('.').pop();
        this.readAsDataURL(files[0]).then((dataUrl) => {
          this.readAsArrayBuffer(files[0]).then((arrayBuffer) => {
            const resp = {
                buffer: arrayBuffer,
                extension: ext,
                filename: files[0].name,
                gzcompressed: null,
                pathname: "",
                query: "",
                url: dataUrl + '?filename=' +  files[0].name
              };
            that.amiProvider.toParse(resp);
          }).catch(error => {
            console.log('oops... something went wrong...');
            console.log(error);
          });
        }).catch(error => {
          console.log('oops... something went wrong...');
          console.log(error);
        });
     }

amiProvider.toParse()

Here the data gets parsed and loaded by calling handleSeries().

For simplicity, I am loading the same file twice, whereas in reality, first, the original scan will get loaded and on user request, it's segmentation map should get loaded. In this case, I am loading the labelmap file but it does not show respective class colors. Everything else is as similar as in the jsfiddle link.

toParse(toBeParsedDict) {
    this.loader = new this.LoadersVolume(this.threeD);
    const toBeParsedDictArr = [toBeParsedDict, toBeParsedDict];
    const promises = [];
    const that = this;
    toBeParsedDictArr.forEach(toBeParsedDict_ => {
      // To avoid shallow copy.
      const copied = {...toBeParsedDict_};
      promises.push(that.loader.parse(copied));
    });

    Promise.all(promises).then(data => {
      console.log(data);
      this.handleSeries(data);
    }).catch(err => console.log(err));

I have also tried with vanilla js (copying entire project from the link above to local system) and the same issue occurred there as well.

Does anyone face such issue before?

NOTE: Original fiddle version has used AMI 0.0.17 ( ThreeJS 86) which is not working at all locally i.e. not loading test files at all. So both versions have been modified. Is there any problem with the version of the modules I am using.

Answer 1

Okay, So after 1 month of headache and testing all AMI versions, got to know that in the angular environment only AMI 0.0.17 is giving a color map.

Though It does not support MGZ/MGH files, I angularised and integrated MGH parser from the latest AMI (AMI 0.32.0) into my project to have support for MGZ/MGH file parsing.

Now it works like a charm.

ami.component.ts

async loadAMIFile(files, hasSegmentationMap = false) {
    const ext0 = files[0].name.split('.').pop();
    const arrayBuffer0 = await this.readAsArrayBuffer(files[0]);
    const resp0 = {
      buffer: arrayBuffer0,
      extension: ext0,
      filename: files[0].name,
      gzcompressed: null,
      pathname: files[0].name,
      query: 'filename=' + files[0].name,
      url: files[0].name
    };

    if (hasSegmentationMap) {
      const ext1 = files[1].name.split('.').pop();
      const arrayBuffer1 = await this.readAsArrayBuffer(files[1]);
      const resp1 = {
        buffer: arrayBuffer1,
        extension: ext1,
        filename: files[1].name,
        gzcompressed: null,
        pathname: files[1].name,
        query: 'filename=' + files[1].name,
        url: files[1].name
      };

      this.amiProvider.toParse([resp0, resp1], hasSegmentationMap);
    } else {
      this.amiProvider.toParse([resp0], hasSegmentationMap);
    }
  }

ami.provider.ts

toParse(toBeParsedDictArr, hasSegmentationMap = false) {
    this.loader = new this.LoadersVolume(this.threeD);
    const promises = [];
    toBeParsedDictArr.forEach(toBeParsedDict_ => {
      const copied = {...toBeParsedDict_};
      if (['mgz', 'mgh'].includes(copied.extension)) {
        const data = this._parseMGH(copied);
        promises.push(this.loader.parse(data));
      }
      promises.push(this.loader.parse(copied));
    });

    Promise.all(promises).then(data => {
      this.handleSeries(data, hasSegmentationMap);
    }).catch(err => console.log(err));
  }

  _parseMGH(data) {
    // unzip if extension is '.mgz'
    if (data.extension === 'mgz') {
      data.gzcompressed = false;  // true
      data.extension = 'mgh';
      data.filename = data.filename.split('.')[0] + '.' + data.extension;
      const decompressedData = PAKO.inflate(data.buffer);
      data.buffer = decompressedData.buffer;
    } else {
      data.gzcompressed = false;
    }

    const mghVolumeParser = new ParsersMgh(data, 0, this.THREE);
    data.volumeParser = mghVolumeParser;
    return data;
  }
}

mghParser.helper.ts

import {Inject} from '@angular/core';
import {VolumeParser} from './volumeParser.helper';

/**
 * @module parsers/mgh
 */

export class ParsersMgh extends VolumeParser {

  // https://github.com/freesurfer/freesurfer/
  // See include/mri.h
  MRI_UCHAR = 0;
  MRI_INT = 1;
  MRI_LONG = 2;
  MRI_FLOAT = 3;
  MRI_SHORT = 4;
  MRI_BITMAP = 5;
  MRI_TENSOR = 6;
  MRI_FLOAT_COMPLEX = 7;
  MRI_DOUBLE_COMPLEX = 8;
  MRI_RGB = 9;
  // https://github.com/freesurfer/freesurfer/
  // See include/tags.h
  TAG_OLD_COLORTABLE = 1;
  TAG_OLD_USEREALRAS = 2;
  TAG_CMDLINE = 3;
  TAG_USEREALRAS = 4;
  TAG_COLORTABLE = 5;
  TAG_GCAMORPH_GEOM = 10;
  TAG_GCAMORPH_TYPE = 11;
  TAG_GCAMORPH_LABELS = 12;
  TAG_OLD_SURF_GEOM = 20;
  TAG_SURF_GEOM = 21;
  TAG_OLD_MGH_XFORM = 30;
  TAG_MGH_XFORM = 31;
  TAG_GROUP_AVG_SURFACE_AREA = 32;
  TAG_AUTO_ALIGN = 33;
  TAG_SCALAR_DOUBLE = 40;
  TAG_PEDIR = 41;
  TAG_MRI_FRAME = 42;
  TAG_FIELDSTRENGTH = 43;


  public _id;
  public _url;
  public _buffer;
  public _bufferPos;
  public _dataPos;
  public _pixelData;
  // Default MGH Header as described at:
  // https://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/MghFormat
  // Image "header" with default values
  public _version;
  public _width;
  public _height;
  public _depth;
  public _nframes;
  public _type; // 0-UCHAR, 4-SHORT, 1-INT, 3-FLOAT
  public _dof;
  public _goodRASFlag; // True: Use directional cosines, false assume CORONAL
  public _spacingXYZ;
  public _Xras;
  public _Yras;
  public _Zras;
  public _Cras;
  // Image "footer"
  public _tr; // ms
  public _flipAngle; // radians
  public _te; // ms
  public _ti; // ms
  public _fov; // from doc: IGNORE THIS FIELD (data is inconsistent)
  public _tags; // Will then contain variable length char strings
  // Other misc
  public _origin;
  public _imageOrient;
  // Read header
  // ArrayBuffer in data.buffer may need endian swap
  // public _buffer = data.buffer;
  // public _version;
  public _swapEndian;

  // public _width;
  // public _height;
  // public _depth; // AMI calls this frames
  // public _nframes;
  // public _type;
  // public _dof;
  // public _goodRASFlag;
  // public _spacingXYZ;
  // public _Xras;
  // public _Yras;
  // public _Zras;
  // public _Cras;
  // @Inject('THREE') public THREE;
  public dataSize;
  public vSize;

  constructor(data, id, @Inject('THREE') public THREE) {
    super();
    /**
     * @member
     * @type {arraybuffer}
     */
    this._id = id;
    this._url = data.url;
    this._buffer = null;
    this._bufferPos = 0;
    this._dataPos = 0;
    this._pixelData = null;
    // Default MGH Header as described at:
    // https://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/MghFormat
    // Image "header" with default values
    this._version = 1;
    this._width = 0;
    this._height = 0;
    this._depth = 0;
    this._nframes = 0;
    this._type = this.MRI_UCHAR; // 0-UCHAR, 4-SHORT, 1-INT, 3-FLOAT
    this._dof = 0;
    this._goodRASFlag = 0; // True: Use directional cosines, false assume CORONAL
    this._spacingXYZ = [1, 1, 1];
    this._Xras = [-1, 0, 0];
    this._Yras = [0, 0, -1];
    this._Zras = [0, 1, 0];
    this._Cras = [0, 0, 0];
    // Image "footer"
    this._tr = 0; // ms
    this._flipAngle = 0; // radians
    this._te = 0; // ms
    this._ti = 0; // ms
    this._fov = 0; // from doc: IGNORE THIS FIELD (data is inconsistent)
    this._tags = []; // Will then contain variable length char strings
    // Other misc
    this._origin = [0, 0, 0];
    this._imageOrient = [0, 0, 0, 0, 0, 0];
    // Read header
    // ArrayBuffer in data.buffer may need endian swap
    this._buffer = data.buffer;
    this._version = this._readInt();
    this._swapEndian = false;
    if (this._version === 1) {
      // Life is good
    } else if (this._version === 16777216) {
      this._swapEndian = true;
      this._version = this._swap32(this._version);
    } else {
      const error = new Error('MGH/MGZ parser: Unknown Endian.  Version reports: ' + this._version);
      throw error;
    }
    this._width = this._readInt();
    this._height = this._readInt();
    this._depth = this._readInt(); // AMI calls this frames
    this._nframes = this._readInt();
    this._type = this._readInt();
    this._dof = this._readInt();
    this._goodRASFlag = this._readShort();
    this._spacingXYZ = this._readFloat(3);
    this._Xras = this._readFloat(3);
    this._Yras = this._readFloat(3);
    this._Zras = this._readFloat(3);
    this._Cras = this._readFloat(3);
    this._bufferPos = 284;
    const dataSize = this._width * this._height * this._depth * this._nframes;
    const vSize = this._width * this._height * this._depth;
    switch (this._type) {
      case this.MRI_UCHAR:
        this._pixelData = this._readUChar(dataSize);
        break;
      case this.MRI_INT:
        this._pixelData = this._readInt(dataSize);
        break;
      case this.MRI_FLOAT:
        this._pixelData = this._readFloat(dataSize);
        break;
      case this.MRI_SHORT:
        this._pixelData = this._readShort(dataSize);
        break;
      default:
        throw Error('MGH/MGZ parser: Unknown _type.  _type reports: ' + this._type);
    }
    this._tr = this._readFloat(1);
    this._flipAngle = this._readFloat(1);
    this._te = this._readFloat(1);
    this._ti = this._readFloat(1);
    this._fov = this._readFloat(1);
    const enc = new TextDecoder();
    let t = this._tagReadStart();
    while (t[0] !== undefined) {
      const tagType = t[0];
      const tagLen = t[1];
      let tagValue;
      switch (tagType) {
        case this.TAG_OLD_MGH_XFORM:
        case this.TAG_MGH_XFORM:
          tagValue = this._readChar(tagLen);
          break;
        default:
          tagValue = this._readChar(tagLen);
      }
      tagValue = enc.decode(tagValue);
      this._tags.push({tagType: tagType, tagValue: tagValue});
      // read for next loop
      t = this._tagReadStart();
    }
    // detect if we are in a right handed coordinate system
    const first = new this.THREE.Vector3().fromArray(this._Xras);
    const second = new this.THREE.Vector3().fromArray(this._Yras);
    const crossFirstSecond = new this.THREE.Vector3().crossVectors(first, second);
    const third = new this.THREE.Vector3().fromArray(this._Zras);
    if (crossFirstSecond.angleTo(third) > Math.PI / 2) {
      this._rightHanded = false;
    }
    // - sign to move to LPS space
    this._imageOrient = [
      -this._Xras[0],
      -this._Xras[1],
      this._Xras[2],
      -this._Yras[0],
      -this._Yras[1],
      this._Yras[2],
    ];
    // Calculate origin
    const fcx = this._width / 2.0;
    const fcy = this._height / 2.0;
    const fcz = this._depth / 2.0;
    for (let ui = 0; ui < 3; ++ui) {
      this._origin[ui] =
        this._Cras[ui] -
        (this._Xras[ui] * this._spacingXYZ[0] * fcx +
          this._Yras[ui] * this._spacingXYZ[1] * fcy +
          this._Zras[ui] * this._spacingXYZ[2] * fcz);
    }
    // - sign to move to LPS space
    this._origin = [-this._origin[0], -this._origin[1], this._origin[2]];
  }

  seriesInstanceUID() {
    // use filename + timestamp..?
    return this._url;
  }

  numberOfFrames() {
    // AMI calls Z component frames, not T (_nframes)
    return this._depth;
  }

  sopInstanceUID(frameIndex = 0) {
    return frameIndex;
  }

  rows(frameIndex = 0) {
    return this._width;
  }

  columns(frameIndex = 0) {
    return this._height;
  }

  pixelType(frameIndex = 0) {
    // Return: 0 integer, 1 float
    switch (this._type) {
      case this.MRI_UCHAR:
      case this.MRI_INT:
      case this.MRI_SHORT:
        return 0;
      case this.MRI_FLOAT:
        return 1;
      default:
        throw Error('MGH/MGZ parser: Unknown _type.  _type reports: ' + this._type);
    }
  }

  bitsAllocated(frameIndex = 0) {
    switch (this._type) {
      case this.MRI_UCHAR:
        return 8;
      case this.MRI_SHORT:
        return 16;
      case this.MRI_INT:
      case this.MRI_FLOAT:
        return 32;
      default:
        throw Error('MGH/MGZ parser: Unknown _type.  _type reports: ' + this._type);
    }
  }

  pixelSpacing(frameIndex = 0) {
    return this._spacingXYZ;
  }

  imageOrientation(frameIndex = 0) {
    return this._imageOrient;
  }

  imagePosition(frameIndex = 0) {
    return this._origin;
  }

  extractPixelData(frameIndex = 0) {
    const sliceSize = this._width * this._height;
    return this._pixelData.slice(frameIndex * sliceSize, (frameIndex + 1) * sliceSize);
  }

  // signed int32
  _readInt(len = 1) {
    const tempBuff = new DataView(this._buffer.slice(this._bufferPos, this._bufferPos + len * 4));
    this._bufferPos += len * 4;
    let v;
    if (len === 1) {
      v = tempBuff.getInt32(0, this._swapEndian);
    } else {
      v = new Int32Array(len);
      for (let i = 0; i < len; i++) {
        v[i] = tempBuff.getInt32(i * 4, this._swapEndian);
      }
    }
    return v;
  }

  // signed int16
  _readShort(len = 1) {
    const tempBuff = new DataView(this._buffer.slice(this._bufferPos, this._bufferPos + len * 2));
    this._bufferPos += len * 2;
    let v;
    if (len === 1) {
      v = tempBuff.getInt16(0, this._swapEndian);
    } else {
      v = new Int16Array(len);
      for (let i = 0; i < len; i++) {
        v[i] = tempBuff.getInt16(i * 2, this._swapEndian);
      }
    }
    return v;
  }

  // signed int64
  _readLong(len = 1) {
    const tempBuff = new DataView(this._buffer.slice(this._bufferPos, this._bufferPos + len * 8));
    this._bufferPos += len * 8;
    const v = new Uint16Array(len);
    for (let i = 0; i < len; i++) {
      /* DataView doesn't have Int64.
       * This work around based off Scalajs
       * (https://github.com/scala-js/scala-js/blob/master/library/src/main/scala/scala/scalajs/js/typedarray/DataViewExt.scala)
       * v[i]=tempBuff.getInt64(i*8,this._swapEndian);
       */
      let shiftHigh = 0;
      let shiftLow = 0;
      if (this._swapEndian) {
        shiftHigh = 4;
      } else {
        shiftLow = 4;
      }
      const high = tempBuff.getInt32(i * 8 + shiftHigh, this._swapEndian);
      let low = tempBuff.getInt32(i * 8 + shiftLow, this._swapEndian);
      if (high !== 0) {
        console.log('Unable to read Int64 with high word: ' + high + 'low word: ' + low);
        low = undefined;
      }
      v[i] = low;
    }
    if (len === 0) {
      return undefined;
    } else if (len === 1) {
      return v[0];
    } else {
      return v;
    }
  }

  // signed int8
  _readChar(len = 1) {
    const tempBuff = new DataView(this._buffer.slice(this._bufferPos, this._bufferPos + len));
    this._bufferPos += len;
    let v;
    if (len === 1) {
      v = tempBuff.getInt8(0); // , this._swapEndian
    } else {
      v = new Int8Array(len);
      for (let i = 0; i < len; i++) {
        v[i] = tempBuff.getInt8(i); // , this._swapEndian
      }
    }
    return v;
  }

  // unsigned int8
  _readUChar(len = 1) {
    const tempBuff = new DataView(this._buffer.slice(this._bufferPos, this._bufferPos + len));
    this._bufferPos += len;
    let v;
    if (len === 1) {
      v = tempBuff.getUint8(0);  // , this._swapEndian
    } else {
      v = new Uint8Array(len);
      for (let i = 0; i < len; i++) {
        v[i] = tempBuff.getUint8(i); // , this._swapEndian
      }
    }
    return v;
  }

  // float32
  _readFloat(len = 1) {
    const tempBuff = new DataView(this._buffer.slice(this._bufferPos, this._bufferPos + len * 4));
    this._bufferPos += len * 4;
    let v;
    if (len === 1) {
      v = tempBuff.getFloat32(0, this._swapEndian);
    } else {
      v = new Float32Array(len);
      for (let i = 0; i < len; i++) {
        v[i] = tempBuff.getFloat32(i * 4, this._swapEndian);
      }
    }
    return v;
  }

  _tagReadStart() {
    if (this._bufferPos >= this._buffer.byteLength) {
      return [undefined, undefined];
    }
    let tagType = this._readInt();
    let tagLen;
    switch (tagType) {
      case this.TAG_OLD_MGH_XFORM:
        tagLen = this._readInt();
        tagLen -= 1;
        break;
      case this.TAG_OLD_SURF_GEOM:
      case this.TAG_OLD_USEREALRAS:
      case this.TAG_OLD_COLORTABLE:
        tagLen = 0;
        break;
      default:
        tagLen = this._readLong();
    }
    if (tagLen === undefined) {
      tagType = undefined;
    }
    return [tagType, tagLen];
  }
}

volumeParser.helper.ts

/** * Imports ***/
// import ParsersVolume from './parsers.volume';
// import * as THREE from 'three';


/**
 * @module parsers/volume
 */
export class VolumeParser {
  public _rightHanded;

  constructor() {
    this._rightHanded = true;
  }

  pixelRepresentation() {
    return 0;
  }

  pixelPaddingValue(frameIndex = 0) {
    return null;
  }

  modality() {
    return 'unknown';
  }

  segmentationType() {
    return 'unknown';
  }

  segmentationSegments() {
    return [];
  }

  referencedSegmentNumber(frameIndex) {
    return -1;
  }

  rightHanded() {
    return this._rightHanded;
  }

  spacingBetweenSlices() {
    return null;
  }

  numberOfChannels() {
    return 1;
  }

  sliceThickness() {
    return null;
  }

  dimensionIndexValues(frameIndex = 0) {
    return null;
  }

  instanceNumber(frameIndex = 0) {
    return frameIndex;
  }

  windowCenter(frameIndex = 0) {
    return null;
  }

  windowWidth(frameIndex = 0) {
    return null;
  }

  rescaleSlope(frameIndex = 0) {
    return 1;
  }

  rescaleIntercept(frameIndex = 0) {
    return 0;
  }

  ultrasoundRegions(frameIndex = 0) {
    return [];
  }

  frameTime(frameIndex = 0) {
    return null;
  }

  _decompressUncompressed() {
  }

  // http://stackoverflow.com/questions/5320439/how-do-i-swap-endian-ness-byte-order-of-a-variable-in-javascript
  _swap16(val) {
    return ((val & 0xff) << 8) | ((val >> 8) & 0xff);
  }

  _swap32(val) {
    return (
      ((val & 0xff) << 24) | ((val & 0xff00) << 8) | ((val >> 8) & 0xff00) | ((val >> 24) & 0xff)
    );
  }

  invert() {
    return false;
  }

  /**
   * Get the transfer syntax UID.
   * @return {*}
   */
  transferSyntaxUID() {
    return 'no value provided';
  }

  /**
   * Get the study date.
   * @return {*}
   */
  studyDate() {
    return 'no value provided';
  }

  /**
   * Get the study desciption.
   * @return {*}
   */
  studyDescription() {
    return 'no value provided';
  }

  /**
   * Get the series date.
   * @return {*}
   */
  seriesDate() {
    return 'no value provided';
  }

  /**
   * Get the series desciption.
   * @return {*}
   */
  seriesDescription() {
    return 'no value provided';
  }

  /**
   * Get the patient ID.
   * @return {*}
   */
  patientID() {
    return 'no value provided';
  }

  /**
   * Get the patient name.
   * @return {*}
   */
  patientName() {
    return 'no value provided';
  }

  /**
   * Get the patient age.
   * @return {*}
   */
  patientAge() {
    return 'no value provided';
  }

  /**
   * Get the patient birthdate.
   * @return {*}
   */
  patientBirthdate() {
    return 'no value provided';
  }

  /**
   * Get the patient sex.
   * @return {*}
   */
  patientSex() {
    return 'no value provided';
  }

  /**
   * Get min/max values in array
   *
   * @param {*} pixelData
   *
   * @return {*}
   */
  minMaxPixelData(pixelData = []) {
    const minMax = [Number.POSITIVE_INFINITY, Number.NEGATIVE_INFINITY];
    const numPixels = pixelData.length;
    for (let index = 0; index < numPixels; index++) {
      const spv = pixelData[index];
      minMax[0] = Math.min(minMax[0], spv);
      minMax[1] = Math.max(minMax[1], spv);
    }
    return minMax;
  }
}

And a little bit change in parse() of AMI 0.0.17 library to accommodate MGH parser. In the future, with the support of the latest AMI with color map integrated properly, the code will work without the need for any change in the library again.

      var volumeParser = null;
        try {
          if (['mgh', 'mgz'].includes(response.extension)) {
            volumeParser = response.volumeParser;
          } else {
            var Parser = _this2._parser(data.extension);
            if (!Parser) {
              // emit 'parse-error' event
              _this2.emit('parse-error', {
                file: response.url,
                time: new Date(),
                error: data.filename + 'can not be parsed.'
              });
              reject(data.filename + ' can not be parsed.');
            }
            volumeParser = new Parser(data, 0);
          }