How to index(elasticsearch) a specific collection of a specific db of mongodb using transporter?

Question

I want to index only specific mongodb collection in elastic search using transporter. Mongodb server has multiple databases having multiple collections. I dont want to index entire database.

Answer 1

If you only want to copy documents in the my_instance collection, use the namespace filter in the pipeline, i.e.

var source = mongodb({
    "uri": "mongodb://172.31.38.157:27017/STOREDB",
    "timeout": "30s",
    "read_preference": "Primary"
})

var sink = elasticsearch({
    "uri": "http://172.31.38.157:9200/INDEX_NAME" 
})

t.Source("source", source, "/^my_instance$").Save("sink", sink, "/.*/")

---

Additionally if you want only specific data from selected collections, the collection_filters setting gets converted to a mongodb query. The commented out example below would only include documents where the i field had a value greater than 10.

var source = mongodb({
  "uri": "mongodb://127.0.0.1:27017/test"
  // "timeout": "30s",
  // "tail": false,
  // "ssl": false,
  // "cacerts": ["/path/to/cert.pem"],
  // "wc": 1,
  // "fsync": false,
  // "bulk": false,
  // "collection_filters": "{\"foo\": {\"i\": {\"$gt\": 10}}}"
})