This problem does not seem to be related to this one.
I am trying to execute a long-running database population job, but after a few hours it fails at seemingly random times. I am running JanusGraph 1.0.0 through the Docker image, and I'm using gremlinpython 3.5.8.
The code below attempts creating 10M vertices and about 90M edges. It uses to fold/coalesce/unfold pattern to only create them if they don't already exist. There is also no batching of operations, so this is extremely slow but is expected to work.
import os
import random
from tqdm import tqdm
from dotenv import load_dotenv
from gremlin_python.process.anonymous_traversal import traversal
from gremlin_python.process.graph_traversal import __, GraphTraversalSource
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
load_dotenv()
GRAPH_DB_HOST = os.getenv("GRAPH_DB_HOST")
GRAPH_DB_USER = os.getenv("GRAPH_DB_USER")
GRAPH_DB_PASSWORD = os.getenv("GRAPH_DB_PASSWORD")
GREMLIN_SEVER_PORT = os.getenv("GREMLIN_SEVER_PORT")
GRAPH_DB_URL = f"ws://{GRAPH_DB_HOST}:{GREMLIN_SEVER_PORT}/gremlin"
g: GraphTraversalSource = traversal().withRemote(
DriverRemoteConnection(GRAPH_DB_URL, 'g',
username=GRAPH_DB_USER,
password=GRAPH_DB_PASSWORD)
)
vertices_to_add = 10_000_000
my_traversal = g
progress_bar = tqdm(total=vertices_to_add)
progress_bar.set_description("Adding vertices")
# add vertices in chunks of size chunk_size
for i in range(vertices_to_add):
g.V().has('person', 'output_id', i) \
.fold() \
.coalesce(
__.unfold(),
__.addV('person').property('output_id', i)
).next()
# create vertices from the last 10 people
if i > 10:
for j in range(1, 11):
g.V().has('person', 'output_id', i - j) \
.inE('my_edge').where(__.outV().has('person', 'output_id', i)) \
.fold() \
.coalesce(__.unfold(),
__.V().has('person', 'output_id', i)
.addE('my_edge')
.from_(__.V().has('person', 'output_id', i - j))
.property('value', random.randint(1, 1000))
).next()
progress_bar.update(1)
progress_bar.close()
ERROR:root:
Received error message '{'requestId': '0127041d-376c-4754-802d-a68549418d23', 'status': {'message': 'Could not execute operation due to backend exception', 'code': 500, 'attributes': {'stackTrace': 'org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception\n\tat org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)\n\tat org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:521)\n\tat org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:284)\n\tat org.janusgraph.diskstorage.BackendTransaction.multiThreadedEdgeStoreMultiQuery(BackendTransaction.java:355)\n\tat org.janusgraph.diskstorage.BackendTransaction.edgeStoreMultiQuery(BackendTransaction.java:313)\n\tat org.janusgraph.graphdb.database.StandardJanusGraph.edgeMultiQuery(StandardJanusGraph.java:568)\n\tat org.janusgraph.graphdb.transaction.StandardJanusGraphTx.lambda$executeMultiQuery$4(StandardJanusGraphTx.java:1252)\n\tat org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:113)\n\tat org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:105)\n\tat org.janusgraph.graphdb.transaction.StandardJanusGraphTx.executeMultiQuery(StandardJanusGraphTx.java:1252)\n\tat org.janusgraph.graphdb.query.vertex.MultiVertexCentricQueryBuilder.execute(MultiVertexCentricQueryBuilder.java:121)\n\tat org.janusgraph.graphdb.query.vertex.MultiVertexCentricQueryBuilder.properties(MultiVertexCentricQueryBuilder.java:179)\n\tat org.janusgraph.graphdb.query.vertex.MultiVertexCentricQueryBuilder.preFetch(MultiVertexCentricQueryBuilder.java:185)\n\tat org.janusgraph.graphdb.tinkerpop.optimize.step.fetcher.HasStepBatchFetcher.prefetchNextPropertiesBatch(HasStepBatchFetcher.java:147)\n\tat org.janusgraph.graphdb.tinkerpop.optimize.step.fetcher.HasStepBatchFetcher.prefetchNextBatch(HasStepBatchFetcher.java:112)\n\tat org.janusgraph.graphdb.tinkerpop.optimize.step.fetcher.MultiQueriableStepBatchFetcher.fetchData(MultiQueriableStepBatchFetcher.java:96)\n\tat org.janusgraph.graphdb.tinkerpop.optimize.step.JanusGraphHasStep.filter(JanusGraphHasStep.java:104)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:41)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:155)\n\tat org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)\n\tat org.apache.tinkerpop.gremlin.process.traversal.util.TraversalUtil.test(TraversalUtil.java:141)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.filter.TraversalFilterStep.filter(TraversalFilterStep.java:63)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:41)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:155)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.hasNext(ExpandableStepIterator.java:47)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.util.ReducingBarrierStep.processAllStarts(ReducingBarrierStep.java:110)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.util.ReducingBarrierStep.processNextStart(ReducingBarrierStep.java:140)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:155)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:55)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.map.FlatMapStep.processNextStart(FlatMapStep.java:48)\n\tat org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:155)\n\tat org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:192)\n\tat org.apache.tinkerpop.gremlin.server.util.TraverserIterator.fillBulker(TraverserIterator.java:63)\n\tat org.apache.tinkerpop.gremlin.server.util.TraverserIterator.hasNext(TraverserIterator.java:50)\n\tat org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor.handleIterator(TraversalOpProcessor.java:350)\n\tat org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor.lambda$iterateBytecodeTraversal$0(TraversalOpProcessor.java:223)\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent exception while executing backend operation EdgeStoreQuery\n\tat org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:79)\n\tat org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52)\n\t... 41 more\nCaused by: com.sleepycat.je.EnvironmentFailureException: (JE 18.3.12) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.12) /var/lib/janusgraph/data Latch timeout. BIN4 currentThread: Thread[JEEvictor,5,main] currentTime: 1704951452786 exclusiveOwner: Thread[Backend[30],5,main] UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed.\n\tat com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230)\n\tat com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835)\n\tat com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844)\n\tat com.sleepycat.je.Environment.checkOpen(Environment.java:2697)\n\tat com.sleepycat.je.Database.checkEnv(Database.java:2413)\n\tat com.sleepycat.je.Database.openCursor(Database.java:939)\n\tat org.janusgraph.diskstorage.berkeleyje.BerkeleyJETx.openCursor(BerkeleyJETx.java:63)\n\tat org.janusgraph.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.openCursor(BerkeleyJEKeyValueStore.java:95)\n\tat org.janusgraph.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.getSlice(BerkeleyJEKeyValueStore.java:155)\n\tat org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreAdapter.getSlice(OrderedKeyValueStoreAdapter.java:88)\n\tat org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:82)\n\tat org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:287)\n\tat org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:284)\n\tat org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66)\n\t... 42 more\nCaused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.12) /var/lib/janusgraph/data Latch timeout. BIN4 currentThread: Thread[JEEvictor,5,main] currentTime: 1704951452786 exclusiveOwner: Thread[Backend[30],5,main] UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed.\n\tat com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:459)\n\tat com.sleepycat.je.latch.LatchSupport.handleTimeout(LatchSupport.java:211)\n\tat com.sleepycat.je.latch.LatchImpl.acquireExclusive(LatchImpl.java:63)\n\tat com.sleepycat.je.latch.LatchImpl.acquireShared(LatchImpl.java:107)\n\tat com.sleepycat.je.tree.IN.latchShared(IN.java:563)\n\tat com.sleepycat.je.tree.Tree.latchChildShared(Tree.java:353)\n\tat com.sleepycat.je.tree.Tree.search(Tree.java:2255)\n\tat com.sleepycat.je.tree.Tree.search(Tree.java:2152)\n\tat com.sleepycat.je.dbi.CursorImpl.searchExact(CursorImpl.java:1972)\n\tat com.sleepycat.je.dbi.CursorImpl.searchExact(CursorImpl.java:1935)\n\tat com.sleepycat.je.dbi.DbTree.getDb(DbTree.java:1708)\n\tat com.sleepycat.je.dbi.DbTree.getDb(DbTree.java:1659)\n\tat com.sleepycat.je.evictor.Evictor$DbCache.getDb(Evictor.java:3394)\n\tat com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:2394)\n\tat com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:2262)\n\tat com.sleepycat.je.evictor.Evictor$BackgroundEvictTask.run(Evictor.java:3297)\n\t... 3 more\n', 'exceptions': ['org.janusgraph.core.JanusGraphException', 'org.janusgraph.diskstorage.PermanentBackendException', 'com.sleepycat.je.EnvironmentFailureException', 'com.sleepycat.je.EnvironmentFailureException']}}, 'result': {'data': None, 'meta': {}}}'
With results dictionary '{'0127041d-376c-4754-802d-a68549418d23': <gremlin_python.driver.resultset.ResultSet object at 0x7f28233dd790>}'
Traceback (most recent call last):
File "/app/src/graph_populate.py", line 432, in <module>
else:
File "/app/src/graph_populate.py", line 184, in populate_outputs_onebyone_getorcreate
raise e
File "/app/src/graph_populate.py", line 177, in populate_outputs_onebyone_getorcreate
).next()
^^^^^^
File "/usr/local/lib/python3.12/site-packages/gremlin_python/process/traversal.py", line 88, in next
return self.__next__()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/gremlin_python/process/traversal.py", line 47, in __next__
self.traversal_strategies.apply_strategies(self)
File "/usr/local/lib/python3.12/site-packages/gremlin_python/process/traversal.py", line 548, in apply_strategies
traversal_strategy.apply(traversal)
File "/usr/local/lib/python3.12/site-packages/gremlin_python/driver/remote_connection.py", line 80, in apply
remote_traversal = self.remote_connection.submit(traversal.bytecode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/gremlin_python/driver/driver_remote_connection.py", line 103, in submit
results = result_set.all().result()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.12/site-packages/gremlin_python/driver/resultset.py", line 90, in cb
f.result()
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/gremlin_python/driver/connection.py", line 90, in _receive
status_code = self._protocol.data_received(data, self._results)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/gremlin_python/driver/protocol.py", line 133, in data_received
raise GremlinServerError(message['status'])
gremlin_python.driver.protocol.GremlinServerError: 500: Could not execute operation due to backend exception
It should be mentioned that I have added additional configuration options in my docker-compose.yml and Dockerfile, mainly regarding memory, as an attempt at reducing issues with large, long-running operations, and to increase speed.
My docker-compose.yml file is as follows:
btc_janus:
build:
context: ./janusgraph/
container_name: btc_janus
shm_size: 2g
ports:
- "${JANUSGRAPH_PORT:-8182}:${JANUSGRAPH_PORT:-8182}"
- "8484:8184"
- "8081:8081"
networks:
- btc-network
volumes:
- ${GRAPH_DB_FOLDER:-./data/btc_janus}:/var/lib/janusgraph
healthcheck:
test: ["CMD", "bin/gremlin.sh", "-e", "scripts/remote-connect.groovy"]
interval: 10s
timeout: 60s
retries: 4
environment:
janusgraph.storage.backend: berkeleyje
janusgraph.tx.log-tx: true
janusgraph.tx.max-commit-time: 100000000000
And the following Dockerfile:
# ./janusgraph/Dockerfile
FROM janusgraph/janusgraph:latest
# "-Xmx8g -Xms8g" crashes the container, and "-XX:MaxPermSize=1024m" is not supported
ENV JAVA_OPTS=" -Xss4096k "
ENV JAVA_OPTIONS="-Xss4096k"
ENV JVM_OPTS="-Xss4096k"
# Copy the index creation script
COPY create_index.groovy /docker-entrypoint-initdb.d/create_index.groovy
For brevity, I have not included the JanusGraph server logs, but here is a snippet of them:
2024-01-10 21:37:32 at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:79) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 ... 41 more
2024-01-10 21:37:32 Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 18.3.12) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.12) /var/lib/janusgraph/data Latch timeout. BIN4 currentThread: Thread[JEEvictor,5,main] currentTime: 1704951452786 exclusiveOwner: Thread[Backend[30],5,main] UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed.
2024-01-10 21:37:32 at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.Database.checkEnv(Database.java:2413) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.Database.openCursor(Database.java:939) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.berkeleyje.BerkeleyJETx.openCursor(BerkeleyJETx.java:63) ~[janusgraph-berkeleyje-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.openCursor(BerkeleyJEKeyValueStore.java:95) ~[janusgraph-berkeleyje-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.getSlice(BerkeleyJEKeyValueStore.java:155) ~[janusgraph-berkeleyje-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreAdapter.getSlice(OrderedKeyValueStoreAdapter.java:88) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.keycolumnvalue.KCVSProxy.getSlice(KCVSProxy.java:82) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:287) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.BackendTransaction$1.call(BackendTransaction.java:284) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 ... 41 more
2024-01-10 21:37:32 Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.12) /var/lib/janusgraph/data Latch timeout. BIN4 currentThread: Thread[JEEvictor,5,main] currentTime: 1704951452786 exclusiveOwner: Thread[Backend[30],5,main] UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed.
2024-01-10 21:37:32 at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:459) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.latch.LatchSupport.handleTimeout(LatchSupport.java:211) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.latch.LatchImpl.acquireExclusive(LatchImpl.java:63) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.latch.LatchImpl.acquireShared(LatchImpl.java:107) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.tree.IN.latchShared(IN.java:563) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.tree.Tree.latchChildShared(Tree.java:353) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.tree.Tree.search(Tree.java:2255) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.tree.Tree.search(Tree.java:2152) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.dbi.CursorImpl.searchExact(CursorImpl.java:1972) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.dbi.CursorImpl.searchExact(CursorImpl.java:1935) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.dbi.DbTree.getDb(DbTree.java:1708) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.dbi.DbTree.getDb(DbTree.java:1659) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.evictor.Evictor$DbCache.getDb(Evictor.java:3394) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:2394) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:2262) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 at com.sleepycat.je.evictor.Evictor$BackgroundEvictTask.run(Evictor.java:3297) ~[je-18.3.12.jar:18.3.12]
2024-01-10 21:37:32 ... 3 more
2024-01-10 21:37:32 05:37:32 WARN org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor.lambda$iterateBytecodeTraversal$0 - Exception processing a Traversal on iteration for request [7ec2741e-ef77-4f96-9954-06295d0cb41d].
2024-01-10 21:37:32 org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
2024-01-10 21:37:32 at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:521) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:284) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.database.StandardJanusGraph.edgeQuery(StandardJanusGraph.java:558) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$null$0(SimpleVertexQueryProcessor.java:124) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:113) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:105) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:101) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.lambda$getBasicIterator$1(SimpleVertexQueryProcessor.java:124) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:73) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:124) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:80) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at com.google.common.collect.Iterables$5.iterator(Iterables.java:742) ~[guava-32.1.3-jre.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.SimpleVertexQueryProcessor.vertexIds(SimpleVertexQueryProcessor.java:103) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.executeIndividualVertices(BasicVertexCentricQueryBuilder.java:365) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.executeVertices(BasicVertexCentricQueryBuilder.java:359) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder$VertexConstructor.getResult(BasicVertexCentricQueryBuilder.java:266) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder$VertexConstructor.getResult(BasicVertexCentricQueryBuilder.java:262) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.execute(VertexCentricQueryBuilder.java:106) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.vertices(VertexCentricQueryBuilder.java:134) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:126) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:131) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:122) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.types.system.ImplicitKey.computeProperty(ImplicitKey.java:94) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.BasicVertexCentricQueryBuilder.executeImplicitKeyQuery(BasicVertexCentricQueryBuilder.java:236) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.vertex.VertexCentricQueryBuilder.properties(VertexCentricQueryBuilder.java:119) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.util.ElementHelper.getValues(ElementHelper.java:48) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.condition.PredicateCondition.evaluate(PredicateCondition.java:72) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.condition.And.evaluate(And.java:55) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.graph.GraphCentricQuery.matches(GraphCentricQuery.java:157) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.query.QueryProcessor.lambda$getFilterIterator$2(QueryProcessor.java:138) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at org.janusgraph.graphdb.util.CloseableIteratorUtils$1.computeNext(CloseableIteratorUtils.java:51) ~[janusgraph-core-1.0.0.jar:?]
2024-01-10 21:37:32 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) ~[guava-32.1.3-jre.jar:?]
2024-01-10 21:37:32 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) ~[guava-32.1.3-jre.jar:?]
I can't help but continually notice com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.12) /var/lib/janusgraph/data Latch timeout. BIN4 currentThread: Thread[JEEvictor,5,main] currentTime: 1704951452786 exclusiveOwner: Thread[Backend[30],5,main] UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed.. This seems to be the culprit of the issue.