Before anything, I know that I WANT to make my own database system, I already made one in the past that worked perfectly, I am simply looking to take on difficult challenges.
This Database system will be memory based. I am looking for methods to read / write data the most efficient way and for that I made a couple of benchmarks, here are the code/results:
// 1. JSON array system | 84.289ms
let testDataJson = new Array(10000000).fill({ id: 1, username: "TestUsername", password: "MyPassword1234", email : "[email protected]" })
testDataJson.push({ username: "FoundUsername" }); // Insert matching element
console.time("dataJson");
console.log(testDataJson.find(elem => elem.username === "FoundUsername");
console.timeEnd("dataJson");
// 2. TEXT array system | 86.041ms
let testDataText = new Array(10000000).fill(`1|TestUsername|MyPassword1234|[email protected]`.split("|"));
testDataText.push(`|FoundUsername||`.split("|"));
console.time("dataText");
console.log(testDataText.find(elem => elem[1] === "FoundUsername");
console.timeEnd("dataText");
Now onto the problematic part, I decided for this version to work using NodeJS Workers to spread the load accross multiples threads. For that I am spreading the array into x amount of workers and split the data array. I am then loading in each worker memory its designated data block and wait 3 seconds to let everything settle. I then run the search, and the result is, quite bad, here is my code
// PROGRAM CODE | 56.698ms
function loadThreads()
{
let dataChunks = fs.readFileSync("./db.txt", 'utf-8').split("\n");
let workersParts = chunkArrayInGroups(dataChunks, 2000000); // File is 10 000 000 Lines, resulting in 5 workers
for (let workerPart of workersParts)
{
const worker = new Worker("./dbTester_thread.js", { workerData: workerPart.join("\n") });
threads.push(worker);
}
}
function performSearch()
{
console.time("threadTest");
for (let thread of threads)
{
thread.on('message', (message) => {
console.timeEnd("threadTest")
});
thread.postMessage("HelloMan")
}
}
// THREAD CODE
const { parentPort, workerData } = require('node:worker_threads');
let data = workerData.split("\n").map(line => line.split(","));
parentPort.on('message', (message) =>
{
let found = data.find(line => line[1] === message);
if (found) parentPort.postMessage("FOUND");
});
My first theory was that maybe parentPort.postMessage is poorly optimised and takes x amount of time to fully contact the main thread, but after trying an immediate postMessage after any message reception, the delay of an instant response was 0.444ms so I am out of explainations.
I am out of solutions besides what I placed up there