I am having some trouble, when trying to syncing my webservice data with my internal expo sqlite database. My main issue which i have are related to performance issues of the sqlite DB i think.
My entities don't have Relations or so,
It takes ages to insert a single new record into my db (almost 2 or 3 minutes), and i can't figure out why.
hope some one is able to help me out ;D before i throw my laptop out of my window.
This is my syncing code: my datasource settings
const config: DataSourceOptions = {
type: "expo",
database: 'otr.db',
migrations: [CreateDriverTable1639438000000,CreateAtions1701436776460,CreateOrderLineTable1639437586071, CreateCustomerTable1639438500000],
driver: SQLite,
synchronize: false,
logging: true,
entities: entities,
subscribers: [],
migrationsRun: true,
}
ApiService.tsx
mapToEntity(data){
//map all json data to Entity data..
return { ..... } as T
}
async sync() {
try {
const entities = await this.findAll();
if (!this.repository || !this.repository.syncEntities) {
console.log("No repository found");
return;
}
const mappedEntities = await Promise.all(entities.map(async (data) => {
try {
return await this.mapToEntity(data);
} catch (e) {
console.log(e);
// Handle the error if needed
return null; // or throw the error
}
}));
await this.repository.syncEntities(mappedEntities);
// await Promise.all(mappedEntities.map(async (entity) => {
// await this.repository?.syncEnity(entity);
// }))
console.log("Synced")
} catch (error) {
console.error('Error during synchronization:', error);
}
baseRepository.tsx
public async syncEntities(dataArray: T[]) {
await this.connection.transaction(async (manager) => {
const repo = manager.getRepository<T>(this.ormRepository.target as EntityTarget<T>);
const batchInsert = dataArray.map(async (data) => {
const entity: T = await repo.findOne({ where: { id: data.id } });
if (!entity) {
return this.create(data,repo);
} else {
Object.assign(entity, data);
return this.update(entity,repo);
}
});
await Promise.all(batchInsert);;
});
}
/**
* Updates the entity in the database.
* @param entity
* @returns
*/
public async update(entity: T, repo?: Repository<T>): Promise<T> {
try {
if (repo != null) {
await repo.update({ id: entity.id }, entity);
} else
await this.ormRepository.update({ id: entity.id }, entity);
return entity;
} catch (error) {
console.log(error);
console.log(`Error updating ${entity}`, error)
}
}
/**
* Creates a new entity in the database.
* Using the ORM, we create a new entity and insert it into the database.
* @param entity
* @returns
*/
public async create(entity: T, repo?: Repository<T>): Promise<T> {
try {
let e;
if (repo != null) {
e = repo.create(entity);
await repo.insert(e); //we know it's a new entity, so we insert it
} else {
e = this.ormRepository.create(entity);
await this.ormRepository.insert(e); //we know it's a new entity, so we insert it
}
return e;
} catch (error) {
console.log(`Error creating ${entity.id}`, error.message)
}
}
i have tried wrapping it my selections into a transaction. but insert is very slow.. also tried without async await, but yeah that did't help. Testing it with a single resource of my webservice such that i know that it is'nt a entity issue..