Singularity Container: I cannot make writable persistent from singularity sandbox to host

Question

I do the following

1. I build a sandbox

   singularity build --sandbox project/ project.recipe

2. I access the sandbox to modify stuff

   singularity shell -B /path/to/host:/path/in/sandbox --writable project

3. I build the image from the sandbox

   singularity build project.sif project

However my changes are not updated. in path/to/host, it is modified but in /path/to/sandbox nothing changes. Since I build the image from the sandbox eventually my changes are not shown in the .sif file. btw /path/to/host/ = /path/in/sandbox. If I omit the bind mount then it is modified in the sandbox and I can build the image with the modifications but then the file is no longer updated in the host. I want to access the sandbox and the able to modify both, the sandbox and the host