Mono repo Kedro project

Question

I started a Kedro project a while ago and started to build different parts of the pipeline which only tangentially interact with each other. In some cases not much at all.

As a consequence, as the project grew, I am starting to get issues with the dependencies given that the installation process uses requirement.txt

Thus I wanted to ask if Kedro was meant to be designed as a mono repo or not. If so, is there an example of how this could be done, please? Thanks in advance.

Answer 1

So Kedro isn't necessarily opinionated on the question of mono vs multi repo, but the modern way to use kedro via Modular pipelines and Micro-packaging is designed with this sort of pattern in mind.

You can even provide a module specific requirements.txt in your pipeline folder which means when users package/consume your reusable pipelines they won't take the global requirements, but just the pipeline specific ones.

Now if you are at a stage where your different pipelines are using very different requirements that are impossible to compile at a global level, it would be recommended to try and keep the requirements aligned or think about breaking it up into smaller repositories.