buf build can't find file when called from same directory as buf.work.yaml

Question

I have a protobuf repository with the following structure:

├── protos
│   ├── common
│   │   ├── buf.yaml
│   │   └── time.proto
│   └── internal
│       ├── buf.yaml
│       └── user
│           └── user.proto
├── buf.gen.yaml
└── buf.work.yaml

The stub for `time.proto is:

syntax = "proto3";
package common;

option go_package = "github.com/my-org/protobuf-gen-go/gopb"; // golang

// Message definition here

and user.proto references this file. Here is its stub:

syntax = "proto3";
package internal.dynamodb;

import "common/time.proto";

option go_package = "github.com/my-org/protobuf-internal/user"; // golang

// Message definition here

Both buf.yaml files look like this:

version: v1
breaking:
  use:
    - FILE
lint:
  use:
    - DEFAULT

and buf.work.yaml looks like this:

version: v1
directories:
 - protos/common
 - protos/internal

Now, when I compile this using protoc I don't get any errors but when I try doing buf build, I can an error:

protos/internal/user/user.proto:4:8:protos/common/time.proto: does not exist

This doesn't happen when I move buf.work.yaml into the protos directory. This is clearly causing issues with building the system. How do I fix this issue?