At work we have a pretty big Python monorepo. The way we scale it is by having many standalone CLI mini apps ( about 80) atm with most of them outputting json/parquet in GCS or bigquery tables. Inputs are the same.

I insisted a lot on this unix (ish as it's not pipes) philosophy. It paid off so far.

We can test each cli app as well as make broader integration tests.