Is there any way to improve CodeBuild caching speeds?

We run an Angular/NX monorepo and up to now we have been using S3 to cache the node_modules directory to reduce build time. This does work, in theory, except that when we run the timings, it takes about 90 seconds to download the cache at the start of the build and 210 seconds to upload the cache to S3 at the end of the build.

Sub-problem: The cache is ~ 700MB. These seem like very long upload/download times to S3 for a service running on AWS. 3MB/s for an upload to S3?

This adds five minutes to the build time, and I have recently discovered that the build runs slightly faster without caching at all and just downloading the dependencies from the web. That's insane.

So I tried to move to a local custom cache, and that seems completely pointless. It seems like no matter how frequently or infrequently you start a build, the cache is never there. Which makes sense; I'm sure there are a few hundred hosts running at any given time, so the odds of having my build allocated to the same host again, are tiny.

We're not doing anything particularly special. It's a standard Linux / Ubuntu build image. The build on its own takes about 40 seconds. For comparison we used CircleCI to see how that runs, and while the uncached build runs in roughly the same time, CircleCI's cache works correctly and cached builds take about 90 seconds in total, version 6-7 minutes in CodeBuild.