CodeBuild - Extremely long build times/caching

Question

My project is a large JS/Yarn monorepo (using workspaces) that is contributed to dozens of times a day, by multiple devs. Every time a PR is opened, we build, lint, and test the project, which requires running `yarn install` before anything else. Once a merge occurs to the main branch, any open PRs must merge with the new main branch, and the PR checker needs to be run again before _that_ branch is merged. As you can imagine, there can be a pretty large backlog of open PRs when we're in crunch time.

I have successfully used (both s3 & local) caching before in smaller projects, however I can't seem to get **local** caching working with our monorepo (this is a large project, so s3 is much too slow for us, as far as I can tell).

When we try using local caching, our build fails with `"EEXIST: file already exists, mkdir '/codebuild/output/src11111111/src/github.com/OUR_ORG/OUR_PROJECT/node_modules/@OUR_PACKAGE/common'".`.

This behavior is documented across the web:
- https://github.com/aws-samples/aws-codebuild-samples/issues/8
- https://stackoverflow.com/questions/55890275/aws-codebuild-does-not-work-with-yarn-workspaces

Some of the remedies include using s3, but as mentioned this is a large project and we update packages at least once every two weeks (some weeks , multiple times), and our initial upload more than doubles our time to 40min. Downloading doesn't save us much time either.

- Are there any reasonable steps we can take to cache our node_modules directories so we don't have to pull from yarn every time? 
- Are there any other solutions to speed up these build times (we're currently ~14min. After optimizing other parts of the build process)? 
- Do you have any example (large) mono repos you can point to as a template?
- Any other tips to speed up our build times?

Answer

I would think you should be able to check for the presence of the directory during your build, and only call `yarn install` if the directory is not present. For example:
```
if [ ! -d "/codebuild/output/src11111111/src/github.com/OUR_ORG/OUR_PROJECT/node_modules/@OUR_PACKAGE/common" ]; then
  yarn install
fi
```
This would only run `yarn install` if the directory does not exist, avoiding this error when the package cache exists due to local caching.

That said, S3 caching does tend to be faster than installing packages, since the files can be pulled without leaving the AWS network, and there is also less processing involved with the cache than there is with any package manager, so this option is worth exploring if the above doesn't work for you.

As for speeding up the build times, there aren't many general suggestions I can make, although if a significant portion of your build is in the `PROVISIONING` state, you may find better performance by ensuring you are on the latest release of the CodeBuild image, as the docker image for this is cached, whereas the older versions may not be.

Unfortunately, I wouldn't have any examples I can offer for large repos using CodeBuild, however you are certainly free to open a case if you'd like us to take a closer look at your environment!

CodeBuild - Extremely long build times/caching

Relevant content