JS monorepos in prod 2: project versioning and publishing
By David WORMS
Jan 11, 2021
- Categories
- DevOps & SRE
- Front End
- Tags
- CI/CD
- Git
- GitOps
- JavaScript
- Monorepo
- Node.js
- Release and features
- Unit tests [more][less]
Never miss our publications about Open Source, big data and distributed systems, low frequency of one email every two months.
One great advantage of a monorepo is to maintain coherent versions between packages and to automatize the version creation and the publication of packages. This article covers the versioning and publishing strategies and best practices before continuing with commit enforcement, unit tests and CI/CD integration in the following articles:
- Part 1: project initialization
- Part 2: versioning and publishing strategies
- Part 3: commit enforcement and changelog generation
- Part 4: unit testing with Mocha and Should.js
- Part 5: merging Git repositories and preserve commit history
- Part 6: CI/CD, continuous integration and deployment with Travis CI
- Part 7: CI/CD, continuous integration and deployment with GitHub Actions
Versioning and publishing strategies
Remember the usage of the --independent
flag in the lerna init
command. It tells Lerna about our versioning strategy. In our case, the packages are all Gatsby plugins, and it makes sense to group them inside a single Git repository. They are however independent packages, with their own release cycle and maturity. Thus, we don’t wish to share a single version for all of them.
It is a time to use Lerna to manage the versioning. Running the lerna version
command does the few things:
- Prompts the user a choice of version for each package since those are managed independently.
- Saves the version inside the
package.json
files. - Creates a tag with the package name and the version.
- Commits the changes and push them to the remote server.
But first, Lerna can customize the commit message. In the lerna.json
configuration file, add the entry:
{
...
"command": {
...
"version": {
...
"message": "chore(release): publish"
}
}
}
And commit the change:
git commit -a -m 'build: customize lerna versioning message'
We can run lerna version
to release an initial version for our two packages:
lerna version
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Assuming all packages changed
? Select a new version for gatsby-remark-title-to-frontmatter (currently 0.0.0) Premajor (1.0.0-alpha.0)
? Select a new version for gatsby-caddy-redirects-conf (currently 0.0.0)
Patch (0.0.1)
Minor (0.1.0)
Major (1.0.0)
Prepatch (0.0.1-alpha.0)
❯ Preminor (0.1.0-alpha.0)
Premajor (1.0.0-alpha.0)
Custom Prerelease
Custom Version
Lerna present us a list of possible incremental versions, including patch
, minor
and major
version as well as prerelease numbers and custom release such as alpha
, beta
and rc
. We are free to choose a different value for each package. Once it is ready, Lerna asks for a final confirmation:
lerna version
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Assuming all packages changed
? Select a new version for gatsby-remark-title-to-frontmatter (currently 0.0.0) Premajor (1.0.0-alpha.0)
? Select a new version for gatsby-caddy-redirects-conf (currently 0.0.0) Preminor (0.1.0-alpha.0)
Changes:
- gatsby-remark-title-to-frontmatter: 0.0.0 => 1.0.0-alpha.0
- gatsby-caddy-redirects-conf: 0.0.0 => 0.1.0-alpha.0
? Are you sure you want to create these versions? Yes
lerna info execute Skipping releases
lerna info git Pushing tags...
lerna success version finished
The gatsby-remark-title-to-frontmatter
package is a major alpha release preparing version 1.0.0
. The gatsby-caddy-redirects-conf
package is a minor alpha release preparing version 0.1.0
.
Private and local NPM registry
The source code is now available on GitHub and the commit is tagged with our version. The tags are gatsby-remark-title-to-frontmatter@1.0.0-alpha.0
and gatsby-caddy-redirects-conf@0.1.0-alpha.0
. The tags reflect the package names and versions.
Our two packages are not yet available to the community. For other users to use them from within their package.json
as dependencies, they need to be published on an NPM registry. Lerna provide the command lerna publish
for this. Before describing how lerna publish
works, a local NPM registry is setup for the sake of testing.
Hosting your private NPM registry has multiple advantages. It provides guarantees that no external parties have access to your packages. It speeds up the downloading process when hosted inside your network. It is free assuming you use an open source registry project and that you have the machine to host it.
For the sake of testing and because NPM publications are not revocable after 48 hours, a local NPM registry is used as an alternative to the official NPM registry. Verdaccio is a simple registry that can be started on your host machine. Skip this step if you wish to publish your packages directly on the official NPM registry. To install it, you can use Docker, Kubernetes or NPM. For example, using NPM:
npm install -g verdaccio
verdaccio
I have personally used minikube and Helm:
minikube start
😄 minikube v1.14.0 on Darwin 10.15.7
✨ Using the hyperkit driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing hyperkit VM for "minikube" ...
🐳 Preparing Kubernetes v1.19.2 on Docker 19.03.8 ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" by default
helm repo add verdaccio https://charts.verdaccio.org
"verdaccio" has been added to your repositories
helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "verdaccio" chart repository
Update Complete. ⎈Happy Helming!⎈
helm install npm verdaccio/verdaccio
NAME: npm
LAST DEPLOYED: Thu Dec 3 11:03:33 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app=verdaccio,release=npm" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward --namespace default $POD_NAME 8080:4873
echo "Visit http://127.0.0.1:8080 to use your application"
export POD_NAME=$(kubectl get pods --namespace default -l "app=verdaccio,release=npm" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward --namespace default $POD_NAME 4873:4873
In both cases, you can open your web browser and navigate to http://127.0.0.1:4873
. The welcome screen asks you to create an account:
npm adduser --registry http://localhost:4873
Username: david
Password:
Email: (this IS public) david@adaltas.com
Logged in as david on http://localhost:4873/.
All future lerna publish
commands can refer to this registry by inserting the --registry
flag. Remove this flag to publish your packages on the public and official NPM registry.
To avoid passing the --registry
flag on every lerna publish
, the lerna.json
can store its location:
{
"command": {
"publish": {
"registry": "http://localhost:4873/"
}
}
}
Storing the registry address in lerna.json
implies that it will be committed to Git and shared with your collaborators.
Alternatively, Verdaccio can be globally defined as your default registry inside your ~/.npmrc
file:
npm set registry http://localhost:4873
cat ~/.npmrc | grep registry=
registry=http://localhost:4873/
NPM publication
It is a time to publish our packages on the NPM registry of our choice with lerna publish
.
There are two interesting strategies allowing Lerna to know which version it shall publish. The first one is to publish packages in the latest commit where the version is not present in the registry with the from-package
argument.
lerna publish from-package
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna WARN Unable to determine published version, assuming "gatsby-remark-title-to-frontmatter" unpublished.
lerna WARN Unable to determine published version, assuming "gatsby-caddy-redirects-conf" unpublished.
Found 2 packages to publish:
- gatsby-remark-title-to-frontmatter => 1.0.0-alpha.0
- gatsby-caddy-redirects-conf => 0.1.0-alpha.0
? Are you sure you want to publish these packages? No
The second one is to look for the tag in the current commit with the from-git
argument. In our case, both have the same consequences.
lerna publish from-git
info cli using local version of lerna
lerna info versioning independent
Found 2 packages to publish:
- gatsby-caddy-redirects-conf => 0.1.0-alpha.0
- gatsby-remark-title-to-frontmatter => 1.0.0-alpha.0
? Are you sure you want to publish these packages? Yes
lerna info publish Publishing packages to npm...
lerna WARN ENOLICENSE Packages gatsby-caddy-redirects-conf and gatsby-remark-title-to-frontmatter are missing a license.
lerna WARN ENOLICENSE One way to fix this is to add a LICENSE.md file to the root of this repository.
lerna WARN ENOLICENSE See https://choosealicense.com for additional guidance.
lerna success published gatsby-remark-title-to-frontmatter 1.0.0-alpha.0
lerna http fetch PUT 201 http://localhost:4873/gatsby-remark-title-to-frontmatter 824ms
lerna success published gatsby-caddy-redirects-conf 0.1.0-alpha.0
lerna http fetch PUT 201 http://localhost:4873/gatsby-caddy-redirects-conf 873ms
Successfully published:
- gatsby-caddy-redirects-conf@0.1.0-alpha.0
- gatsby-remark-title-to-frontmatter@1.0.0-alpha.0
lerna success published 2 packages
For the sake of clarity, the notice
output logs are filtered.
Package content filtering
Not every file needs to be published. Content which is not stored in Git shall probably not be published. Secret, certificates, private configuration are such examples. By default, NPM and Yarn look at your .gitignore
files and import its directives unless there is a .npmignore
.
But being stored and shared in Git doesn’t necessarily mean it shall be publish in NPM. Your packages must be as small as possible. For example, there is no need to publish your tests.
You could copy the .gitignore
file at the root of your packages into a .npmignore
file and start adding new rules. There is however, to my opinion, a much better approach. Use the files
property from your package.json
file. From the package.json documentation:
The optional
files
field is an array of file patterns that describes the entries to be included when your package is installed as a dependency. File patterns follow a similar syntax to.gitignore
, but reversed: including a file, directory, or glob pattern (*
,**/*
, and such) will make it so that file is included in the tarball when it’s packed. Omitting the field will make it default to["*"]
, which means it will include all files.
However, certain files will be included such as the README, CHANGELOG and LICENSE files.
The gatsby-remark/title-to-frontmatter/package.json
and gatsby/caddy-redirects-conf/package.json
files are enriched with:
{
"files": [
"/lib"
]
}
Then, the changes are committed:
git commit -a -m 'build: include lib in published packages'
If we were making changes to a package with a doc
folder and its associated content, it would be committed to Git but not published.
Selective package publication
Notice how Lerna is complaining about our packages which does not provide a license file. It also proposes us to place a LICENSE.md
file at the root of the repository. Let do just that and release a new version.
curl \
https://raw.githubusercontent.com/adaltas/node-csv/master/LICENSE \
-o LICENSE.md
git add LICENSE.md
git commit -m 'build: MIT license'
lerna publish from-git
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna notice from-git No tagged release found
lerna success No changed packages to publish
This time, running lerna publish
has no effect even if there are some changes since the last commit. This is because the commit didn’t affect any of the published packages defined as Yarn workspaces. This is also because the root package is not published, since it is marked as private inside the package.json
file:
cat package.json | grep private
"private": true,
Selective package publication based on content
The lerna publish
command only publish packages with changes. It can go a step further by not taking into account changes in some selected files.
The command.publish.ignoreChanges
is an array of globs that won’t be included in lerna changed/publish
.
It can be globally defined in your lerna.json
configuration file, for example to not publish new release if the changes where only in tests:
{
...
"ignoreChanges": [
"**/test/**"
]
}
We need to commit the changes and create new versions before testing it:
git commit -a -m "build: lerna version ignore test"
# Note, select 'prerelease' to match the version below
lerna version
...
Changes:
- gatsby-remark-title-to-frontmatter: 1.0.0-alpha.0 => 1.0.0-alpha.1
- gatsby-caddy-redirects-conf: 0.1.0-alpha.0 => 0.1.0-alpha.1
? Are you sure you want to create these versions? Yes
lerna info execute Skipping releases
lerna info git Pushing tags...
lerna success version finished
Once a new test is committed in one of the packages, no version is proposed:
echo 'console.warn("TODO")' > gatsby-remark/title-to-frontmatter/test/index.js
git add gatsby-remark/title-to-frontmatter/test/index.js
git commit -m 'test(title-to-frontmatter): todo tests'
lerna version
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Looking for changed packages since gatsby-caddy-redirects-conf@0.1.0-alpha.1
lerna info ignoring diff in paths matching [ '**/test/**' ]
lerna success No changed packages to version
Files modification can be filtered from lerna version
and lerna change
by appling the --ignore-changes
flag or modifying the lerna.json
file with the ignoreChanges
property. For example, to discard a new version when only a typo was made to a README file.
Automatic incremental versions
Let’s make some change to one of our package. The gatsby-caddy-redirects-conf
package, currently at version 0.1.0-alpha.1
, deserves a README.
echo \
'# Package `gatsby-caddy-redirects-conf`' \
> gatsby/caddy-redirects-conf/README.md
git add gatsby/caddy-redirects-conf/README.md
git commit -m "docs(title-to-frontmatter): new readme file"
Instead of having Lerna asking us which version to set, we tell Lerna to automatically increment the current prerelease version:
lerna version prerelease
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Looking for changed packages since gatsby-caddy-redirects-conf@0.1.0-alpha.1
lerna info ignoring diff in paths matching [ '**/test/**' ]
Changes:
- gatsby-caddy-redirects-conf: 0.1.0-alpha.1 => 0.1.0-alpha.2
? Are you sure you want to create these versions? Yes
lerna info execute Skipping releases
lerna info git Pushing tags...
lerna success version finished
Since we indicate which version we wish with the prerelease
SemVer keyword, Lerna doesn’t need to ask us the targeted version and, instead, only asks to validate its suggestion.
Note, to graduate from a prerelease cycle with the Conventional Commit, the command looks like lerna version --conventional-commits --conventional-graduate
. We cover the Conventional Commit in the follow up article.
CI/CD, separation between versioning and publishing
If you try to run the lerna publish
command without executing lerna version
before, you will notice that it first creates a version of the package and then publishes the package. Why did we call lerna version
if it is optional?
Versioning is separated from publishing because I don’t execute the two commands in the same location. For example, I can run lerna version
manually, when I feel confident that my package deserves a new release, and automate lerna publish
remotely on a CI/CD platform, when all the checks and tests successfully pass and once a new version has been detected.
Cheat sheet
- Custom commit message from Lerna
Modifylerna.json
:Commit the change:{ ... "command": { ... "version": { ... "message": "chore(release): publish" } } }
git commit -a -m 'build: customize lerna versioning message'
- Create a new version:
Automatic increment the prerelease version:
lerna version
Unless the modified files match a pattern:lerna version prerelease
lerna version --ignore-changes '**/*.md' '**/__tests__/**'
- Plushing a new version
Or conditionnaly if there is a release tag:
lerna publish
lerna publish from-git
- Using a custom repository
Add theregistry
flag:Or editlerna publish --registry http://localhost:4873/
lerna.json
:Or edit{ "command": { "publish": { "registry": "http://localhost:4873/" } } }
~/.npmrc
:npm set registry http://localhost:4873
- Content filtering
In ‘package.json’:{ "files": [ "/lib" ] }
Conclusion
The Lerna version
and publish
commands come with many arguments which are worth to investigate. We will cover next how to enforce the commit message format, how to run unit tests and how to automate the publication of packages in a CI/CD environment.