The Road to CouchDB 3.0: Goodbye Travis, Hello Jenkins, Much Improved Continuous Integration

This is a post in a series about the Apache CouchDB 3.0 release. Check out the other posts in this series.

In the lead up to CouchDB 3.0, we have noticed that our needs for running automated testing had outgrown what the fine folks at Travis CI provide for free, so we started looking for alternatives.

We had multiple goals for this new CI solution:

  • quick startup time: during busy times, Travis’ free offering sometimes takes an hour to start working, which is entirely fair, but we needed something better.
  • better performance: we spent quite some time hardening our test suite against Travis’s environment of resource constrained machines, to ensure we have no false-negative reports due to timeouts and other issues. In some cases, this meant writing a test case that did not represent real-world usage of the code it was testing. We are also always happy for a faster test runtime, even on modern machines, our test suite takes about 20 minutes to run, add a matrix of different configurations, and we need some serious compute to move quickly and confidently.
  • broader platform support: CouchDB 3.0 now officially supports ARM and PPC platforms in 32bit and 64bit, mainly because our new solution can automatically run tests on these platforms each time we change code. We also now (again) test automatically on FreeBSD systems, and a macOS build machine is in the works.
  • nightly binaries: we wanted to be able to build binaries for all supported platforms for PRs and master for improved testing of as-of-yet unreleased features. This now happens automatically. You can subscribe to the Developer mailing list to get access to these binaries.

Our new CI solution is based on Jenkins via CloudBees with worker servers donated by IBM/Cloudant and MacStadium and it supports all of the above requirements

However, none of this would have been made possible without the untiring efforts of Joan “wohali” Touzet who pulled all the strings together to making this happen, from organising and configuring donation hardware, to sorting out ASF policy to make sure we are all aligned, to writing oh so many build scripts to tie the whole service together. Some of Joan’s time was donated by Neighbourhoodie Software.

Meet the new https://ci.couchdb.org.

The Road to CouchDB 3.0: Prepare for 4.0

This is a post in a series about the Apache CouchDB 3.0 release. Check out the other posts in this series.

In early 2019, the IBM/Cloudant team proposed a major change to CouchDB: replace the storage and clustering subsystems with FoundationDB.

The reasoning is straightforward: how much effort would it be to bring the existing underpinnings into the 2020s and beyond? Are there alternative options? Can we use something that exists already?

After about seven months of diligent research, design and planning, we concluded that it would not only be feasible, but also beneficial to CouchDB to make the move to FoundationDB.

We have a lot more to say about this in the future, but if you want to follow along at home, check out the RFCs where we discuss the various technical details.

One of the consequences for moving foundations (pun!), is that the new one is built on different tradeoffs than the old one, which has served us very well so far. This in turn leads to certain limits that we’ll have to impose on certain operations in CouchDB 4.0 and beyond.

Since it is beneficial to keep to those limits already in 3.0, and to make the transition to 4.0 easier in the future, we have introduced, or reduced limits to certain things in CouchDB in line with the limits we believe that CouchDB 4.0 will have.

If you need to go beyond those limits, they are configurable for all of CouchDB 3.x, but won’t be configurable at all in 4.x.

For now, the only limit we’ve added is max_document_size, it drops from 4GB, which was a theoretical limit to begin with, to 8MB, which is much more reasonable in terms of CouchDB performance.