Splitting up test suites at Etsy

Following last weeks post on how Esty manages development and operations they have posted another good blog post. This particular posts talks about how they have divided up their test suit to enable very short deployment times. Here’s the introduction:

Lately we’ve been deploying code 25 times a day or more, and running automated tests every time we deploy. Conservatively, we run our tests about 25 times on a normal business day.

At Etsy, deployments are managed by engineers, not by ops or a release management team. The idea is: you write code, and then you deploy it to production. Even dogs deploy code. By the time 8am rolls around on a normal business day, 15 or so people and dogs are starting to queue up, all of them expecting to collectively deploy up to 25 changesets before the day is done.

In order to do this, Esty have classified their tests into: Unit, Integration, Network, Smoke and Functional tests. They also have little tolerance for slow tests and tests that fail intermittently.

The article hints at some interesting practices and tools including the use of Jenkins for Continuous Deployment, and cumber, selenium and phpUnit for testing.

Run end-to-end, the 7,000 trunk tests would take about half an hour to execute. We split these tests up into subsets, and distribute those onto the 10 machines in our Jenkins cluster, where all the subsets can run concurrently. Splitting up our test suite and running many tests in parallel, gives us the desired 11 minute runtime.

Read the full post here.

, , , , ,

Comments are closed.