Desktop CI

Discussion on CI for the Desktop

What is the workflow we want?

  • Feature/bugfix branch proposed. Tests are run, only review/merge if green.
    meskio: just to be clear, will be nice to be able to run the tests for branches in our own repositories even if we haven’t done a pull-req for them yet. But it will be ok for me if they are only run in case of pull-req.
  • Merge: tests are run. Blame someone if not green.
  • When tagging/packaging: ability to test the release/xx branches before tagging.

Solution

  • Travis for public Pull Requests from GitHub, easy to set up and for contributors to see.
  • Buildbot for internal use, it is able to follow complex flows and every desktop dev can do Python to code them.

Notes from Jenkins and Buildbot

Jenkins vs Buildbot

  • Jenkins is ready to be used, but is difficult to adapt to unsupported workflows
  • Buildbot is slow to set up, but you can specify much more detailed workflows → “This flexibility has led to its use in a number of high-profile open-source projects, including Chromium, WebKit, Firefox, Python, and Twisted.”
  • Devpi has out of the box Jenkins integration: “You can set a per-index Jenkins trigger for automatically tox-testing anyuploaded release file and query releases for their test results.” (doc.devpi.net/latest/)
  • Jenkins is Java, Buildbot is Python. Desktop folks know the latter.

Needed Functionality

Interesting testing functionality (for the future, but worthy to explore if we can do it easily)

  • Headless tests for Qt. We had this at some point, it’s very interesting because it allows to test the behavior of the Qt GUI (signals, slots etc).
  • Integration tests: simulate clicks, etc, test on a vm/container that, ie, eip has created a tunnel, or that a mail can be sent/received.
    • We never got close to integration tests, but maybe it’s time to think about it seriously.
  • Packaging tests: ivan has done some work about this. The idea is that, for each time that we build a debian package, we want to check if it can be installed on a pristine box, and launched successfully (this might be easier once that we have a completely decoupled backend).

First approach to Buildbot vs Jenkins

  • Buildbot seems to be used by large projects such as Python and Chromium, but maintainance is a pain because there is nothing pre-built: you need to code everything, and for me that’s as if we had a new project to handle.
  • I really like the github pullreq plugin of jenkins, as well as how easy it is to find appropriate documentation for whatever you come up with.
  • I like the Cobertura plugin for Jenkins, too.

Second approach to Buildbot vs Jenkins.

  • Jenkins has a lot of plugins, but they don’t seem to me as reliable enough to consider them a competitive advantage against Buildbot. For example, I wasn’t able to set up the GitHub plugin correctly.
  • Buildbot makes a breath for software developers to reuse and improve the CI configuration thanks to being Python code. VCSing the configuration is a great plus.

Notes from Travis

  • Open Source, we could just set up an account and specify our requirements in travis.yml → no setup/management overhead from the beginning
  • Proposed flow is the default, batteries included! hehehe
  • Master/slaves inside their cloud, with their VMs
  • Facebook does coverage with python coverage + coveralls: github.com/tornadoweb/tornado
  • Future plans on deploying their stack onto our own infrastructure (until then, we basically depend on Github), and propose them to include the functionality of specifying a list of ips already set up to be slaves, for testing.docs.travis-ci.com/user/languages/python/
  • Support multiple python versions: http://docs.travis-ci.com/user/languages/python/

Notes from Gocd

  • Interesting evolution of the CI concept to CD (continuous integration vs continuous delivery).
  • We need to fix the run_tests.sh script to be able to use gocd for the most simple case. We’ve got no time to fix it, since buildbot is already doing that.

Notes from Gitlab-CI

  • In a quick search, docs don’t seem too detailed for me, there are things I haven’t been able to find (perhaps they are not offered) such as master/slaves, GitHub hooks…
  • It doesn’t show anything competitive against the combination of Travis and Buildbot.