What is the workflow we want?¶

• Feature/bugfix branch proposed. Tests are run, only review/merge if green.
  meskio: just to be clear, will be nice to be able to run the tests for branches in our own repositories even if we haven’t done a pull-req for them yet. But it will be ok for me if they are only run in case of pull-req.
• Merge: tests are run. Blame someone if not green.
• When tagging/packaging: ability to test the release/xx branches before tagging.

Solution¶

• Travis for public Pull Requests from GitHub, easy to set up and for contributors to see.
• Buildbot for internal use, it is able to follow complex flows and every desktop dev can do Python to code them.

Notes from Jenkins and Buildbot¶

Jenkins vs Buildbot¶

• Jenkins is ready to be used, but is difficult to adapt to unsupported workflows
• Buildbot is slow to set up, but you can specify much more detailed workflows → “This flexibility has led to its use in a number of high-profile open-source projects, including Chromium, WebKit, Firefox, Python, and Twisted.”
• Devpi has out of the box Jenkins integration: “You can set a per-index Jenkins trigger for automatically tox-testing anyuploaded release file and query releases for their test results.” (doc.devpi.net/latest/)
• Jenkins is Java, Buildbot is Python. Desktop folks know the latter.

Needed Functionality¶

• Be able to run the test suites, and watch each test succuess/fail with informative output.
  • mix of unittest/trial.
• Run the test suites (watching each python package) every time that a feature branch is a candidate to be merged
  • github hooks, for pullreqs?
    • Jenkins: wiki.jenkins-ci.org/display/JENKINS/Git...
    • Buildbot: everything I’ve read says they’re using a tool apart from Buildbot to do the hook
• Run the test suites on each merge
  • Jenkins: wiki.jenkins-ci.org/display/JENKINS/Git... (step by step conf: fourword.fourkitchens.com/article/trigg... ) → doesn’t work properly/too difficult to configure
  • Buildbot: docs.buildbot.net/0.8.12/manual/cfg-sta...
• Run the test suites on demand.
• Run the test suites on multiple slaves / platforms.
  • Ideally, each of us developers could maintain a slave in our own machines.
  • We could host a distributed farm of win/mac/etc vms.
  • Jenkins: http://www.parsed.io/setup-a-mac-slave-for-jenkins/ mac, wiki.jenkins-ci.org/display/JENKINS/Ste... windows/linux
  • Buildbot: http://danir.us/2013/may/27/continuous-integration-web-apps-buildbot/
• Integration with python coverage.
  • Jenkins: wiki.jenkins-ci.org/display/JENKINS/Cob... + coverage.readthedocs.org/en/coverage-4....
  • Buildbot: simply add the call to a step

• Notifications (irc/mail). Easy, I guess
  • Jenkins: irc → wiki.jenkins-ci.org/display/JENKINS/IRC..., email → wiki.jenkins-ci.org/display/JENKINS/Ema...
  • Buildbot: email → docs.buildbot.net/0.8.1/MailNotifier.html, irc → docs.buildbot.net/0.8.12/tutorial/tour....
• For each run, ideally recreate the whole virtualenv. For this to be efficient, we could use:
  • virtualenv – this is a must
    • Jenkins: wiki.jenkins-ci.org/display/JENKINS/Shi...
    • Buildbot: example using a virtualenv per slave → github.com/danirus/buildbot-sample-conf...
  • tox – to run the suite on several python versions – we started with this, but I think we’re dropping support for py26.
    • Jenkins: wiki.jenkins-ci.org/display/JENKINS/Shi...
    • Buildbot: nothing seen in a quick search of “buildbot tox”
  • devpi – fast local pypi mirror
    • Jenkins: doc.devpi.net/latest/userman/devpi_misc...
    • Buildbot: nothing found in a quick search of “buildbot devpi”
  • wheels – fast binary package format
    • Jenkins: http://cowboyprogrammer.org/building-python-wheels-for-windows/
    • Buildbot: wheels are generated from pip, so there should be no problem

Interesting testing functionality (for the future, but worthy to explore if we can do it easily)¶

• Headless tests for Qt. We had this at some point, it’s very interesting because it allows to test the behavior of the Qt GUI (signals, slots etc).
• Integration tests: simulate clicks, etc, test on a vm/container that, ie, eip has created a tunnel, or that a mail can be sent/received.
  • We never got close to integration tests, but maybe it’s time to think about it seriously.
• Packaging tests: ivan has done some work about this. The idea is that, for each time that we build a debian package, we want to check if it can be installed on a pristine box, and launched successfully (this might be easier once that we have a completely decoupled backend).

First approach to Buildbot vs Jenkins¶

• Buildbot seems to be used by large projects such as Python and Chromium, but maintainance is a pain because there is nothing pre-built: you need to code everything, and for me that’s as if we had a new project to handle.
• I really like the github pullreq plugin of jenkins, as well as how easy it is to find appropriate documentation for whatever you come up with.
• I like the Cobertura plugin for Jenkins, too.

Second approach to Buildbot vs Jenkins.¶

• Jenkins has a lot of plugins, but they don’t seem to me as reliable enough to consider them a competitive advantage against Buildbot. For example, I wasn’t able to set up the GitHub plugin correctly.
• Buildbot makes a breath for software developers to reuse and improve the CI configuration thanks to being Python code. VCSing the configuration is a great plus.

Notes from Travis¶

• Open Source, we could just set up an account and specify our requirements in travis.yml → no setup/management overhead from the beginning
• Proposed flow is the default, batteries included! hehehe
• Master/slaves inside their cloud, with their VMs
• Facebook does coverage with python coverage + coveralls: github.com/tornadoweb/tornado
• Future plans on deploying their stack onto our own infrastructure (until then, we basically depend on Github), and propose them to include the functionality of specifying a list of ips already set up to be slaves, for testing.docs.travis-ci.com/user/languages/python/
• Support multiple python versions: http://docs.travis-ci.com/user/languages/python/

Notes from Gocd¶

• Interesting evolution of the CI concept to CD (continuous integration vs continuous delivery).
• We need to fix the run_tests.sh script to be able to use gocd for the most simple case. We’ve got no time to fix it, since buildbot is already doing that.

Notes from Gitlab-CI¶

• In a quick search, docs don’t seem too detailed for me, there are things I haven’t been able to find (perhaps they are not offered) such as master/slaves, GitHub hooks…
• It doesn’t show anything competitive against the combination of Travis and Buildbot.