Hi,

I’ll be developing crabgrass for (at least) the next 6 month and want to keep everyone updated with the progress. You can find the development roadmap on the group wiki.

Week 1 & 2

For the first month I am mostly focussing on upgrading rails and the gems (libraries) we are using. I want to get crabgrass to at least work on top of rails 3.2. As that is still receiving security updates as of today. Updating the underlying libraries requires quite a lot of code changes all over the place so it can introduce new bugs. That’s why i want to get it out of the way as early as possible so I can make crabgrass more stable in the coming months. Upgrading is also a good start to become more familiar with the code base again and it will allow me to fix bugs using the more recent syntax.

Besides the upgrade I started out focussing on a few database performance issues i had observed. I also looked into optimizing caching of things that are reused on every page.

I will hopefully finish the rails 3.2 upgrade tomorrow. I will use the rest of this week to test it on the development server, write tests for bugs i run into and fix them. At the end of the week i will decide if i dare push things further and migrate to rails4. Rails4 comes with strong_parameters, nested_caching for views and turbo links. Each of them are very interesting but also might have their issues with the current codebase. So i will probably play with the gems available for rails 3.2 this week to find out if a migration seems feasible.

All in all I still run into lot of things in the codebase i would like to improve. I try to focus on the dependencies for now and leave other improvement for later.

Thanks for this! I’m SO EXCITED that this is happening.

Week 3

TLDR: Things are going as planned. Crabgrass now runs on rails 3.2 and we’ll stick to that for now.

rails 3.2

The upgrade went rather smooth. I also started cleaning up the use of deprecated code. However we are still using rails plugins which are deprecated as of rails 3.2 and will be removed in rails 4. I replaced all external plugins with more recent gems. I also moved two really small ones that are not maintained anymore into our own lib folder and removed one entirely.

rails 4

Plugins are completely removed in rails 4.0. I cleaned up the external ones but we also make use of plugins for a lot of things inside crabgrass – namely the page types and install specific mods. We also use them for a number of internal libraries. So moving to rails 4.0 will require quite a bit of moving things around.

In addition rails 4.0 is phasing out of security support. Once rails 4.2 is released only rails 3.2, 4.1 and 4.2 will be supported. 3.2 will be supported until rails 5 is release middle of next year. So I will not migrate to rails 4 for now.

However rails 4 has a number of interesting additions that have been backported to rails 3.2:

1. cache_digests – makes nested caches of fragments of codes less cumbersome
2. turbo_links – keep the js and css loaded and fetch links on the same page using ajax to speed up page loads
3. strong_params – check all submitted parameters in requests before using them to create or alter data.

I already included 1).
I played some with 2) and it works but breaks with some of the javascript we use. Seems like a day or two of work to fix this but no priority for now.
3) will be quite cumbersome but it’s a great security addition and i noticed a vulnerability where it would have helped just this week.

plans for week 4 – tests and security

I ran into a few smaller security issues this week. We were too permissive in a few places. So I will use this week to make our security policies more strict:

• restrice routes to only match one http verb
• use strong_params in controllers
• require authorization check in the ApplicationController (all other controllers will inherit it) and remove it from the controllers that do not need it.

At the same time we should have tests that fail if we are being too restrictive or too permissive. So I will also make sure there are tests for all controllers and they verify the permissions in use.

thank you much, azul

azul: thanks for this update! :)

Week 4

In short… I only got to the tests part of “tests and security” last week. However that worked out really well and i will focus on security this week.

First look at route security

We had a few routes for accessing controllers from the different page types that were not specific enough. For example they allowed to send GET requests were POST should have been required or they allowed specifying the controller and action directly. This is a security issue because one can construct urls from it that would trigger actions in ways they were not meant to be used. Allowing GET instead of POST also means these urls can be triggered from links and sources loaded automatically on pages such as images.

From this it was quite clear that the page routes would need a rewrite.

test one two – test test

We used to have a good test coverage of the different page types but had stopped running them as part of the usual test routines on the CI server. So a lot were broken because the code under test had changed. I fixed those and added an integration test for each page type. Integration tests basically try out the application using a headless browser. The one i am using for this evaluates javascript and also tests things like drag & drop for sorting a gallery and so on.

So now we have a set of tests that test the main functionality of all page types by “clicking through” them for us.

back to security

Looking at the routes again I decided it would make sense to turn the overly flexible routes into restful routes. Restful routes make use of the standard http verbs to indicate the type of action that should happen: GET, POST, PUT, DELETE. GET will lead to actions that show things (show and index) only and thus can not be used to trick people into changing data.

At the same time we need a bit of flexibility so that page types can use their own controllers without having to specify routes for each and every one of them. Turns out there is a way to combine restful routes with picking your own controller:

  scope path: 'pages/:page_id/:controller' do
    resources :page_items, path: ''
  end

This will determine the action using the HTTP verbs and thus cleanly separate GET from POST actions and still allow to pick any controller in the url.

plans for week 5

The new routing limits the actions that can be used in page controllers to the default ones. I added sort and print actions for good measure because they are used by a lot of page types. But still some page controllers have to be split up. We have already tried to follow this pattern during the last development sprint but had not applied it to all page types. Now I am going through the remaining page types and turning actions like ‘add_task’ in the TaskListsController into default actions like ‘create’ on a new TasksController. I will definitly finish the route rework this week. I am also looking into adding strong_parameters as a way of securing against manipulated parameters. But I am not sure i can finish that this week.

I notice that this report got a bit technical I’m happy to explain the technical parts in plain english. Feel free to ask.

The big picture

The first month is almost over. I’ve focussed on the bigger reworks that touched large parts of the code. I will finishing the security changes this week and parts of the next. There’s still a bunch of larger changes I would like to make such as upgrading to newer versions of rails and ruby, using jquery instead of prototype and so on.

However I will leave those things for later and move on to the bug fixing phase. I will go through the bug reports, add tests for them and fix them one after the other.

Excellent. Thanks for the feedback Azul, it’s much appreciated.

As for your planned work with bug reports, feel free to comment the reports themselves. I’m interrested to know what is helping or not, and being able to make better one.

Thanks a lot WhilelM. I have not deployed the current state to the test server yet.

I think i will do so next week. It would be great if you could play with it some at that point. In particular i might have fixed some bugs during the rewrite. I’ll go through the list of bugs and assign priorities. Maybe I can send some your way at that time so you can check if they still need to be fixed. That would be sweet.

Yay for fixing security holes!

Maybe I can send some your way at that time so you can check if they still need to be fixed. That would be sweet.

Sure, let me know =)

Week 5

Last weeks report is a bit late this week because I wanted to finish the last bits and pieces before reporting back. I moved on locking down the routes to the ones we really need and just merged that. There’s two routes i still need to take care of but then we will have all routes use fixed http verbs and point to a hardcoded controller. No more setting of the controller or the action in the url itself. I also removed routes to actions that did not exist and made controller functions that are not meant to be used as actions protected.

All in all this means we now have a fixed set of ways of accessing the application and can focus on securing those without thinking about funky ways of accessing controller methods that were not meant to be accessed.

Which brings us to the next step… securing the actions against malicious parameters. I’ll start including strong_parameters and look into the authorization code on a per controller basis this week. That will conclude the security overhaul and bugfixes are next.

timeline

I had planned to use a the first month for upgrading and the second for bug fixes. The upgrade was basically done by the end of October as planned. I moved on to reducing the attac surface so I can catch new bugs this introduced. So I’m moving on to bugfixes about two weeks later than I had planned. I will see how the bugfix phase goes update the timeline on we.riseup.net/cgdev at the end of the month.

Week 6

I did not manage to work as much on crabgrass in the last week as i had hoped… But never the less the strong_parameters inclusion is done and I am finally moving on to bugfixes. Yay! It will be interesting to see how that goes. I also updated the timeline some already.

Week 7 – bugfixing time

After a little bit of final cleanup for the strong parameters I now finally moved on to fixing bugs and writing tests that would catch regressions. I’m done with all issues related to the task list page and the page search so far. The page search was not working at all and that was really bothering me but it turned out to be a quick fix.
You can also track my progress on labs.riseup.net/code/projects/cg-core/i...
Right now i have fixed 6 / 55 reported bugs.

Week 8 and 9 – stuck (a bit)

Last week I set out to continue the bugfix run and fixed some bugs initially. When writing an integration test for one of them I noticed that test was failing for weird reasons so i started digging deeper and discovered an issue with our permission system. We’ve replaced large parts of the permission implementation in the core rework and it’s still not 100% smooth – especially because there are so many cornercases to keep in mind.

So this week I looked into options to fix that particular issue and just send an email out to elijah to hear his opinion. I’m also recovering from a cold so not as productive as I would like to be.

I hope to be back in full swing next week and resolve the permission issue and resume the bugfixes. Right now I am at 9 / 54 bugs fixed (one actually was a feature request).

I just read through this update list. Really excited to see all the progress.

Week 10 – moving on with bugs

The permission issue and the randomly failing tests that were holding me back are finally fixed. So I was able to work through the list of bugs one by one. I mostly focussed on page search and the actions one can perform on pages. I also found some new bugs so now the bug count is 14 / 56 fixed.

Currently i am tweaking the user directory. The autocomplete was not working (fixed) and I’m in the process of applying a bit of styling. The autocomplete code was always bothering me and i finally found a good way to implement it and reduce our dependency on prototype – a javascript library that we want to replace with jquery eventually.

I will apply the new approach to autocomplete dialogs all over the place and then move on with bugfixes.

three months in

last three weeks

woohaa – time sure runs by quick lately. I was a bit stuck in weeks 11 and 12 and came back to speed in week 13 knowing that last weekend would be the first bigger QA session. You can see the difference on the contributions chart on www.github.com/azul

I stayed focussed on bug fixes most of the time. But i also reworked the people and group directories some. Now you can search your own groups and your friends and peers.

documentation sprint

Some friends in Paris met to continue writing a manual for the new version of crabgrass. So the test server had to be updated and I spend the last week and some of the weekend fixing bugs so their documentation went fine.
This mostly worked out. Only sunday my last deploy failed and the server was not usable – and I had already left. Good thing they had their own server to play with.

Naturally the documentation session also revealed a number of new bugs so now the bug count is 25 / 70 fixed. However bugs are also getting smaller and less severe.

Next steps

I started talking to micah about setting up the new server. We’ll meet again next week.

So while I will continue fixing bugs I will also work on getting things ready for the new server and start looking into customization for we.riseup.net.

0.6.0-beta release

I’m about to tag the core rework with 0.6.0-beta. This marks the first crabgrass release in a long time. I spend the day going through the issues list, QAing and closing a lot of issues and organizing the bug tracker. The 0.6.0-beta version has 62 closed issues now – most of them fixed during the last months.

next steps

There’s 49 open issues in 0.6.0 (36 bugs + support and tiny feature requests). While micah is preparing the new server i will continue fixing bugs. Then we will start test deploys and customize crabgrass for the new we.riseup.net.

Next week i will send out a call for help with translations to the translation group. Once the new server is up we can probably use a hand with testing.

EXCITING!

Hi! When is the 0.6.0 version going to deployed on we.riseup.net? where can I find the beta deployment? how can I help with testing?

Hi Betta, thanks for asking.
Here’s my latest report:

I just released crabgrass 0.6.0. It has seen quite a bit of QA on the dev server by a number of people. Currently I am working on setting up the new server together with micah.

The process was delayed a bit by a number of factors:

• I fixed all the issues that came up during QA as i did not want to release with known bugs. We will probably still find enough in the month to come.
• I worked less ours than expected due to illness and family things.
• Micah has been busy keeping other riseup services available.

Never the less we should have the new server up and open for broad testing and customization soon. That will bring along a number of fixes to be included in 0.6.1.

I also updated the translation keys. They’ll remain frozen and I uploaded them to transifex. I’ll ask people for translations in the translators group on we.riseup.net. We have some great support from a french group that’s been using crabgrass a lot.

I’ll post some instructions for helping with testing and translations.

Cheers,
Azul

I’m a sophomore in college and computer science student interested in doing crabgrass development as much as I can (hopefully 10-15 hours a week). I just finished learning Ruby, and am now beginning to peruse the Rails documentation.

I’m particularly interested in developing new tools for consensus decision making within and between groups, but am happy to do back-end nuts and bolts stuff as well.

Is there anyway I can help?

tartar_socks, great to hear you’re interested. I’ll be very busy this week and the next. However the best way to get started probably is to setup crabgrass on your own machine for development. That’s also super helpful for us in order to verify our documentation. If you run into anything send me a message or open an issue on labs.riseup.net/code/projects/crabgrass

After that i would suggest picking a tiny little thing you would like to work on. That way you can become familiar with the dev process. For getting started i recommend forking crabgrass on github: github.com/riseuplabs/crabgrass-core. Once you have sth. to include – send us a pull request.

One example that comes to my mind would be implementing this feature request: labs.riseup.net/code/issues/8440 – a navigation thing that takes you back from the print view of a page.

Great – will work on this in the near future. I am having difficulty messaging you though – when I click on your name, it says the user is not found. Could you send me a message or something? I made a @riseup.net email account too of the same name

tartar_socks – it’d be great if you contributed! you can get on irc.indymedia.org #crabgrass for real time chatting. i think azul is there.

server and translation progress

The new server is moving along nicely. micah and me are working hand in hand there. We still need to finish the last symlinks and probably update the database one more time and then it should be ready for testing. I’ve added a bit of customization and it looks like guido will also be working on that.

There’s also progress with translations. The english original is on transifex and there’s a callout for translation. So far portuguese has been translated fully. French is at 97% or so. Please help translate so we can release crabgrass with a fully translated ui for as many languages as possible.

I’ve also taken a look at the current data on we.riseup.net and checked each record for validity. Turns out there were > 70.000 invalid records. Most of them page history entries for pages that have long been deleted. But also duplicate logins and group names and so on. I created scripts to clean them up. I’m not yet fully done but I want to migrate to the new server without invalid records. That way we can keep track of the new ones and fix the issues that lead to invalid records in the first place.

Great news about the work about the data validation and tools to clean things! I had suspected with we.r.n’s long history that this might be the case and I fully agree with your decision to clean things before moving to the new version and server. Another thing we’ve never really done with we.r.n is any sort of mysql tuning, slow query analysis, etc. As you/micah are migrating things if there are opportunities to look at this that would be cool. The new server is faster hardware, which always helps, but it’s always better if the code is more efficient. (on a someone related note, we still see a slowdown on current we.r.n at 1am PST due to backups, strategies for creating backups that don’t slow down the site and also capture data in a consistent way would be good to have)

Hi, when is the new version going to be in we.riseup.net? Is there any way to take a look at the new features in the new server?

Hi Betta,

The new server is ready and about to be tested. First round of testing will start with the riseup collective. I hope we’ll be able to expand the group of testers fast in the coming weeks. But that depends on how many issues they find.

Cheers,
Azul

Review and bug fixes under way

The new server is up and running. It’s been tested by the riseup collective for about a week and i fixed a bunch of issues that were found in that time.
I’m slowly growing the group of testers now so i can keep up with the reported bugs.
Send me a message if you would like to get involved and have not been invited yet.

Migration ahead but still unscheduled

The timeline for the final move is yet to be scheduled so that we have the capacities to get it done smoothly and fix things that might come up. I would say it’s a few weeks ahead. I will let you know here once a date has been picked.