Developing client sites in the open

In this article we look at some of the challenges of developing client sites as Open Source

We're currently working with an existing client - a public policy group - to replace their main marketing website. It's a pretty standard project for us with lots of marking and business development content, along with some useful search integration. What sets this project apart for us, is their policy to make all new systems, including websites, developed by or for them to be Open Source by default.

Of course, we're not the first agency to develop Open Source software. We've made packages and contributed to the Open Source Umbraco project for over a decade. But this is the first time we've delivered a client site designed from the outset to be Open Source through the life of the project. It's required us to take a step back and approach the project from a different angle than we would usually take and has raised some interesting challenges along the way.

Our development standard is to use a source code repository. Fortunately our client  use our preferred repository tool, GIT. The downside is that because this project is open by design, the GIT repository is publicly accessible, meaning that anyone can browse and view the entire history of every change ever made. It follows then that anyone can make copies of the work and reuse it as they wish.

Depending on your point of view this can be hugely liberating, because all your work is out in the open and as such your progress, triumphs and challenges are there for everyone in the world to see. On the flip side this can be more than a little intimidating if you value your privacy as a dev.

The biggest practical issue for us, which is a departure from our normal workflow, is the fact that you can't open source everything, and so we need to pick and choose what can go into the repository.  

This impacts how we store sensitive content - those pieces of information  vital to the running of the system that can't be shared publicly - such as passwords and cryptographic keys - collectively called secrets. Normally we store these in config files in the repository and deploy them as part of our normal deployment processes.  Instead for this project these are now stored outside of the GIT repository. For internal development these files are managed in separate config files. However for deployment to external servers, they’re transformed using Configuration transforms by our client’s build server, Jenkins, and injected using Octopus deploy. This process allows our development team to manage configuration in one place outside of the GIT repository and still make it easy for our client to provide their deployment details with minimum fuss.

Our final challenge is that not all secrets are text based passwords and keys. Some of the Umbraco packages we commonly use, such as Umbraco Forms, Courier and SEO Checker, incorporate license files in the form of binary files stored in the /dll/ folder that unlock premium features. These valuable files would allow anyone to misuse premium packages paid for by us and for that reason can't be shared publicly in the repository. Jenkins and Octopus Deploy would also struggle to easily inject these. Our solution is to store them in a known location on each web server, and configure Octopus Deploy to run a Powershell Script to copy the files each time a deployment takes place.

What we’ve learnt is that opening a client site as public requires basics of good practice. Good comments and commit messages are fundamental -- no more blaming someone in a commit message. It's also important to ensure that the software is regularly patched and thoroughly security tested because making the software open allows hackers to identify attack vectors in the software.

Despite the challenges of Open Sourcing a client site, there are a number of significant benefits. Conversely the benefits of Open Sourcing a client site are significant - code quality improves when developers know they’re sharing their work openly; the cost of developing projects may reduce as services can work from a shared pool of Open services; and transition from one supplier to another is simplified when the code is written openly from the start.

Fundamentally coding in the open requires nothing but a sensible, mindful, approach to development following industry best practice.  While we don't think it will ever likely become the norm for client work, we have found it to be a useful exercise encouraging us to test and review our own processes and underlining our belief that one size doesn’t fit all.

Paul Marden

Paul is on Twitter as

Jamie Pollock

Jamie is on Twitter as