Hibri Marzook Musings on technology and occsionally photography

Continuous Delivery culture, not tools – Notes from an Open Space session

I facilitated an open space session at the Pipeline Conference in London, to discuss focussing on a  culture of Continuous Delivery (CD) than the tools. We listed a few of the anti-patterns seen in the wild

Culture not tools 

  • The CD guy.
  • Devops person
  • The team not owning CD.
  • Too much standardisation. One person writes the rulebook and forced on every team.
  • No culture of change. The delivery pipeline and code is fixed. No one touches it.
  • No culture of learning.
  • Too much focus on the technology.
  • Cherry picking practices and missing the connection between practices.

We then discussed how to communicate CD, without using  technical terminology, and use language that the rest of the organisation understands. Especially in terms that senior management can relate to.

Techies do a bad job of communicating the process of building software. Don’t mention TDD, CD, Devops when talking to management. Learn to talk in terms of business goals.  Adaptive thinking,  story telling and impact mapping are good tools to have.

Communication

Anthony Green described a pyramid of skills/terms to use when talking about CD.

Pyramid of skills

Techies start at the apex of the pyramid when talking about CD and doing a CD transformation. Instead we should take ideas from the human sciences to involve non-technical people in the process.

Individuals learn, but organisations are bad at learning. How to build a culture of learning across the organisation ? How does the organisation learn ?  In most organisations failure is penalised.

Learning

There were many suggestions to improve organisational learning.

  • Empowering, remove shackles.
  • No job titles. Titles restrict employees. However, titles are useful for HR. Is HR useful ?
  • Culture interviews.
  • Get better at recruitment. Pair programming interviews. Grow people.

We discussed a few techniques to learn agile practices without the tools and technology.  Agile games such as the Lean Lego Game and the Kanban Pizza Game help introduce the CD thinking without getting mired in technical discussions. [I facilitated an open space session at the Pipeline Conference in London, to discuss focussing on a  culture of Continuous Delivery (CD) than the tools. We listed a few of the anti-patterns seen in the wild

Culture not tools 

  • The CD guy.
  • Devops person
  • The team not owning CD.
  • Too much standardisation. One person writes the rulebook and forced on every team.
  • No culture of change. The delivery pipeline and code is fixed. No one touches it.
  • No culture of learning.
  • Too much focus on the technology.
  • Cherry picking practices and missing the connection between practices.

We then discussed how to communicate CD, without using  technical terminology, and use language that the rest of the organisation understands. Especially in terms that senior management can relate to.

Techies do a bad job of communicating the process of building software. Don’t mention TDD, CD, Devops when talking to management. Learn to talk in terms of business goals.  Adaptive thinking,  story telling and impact mapping are good tools to have.

Communication

Anthony Green described a pyramid of skills/terms to use when talking about CD.

Pyramid of skills

Techies start at the apex of the pyramid when talking about CD and doing a CD transformation. Instead we should take ideas from the human sciences to involve non-technical people in the process.

Individuals learn, but organisations are bad at learning. How to build a culture of learning across the organisation ? How does the organisation learn ?  In most organisations failure is penalised.

Learning

There were many suggestions to improve organisational learning.

  • Empowering, remove shackles.
  • No job titles. Titles restrict employees. However, titles are useful for HR. Is HR useful ?
  • Culture interviews.
  • Get better at recruitment. Pair programming interviews. Grow people.

We discussed a few techniques to learn agile practices without the tools and technology.  Agile games such as the Lean Lego Game and the Kanban Pizza Game help introduce the CD thinking without getting mired in technical discussions. ](https://twitter.com/matthewpskelton) is doing interesting work in this area, with a workshop to experience devops and a collaborative culture at http://web.experiencedevops.org/ .

Everyone should read [I facilitated an open space session at the Pipeline Conference in London, to discuss focussing on a  culture of Continuous Delivery (CD) than the tools. We listed a few of the anti-patterns seen in the wild

Culture not tools 

  • The CD guy.
  • Devops person
  • The team not owning CD.
  • Too much standardisation. One person writes the rulebook and forced on every team.
  • No culture of change. The delivery pipeline and code is fixed. No one touches it.
  • No culture of learning.
  • Too much focus on the technology.
  • Cherry picking practices and missing the connection between practices.

We then discussed how to communicate CD, without using  technical terminology, and use language that the rest of the organisation understands. Especially in terms that senior management can relate to.

Techies do a bad job of communicating the process of building software. Don’t mention TDD, CD, Devops when talking to management. Learn to talk in terms of business goals.  Adaptive thinking,  story telling and impact mapping are good tools to have.

Communication

Anthony Green described a pyramid of skills/terms to use when talking about CD.

Pyramid of skills

Techies start at the apex of the pyramid when talking about CD and doing a CD transformation. Instead we should take ideas from the human sciences to involve non-technical people in the process.

Individuals learn, but organisations are bad at learning. How to build a culture of learning across the organisation ? How does the organisation learn ?  In most organisations failure is penalised.

Learning

There were many suggestions to improve organisational learning.

  • Empowering, remove shackles.
  • No job titles. Titles restrict employees. However, titles are useful for HR. Is HR useful ?
  • Culture interviews.
  • Get better at recruitment. Pair programming interviews. Grow people.

We discussed a few techniques to learn agile practices without the tools and technology.  Agile games such as the Lean Lego Game and the Kanban Pizza Game help introduce the CD thinking without getting mired in technical discussions. [I facilitated an open space session at the Pipeline Conference in London, to discuss focussing on a  culture of Continuous Delivery (CD) than the tools. We listed a few of the anti-patterns seen in the wild

Culture not tools 

  • The CD guy.
  • Devops person
  • The team not owning CD.
  • Too much standardisation. One person writes the rulebook and forced on every team.
  • No culture of change. The delivery pipeline and code is fixed. No one touches it.
  • No culture of learning.
  • Too much focus on the technology.
  • Cherry picking practices and missing the connection between practices.

We then discussed how to communicate CD, without using  technical terminology, and use language that the rest of the organisation understands. Especially in terms that senior management can relate to.

Techies do a bad job of communicating the process of building software. Don’t mention TDD, CD, Devops when talking to management. Learn to talk in terms of business goals.  Adaptive thinking,  story telling and impact mapping are good tools to have.

Communication

Anthony Green described a pyramid of skills/terms to use when talking about CD.

Pyramid of skills

Techies start at the apex of the pyramid when talking about CD and doing a CD transformation. Instead we should take ideas from the human sciences to involve non-technical people in the process.

Individuals learn, but organisations are bad at learning. How to build a culture of learning across the organisation ? How does the organisation learn ?  In most organisations failure is penalised.

Learning

There were many suggestions to improve organisational learning.

  • Empowering, remove shackles.
  • No job titles. Titles restrict employees. However, titles are useful for HR. Is HR useful ?
  • Culture interviews.
  • Get better at recruitment. Pair programming interviews. Grow people.

We discussed a few techniques to learn agile practices without the tools and technology.  Agile games such as the Lean Lego Game and the Kanban Pizza Game help introduce the CD thinking without getting mired in technical discussions. ](https://twitter.com/matthewpskelton) is doing interesting work in this area, with a workshop to experience devops and a collaborative culture at http://web.experiencedevops.org/ .

Everyone should read](http://www.amazon.co.uk/gp/product/0712678867/ref=as_li_ss_tl?ie=UTF8&camp=1634&creative=19450&creativeASIN=0712678867&linkCode=as2&tag=hibrinet-21 “Maverick”) by Ricardo Semler.

Matthew also highlighted how we are good at spotting bad software architecture, but don’t spot queues and bottlenecks in organisational culture.  The sketch below would be recognised as having a bottleneck if it was a software system, but can we spot the bottleneck if this was an org chart ?

Queues

 

At the end, there was consensus that it all comes down to having good people.

Thanks to everyone who attended the open space session. Most of all to the conference organisers for putting together a well organised, and very thought-provoking event.

Example content

Howdy! This is an example blog post that shows several types of HTML content supported in this theme.

Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.

Curabitur blandit tempus porttitor. Nullam quis risus eget urna mollis ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.

Etiam porta sem malesuada magna mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.

Inline HTML elements

HTML defines a long list of available inline tags, a complete list of which can be found on the Mozilla Developer Network.

  • To bold text, use <strong>.
  • To italicize text, use <em>.
  • Abbreviations, like HTML should use <abbr>, with an optional title attribute for the full phrase.
  • Citations, like — Mark otto, should use <cite>.
  • Deleted text should use <del> and inserted text should use <ins>.
  • Superscript text uses <sup> and subscript text uses <sub>.

Most of these elements are styled by browsers with few modifications on our part.

Heading

Vivamus sagittis lacus vel augue rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.

Code

Cum sociis natoque penatibus et magnis dis code element montes, nascetur ridiculus mus.

// Example can be run directly in your JavaScript console


// Create a function that takes two arguments and returns the sum of those arguments

var adder = new Function("a", "b", "return a + b");

// Call the function

adder(2, 6);
// > 8

Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.

Lists

Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.

  • Praesent commodo cursus magna, vel scelerisque nisl consectetur et.
  • Donec id elit non mi porta gravida at eget metus.
  • Nulla vitae elit libero, a pharetra augue.

Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.

  1. Vestibulum id ligula porta felis euismod semper.
  2. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.
  3. Maecenas sed diam eget risus varius blandit sit amet non magna.

Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.

HyperText Markup Language (HTML)
The language used to describe and define the content of a Web page
Cascading Style Sheets (CSS)
Used to describe the appearance of Web content
JavaScript (JS)
The programming language used to build advanced Web sites and applications

Integer posuere erat a ante venenatis dapibus posuere velit aliquet. Morbi leo risus, porta ac consectetur ac, vestibulum at eros. Nullam quis risus eget urna mollis ornare vel eu leo.

Tables

Aenean lacinia bibendum nulla sed consectetur. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Name Upvotes Downvotes
Totals 21 23
Alice 10 11
Bob 4 3
Charlie 7 9

Nullam id dolor id nibh ultricies vehicula ut id elit. Sed posuere consectetur est at lobortis. Nullam quis risus eget urna mollis ornare vel eu leo.


Want to see something else added? Open an issue.

Richard Feynman’s observations on the reliability of the Space Shuttle.

I’ve been reading a lot on Richard Feynman lately. I find his character and his unique approach to learning appealing.

In the book “What Do You Care What Other People Think”, he reminisces about his time on the Rogers Commission investigation into  the Space Shuttle Challenger disaster. The book contains his appendix to the report. These are Feynman’s personal observations : Appendix  F – Personal observations on the reliability of the Shuttle

A few key points stood out to me that are relevant to how we build software.
  1. Becoming immune to small failures. NASA ignored minor errors, and modified their quality certification to account for these errors. NASA did this without investigating the systemic failures behind the errors.
  2. It didn’t fail in the past, therefore it will keep on working.
  3. Difference in culture. During the Apollo program, there was shared responsibility. If there was a problem with an astronaut’s suit, everyone was involved till the problem was solved. In the Space Shuttle era, someone else designed the engines, another contractor built the engines and someone else was responsible for installing the engines. They lost sight of the common goal. It was someone else’s problem.
  4. The Space Shuttle was built in a top down manner (big design up front ?). There was constant firefighting to keep it all working. The engines were rebuilt each time. Instead of a bottom up manner, using parts that were known and proven to work.
  5. His observations appreciates the efforts of the software engineering team though. Their testing was rigorous  and he wonders why other teams were not as good as them

What do tests tell us ?

Back in September 2012, I gave a talk at Dev’s in the Ditch on what tests tell us.  The talk resonated with a post by Matt Wynne today on fixing slow builds.

Optimising a slow build ? You are solving the wrong problem.

Slow builds and test suites with a large number of tests are an architecture smell. An indicator that there is something wrong with how you have built the application. I covered a few other things that tests or the lack of them tell us, in my talk.

How we do deployments at 7digital.

At 7digital, deployments to a production environment are a non-event. On a given day, there can be at least 10 releases to production during working hours. On some days even more. Specially on Thursdays before the 4pm cut off, as we don’t deploy to production after 4pm and on Fridays.  

Deployments to our internal testing environments happen constantly, on every commit. I’m able to deploy a quick fix, or patch something in production without having to make changes on a live server. We rarely roll back, instead roll forward by fixing the issue. This is made possible due to our investment in build and deployment tools. We attempt to treat these with the same care as our production code.

This post is about how it works.

A little bit about our stack.

Our services run on the .Net stack, with SQL server back ends mostly. We use IIS 7and IIS 6, load balanced  behind HAproxy.

We use Teamcity, to trigger Rake scripts, to do our deployments. The Albacore gem is used for the majority of tasks. We use code derived from Dolphin deploy to configure IIS.

 

In the beginning.

We used MSBuild for building our solutions and deploying software. However, this was very painful, and led me on a personal crusade to get rid of Msbuild for deployments.  XML based build frameworks, limit what you can do to what is defined in the particular framework’s vocabulary. A big pain was having to store configuration and code in the same msbuild xml files. It wasn’t possible to separate the two without writing custom tasks.

A build framework, in a programming language, allows you to be much more fluent and write readable scripts. You have the power to use all the libraries,  at your disposal to write deployment code instead of being limited to a XML schema definition. In addition to Ruby, we have  a couple of projects using Powershell and psake.

 

The current setup.

 

deployment

 

The diagram above shows the major parts of our deployment pipeline.

We keep the build and deployment code along with the project code, to maintain a self contained package, with minimal dependencies on anything else.

A project has a master rake file, named rakefile.rb in the root directory of the project. This rake file references all the other shared rake scripts and ruby libraries needed for build and deployment.

These libraries and scripts are kept in a sub directory named build. A typical project structure is like;

root
    build
        conf
        lib
   src
       XX.Unit.Tests
       XX.Integration.Tests
       XX.Web

The conf directory contains the configuration settings for IIS, including the host headers, app pool settings and .net framework version settings.

The Albacore build gem has everything that is needed to build a .Net solution. We use it to compile our code on Teamcity and to run our tests.

When something is checked into VCS (git), Teamcity triggers off a build and compiles the code. This build process will package the deployment scripts and the web site package, which will be used for deployment. Teamcity stores these as artefacts, and this allows us to reuse them without building again.

To deploy a website, a Teamcity build agent, retrieves all necessary zipped packages, un-compresses them to the current working directory.

The build agent calls a rake task, with the parameters;

   rake deploy[environment, version_to_deploy, list_of_servers, build_number]

An example

   rake deploy[“live”,”1.2”,”server1;server2;server3”,”123”]

The environment parameter specifies which deployment settings to use.  Deployment settings are stored in  YAML files, that the rake scripts read.  A YAML file for IIS settings looks like;

uat:
 site_name: xxx.dns.uat
 host_header: 80:xxx.dns.uat:*
                        443:xxx.dns.uat:*
 dot_net_version: v4.0

live:
 site_name: xxx.dns.com
 host_header: 80:xxx.dns.com:*
                        443:xxx.dns.com:*
 dot_net_version: v4.0

 

We can add a new environment, and change settings for an existing environment by changing a configuration .yml file, without having to change deployment scripts.

The version_to_deploy parameter, loosely translates to a virtual directory. This is ignored for websites that deploy to the root. The list of servers is an arbitrary list of servers that we deploy to. This allows us to deploy to a single server or a cluster.

The rake deploy task, calls two other rake tasks, for each server in the list of servers. The first task is to copy all deployment scripts and the web package to the target server. The second is to trigger a remote shell command to do the actual installation process.

In pseudo code

  _deploy

       foreach(server in servers) </p>

             copy scripts and packages to server

             trigger remote installation</em> on server            

The actual installation process does not happen from the build agent, but on the target server. The build agent does not have the the necessary network access and admin rights. Our servers expose only SSH.

The deployment sequence is controlled by chaining rake tasks. This allows us to run any of the tasks individually from the command line to do a manual deployment or to test.

The remote installation task, copies all the web site binaries to the correct locations under IIS, and configures IIS. Application pools under IIS are stopped, while this happens, and the virtual directory and if needed the web site are rebuilt. The application pools are restarted after this.

The deployment  repeats the process on the next server in the list.

 

The future.

What we have now helps us a lot, and allow us to scale up to this point. However, to grow even more there are a few things that hold us back.

For example, a lot of infrastructure details creep into our configuration and scripts and stored in source control, which is mostly used by devs. This means that  when our operations folk make a change to the infrastructure the devs have to change our configuration settings to reflect this. I would like to have all configuration settings stored somewhere, and the scripts would call out to a service to get all the settings for a particular environment and application. This service would be maintained by devops, and will be synchronized with changes made to the infrastructure.

The same can be done for the list of servers. Instead of a developer having knowledge of what servers comprise an environment, the script could ask the same service, to give a list of servers that are in a given environment. This will allow us to scale transparently, by adding a new server to the list and doing a fresh deploy.

 

Summary.

I’ve tried to capture an overview of how we deploy our software at 7digital. There is a lot of detail I haven’t gone into. Especially the nitty gritty of setting up IIS host headers, ports and app pool settings. A build and deployment framework is something we do from day one of any new project. We make sure that we have a skeleton application deployed all the way to production before any new code is written.

Feel free to get in touch if you have any specific questions.

Resources:

http://codebetter.com/benhall/2010/10/22/dolphin-deploy-deploying-asp-net-applications-using-ironruby/

http://albacorebuild.net/