Hibri Marzook Musings on technology and systems thinking

My NodeJS development workflow

I’m building a NodeJS + Angular application and there are a few things I want a smooth dev workflow and improve developer happiness. Yay !!

  1. Run the node server process and restart automatically when I make changes
  2. Concatenate my client side JS code, and merge libraries such as angular, jquery and bootstrap.
  3. I don’t want to check in concatenated files.
  4. I don’t want to check in modules or libraries, even for client side scripts
  5. I want to catch typos and style violations.
  6. I want tests to run in the background.

Grunt is the task runner I’m using to automate the tasks. Let’s look at each of these tasks.

Automate the NodeJS server process with nodemon.

npm install --save-dev nodemon

Now, run

nodemon

Eventually, you’ll want to trigger nodemon through grunt. Get the grunt-nodemon plugin. The plugin registration and the Grunt config is below.

grunt.loadNpmTasks('grunt-nodemon');

nodemon: {
    dev: {
        script: 'bin/www',
        ignore:  ['node_modules/**','bower_components/**','public/**']
    }
}

I’ve configured nodemon to ignore javascript that is not my code, and ignore changes to client side javascript. The path to the node server script is set in the script property. Since I’ve used express to create an app, this is ‘bin/www’.

Great, now executing grunt nodemon  will watch for changes and restart the server.

Next, I want to avoid all those pesky typos, misplaced semi-colons and get into the habit of writing good Javascript. I’ll use JSHint.

npm install --save-dev grunt-contrib-jshint

Configure JSHint as below,

grunt.loadNpmTasks('grunt-contrib-jshint');

jshint: {
    all: ['Gruntfile.js', 'lib/**/*.js', 'test/**/*.js','public/javascripts/app/**/*.js']
}

This configures JSHint to run on my tests, client side javascript and server code.

As with nodemon, I want JSHint running in the background and checking my code as I type and save files. This gives me quick feedback. Because JSHint doesn’t do this, I’ll introduce another plugin to the Gruntfile.

This is the grunt watch plugin, which will watch for changes in my source directory and run other Grunt tasks.

npm install --save-dev grunt-contrib-watch

Configure grunt-watch as below,

grunt.loadNpmTasks('grunt-contrib-watch');
watch: {
 scripts: {
 files: ['**/*.js','!node_modules/**','!bower_components/**','!public/javascripts/dist/**'],
 tasks: ['jshint']
 }
}

This sets up grunt-watch to watch over all of my code. Ignore directories by placing an exclamation mark before the path spec.

Great, now if I execute grunt watch on the command line, it will start watching for changes and run jshint when a change is detected. Sweet.

There is a catch though. I want to run nodemon and watch at the same time. I could run both in separate shells, but that would mean switching back and forth.

Let’s introduce grunt-concurrent. This executes grunt tasks concurrently. Now I can run nodemon and watch at the same time.

npm install --save-dev grunt-concurrent

Configure as below;

grunt.loadNpmTasks('grunt-concurrent');


concurrent: {
    dev: {
        tasks: ['nodemon', 'watch'],
        options: {
            logConcurrentOutput: true
        }
    }
}

The config above, tells grunt-concurrent to run nodemon and watch. Both tasks are grouped logically under a “dev” subtask.

If I execute grunt concurrent on the command line, I’ll have my server restarting automatically and JSHint watching my code. To reduce what I need to type even more, I’ll set up grunt to run this task as default

grunt.registerTask('default', ['concurrent']);

All I need to do is type “grunt” and have it do all the work done for me.

This gives me a  basic framework to add other tasks. Grunt allows me to group tasks. For example, I want to group all of my build time tasks as one. My build time tasks are to run jshint, run tests, merge libraries like angular and concatenate client side javascript.

grunt.registerTask('build', ['jshint','karma','concat','bower_concat']);

Instead of watch running JSHint only, let’s change it to run the build task.

watch: {
 scripts: {
 files: ['**/*.js','!node_modules/**','!bower_components/**','!public/javascripts/dist/**'],
 tasks: ['build']
 }
}

Here is the complete Gruntfile https://gist.github.com/hibri/6ccd0aef805353dc8260

In addition to the tasks described above, I’ve also used the plugins listed below

  • grunt-bower-concat : merges all the dependencies installed via bower
  • grunt-contrib-concat : merges all the client side javascript I’ve written. This includes all my angularjs code.
  • grunt-karma : To run tests
  • grunt-bower-task : To run bower install through grunt.

You’ll also notice I’ve got the following task configured.

grunt.registerTask('heroku:production', ['bower:install','concat', 'bower_concat']);

The deployment to Heroku uses the task. I want the concatenation to run when I push to production, so I don’t have to check in any build artefacts to source control.

Heroku doesn’t run Grunt natively during a deploy, but they do support build packs. I’ve configured Heroku to use a nodejs buildpack with grunt support.

See https://github.com/mbuchetics/heroku-buildpack-nodejs-grunt for more.

This is the build framework I’m starting with, and most certainly will grow with the project. A way of sharing ignored files between tasks would be nice. There is some duplication already.

What’s your workflow ?

The Fuji X100T – First Impressions

I love gadgets. Cameras even more. Over the past decade, I’ve gone through quite a collection of photography devices. Starting with a Canon EOS 350 film camera, which I still have and up to a 5D, and even an old Polaroid SX-70.

However, none of these cameras created an emotional bond, or felt like that they were made for me. Heck, I think the 5D is a brute who can make great picture of anything I throw at it.

That changed when I got the Fuji X100. I bought it as a lightweight travel camera. It changed how I approached photography. I wasn’t worried about the perfect settings anymore. The tactile controls of the X100 gave a physical element to creating photographs.

The X100 was a difficult camera though. It was moody and needed coaxing to make the best of the available light. The auto focus was slow, and the manual focus was, unusable. The camera did deliver when I’ve had the patience.

In Arles, France, earlier this summer, I learned the art of shooting from the hip. Inspired by the excellent book, with the same name by Johnny Stilletto. I needed a camera that is responsive as a natural reflex, to capture the scene. A camera that encourages a minimalistic style of shooting.

The Fuji X100T shows maturity of the X100 line. The X100 and the X100s were brave steps towards making cameras beautiful again. Naturally, I placed a pre-order for the X100T and when it arrived took it out for a little spin this weekend.

The auto focus is fast. The start up is fast. The camera is ready by the time I lift it up to my eye. The optical view finder (OVF) has wider coverage compared to the X100. The electronic view finder (EVF) is responsive with a crisper view. It doesn’t lag. Previously the tiny bit of lag was enough to mess my composition. The view finder has a tidy layout, with all the indicators outside of the frame. There is not much that gets in the way of composing a scene. In addition I’ve been turning off all but the essential indicators.

The manual focus is usable. The focus ring turns smoothly but still as not as smooth as on a EF lens. It still has little bit of a tactile disconnect between what I do and what I see. The 1/3 aperture step change on the control ring is a welcome change. Previously,I didn’t like using the command dial to change to f3.5. The buttons on the back give the same solid feedback as the rest of the control dials. Control consistency is great. On the X100, the dial was a tad hypersensitive and I’ve switched to the wrong shooting mode accidentally, on many occasions.

I love the Wifi connectivity and remote shooting capabilities. I’m using an iPad for my editing workflow, and this will be excellent while travelling. I can make do with fewer SD cards now.

I’m not entirely sold on the film simulations, and have rarely used it. I’m impressed with how little post processing I’ve had to do with what I’ve shot on the X100, even in RAW mode. The X100T can shoot in a square aspect ratio, in camera. This is great for photos that are posted to Instagram later.

I doubt the reliability of the battery level indicator on the X100T. It showed that the battery was almost empty, but after turning it off and on again it went back to showing that it’s 75% full. I expect the battery to last at-least for a whole day’s worth of shooting. I have to question why Fuji doesn’t include an adaptor ring and a lens hood with the camera, considering it’s a premium compact.

Overall, I love how the X100T responds to my instinct and doesn’t stand in the way of snapping up what’s happening around me. I’ve never been this excited to use a camera. Well, at least till the next one comes out.

 

 

Accepting Uncertainty

This is one of those things that should be filed under continuous delivery anti-patterns. One of the anti-patterns I see in continuous deployment, is the need to make sure that the software is flawless before it’s released. This anti-pattern manifests itself, in the need to have several stakeholders test and approve a release. The release dance takes days, when something is good enough to be on a production system.

There is a large suite of long running tests that must be run, before a release is approved. There is a release management dance that must be done before a release. Cue the constant conversations around the office about the impending release.

Instead, let’s accept the inherent uncertainty in building software. Any non-trivial system is complex. We can’t predict it’s behaviour to any accuracy. We can only observe and react to any unexpected behaviour.

This is the key tenet in building a continuous deployment pipeline. The ability to react to uncertainty.

Chasing certainty with more automated tests will only give diminishing returns. There should be enough tests to increase the confidence level. That’s it. Nothing more.

The rest comes from observing how the software behaves. By monitoring and gathering data. Use that,  to react. Add a few more tests. Rinse, repeat. Iterate.

The ability to iterate and react will give better quality software in the long term, than a stranglehold with tests and testers.

 

Continuous Delivery culture, not tools – Notes from an Open Space session

I facilitated an open space session at the Pipeline Conference in London, to discuss focussing on a  culture of Continuous Delivery (CD) than the tools. We listed a few of the anti-patterns seen in the wild

Culture not tools 

  • The CD guy.
  • Devops person
  • The team not owning CD.
  • Too much standardisation. One person writes the rulebook and forced on every team.
  • No culture of change. The delivery pipeline and code is fixed. No one touches it.
  • No culture of learning.
  • Too much focus on the technology.
  • Cherry picking practices and missing the connection between practices.

We then discussed how to communicate CD, without using  technical terminology, and use language that the rest of the organisation understands. Especially in terms that senior management can relate to.

Techies do a bad job of communicating the process of building software. Don’t mention TDD, CD, Devops when talking to management. Learn to talk in terms of business goals.  Adaptive thinking,  story telling and impact mapping are good tools to have.

Communication

Anthony Green described a pyramid of skills/terms to use when talking about CD.

Pyramid of skills

Techies start at the apex of the pyramid when talking about CD and doing a CD transformation. Instead we should take ideas from the human sciences to involve non-technical people in the process.

Individuals learn, but organisations are bad at learning. How to build a culture of learning across the organisation ? How does the organisation learn ?  In most organisations failure is penalised.

Learning

There were many suggestions to improve organisational learning.

  • Empowering, remove shackles.
  • No job titles. Titles restrict employees. However, titles are useful for HR. Is HR useful ?
  • Culture interviews.
  • Get better at recruitment. Pair programming interviews. Grow people.

We discussed a few techniques to learn agile practices without the tools and technology.  Agile games such as the Lean Lego Game and the Kanban Pizza Game help introduce the CD thinking without getting mired in technical discussions. 

Everyone should read “Maverick” by Ricardo Semler.

Matthew also highlighted how we are good at spotting bad software architecture, but don’t spot queues and bottlenecks in organisational culture.  The sketch below would be recognised as having a bottleneck if it was a software system, but can we spot the bottleneck if this was an org chart ?

Queues

 

At the end, there was consensus that it all comes down to having good people.

Thanks to everyone who attended the open space session. Most of all to the conference organisers for putting together a well organised, and very thought-provoking event.

Example content

Howdy! This is an example blog post that shows several types of HTML content supported in this theme.

Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean eu leo quam. Pellentesque ornare sem lacinia quam venenatis vestibulum. Sed posuere consectetur est at lobortis. Cras mattis consectetur purus sit amet fermentum.

Curabitur blandit tempus porttitor. Nullam quis risus eget urna mollis ornare vel eu leo. Nullam id dolor id nibh ultricies vehicula ut id elit.

Etiam porta sem malesuada magna mollis euismod. Cras mattis consectetur purus sit amet fermentum. Aenean lacinia bibendum nulla sed consectetur.

Inline HTML elements

HTML defines a long list of available inline tags, a complete list of which can be found on the Mozilla Developer Network.

  • To bold text, use <strong>.
  • To italicize text, use <em>.
  • Abbreviations, like HTML should use <abbr>, with an optional title attribute for the full phrase.
  • Citations, like — Mark otto, should use <cite>.
  • Deleted text should use <del> and inserted text should use <ins>.
  • Superscript text uses <sup> and subscript text uses <sub>.

Most of these elements are styled by browsers with few modifications on our part.

Heading

Vivamus sagittis lacus vel augue rutrum faucibus dolor auctor. Duis mollis, est non commodo luctus, nisi erat porttitor ligula, eget lacinia odio sem nec elit. Morbi leo risus, porta ac consectetur ac, vestibulum at eros.

Code

Cum sociis natoque penatibus et magnis dis code element montes, nascetur ridiculus mus.

// Example can be run directly in your JavaScript console


// Create a function that takes two arguments and returns the sum of those arguments

var adder = new Function("a", "b", "return a + b");

// Call the function

adder(2, 6);
// > 8

Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa.

Lists

Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Aenean lacinia bibendum nulla sed consectetur. Etiam porta sem malesuada magna mollis euismod. Fusce dapibus, tellus ac cursus commodo, tortor mauris condimentum nibh, ut fermentum massa justo sit amet risus.

  • Praesent commodo cursus magna, vel scelerisque nisl consectetur et.
  • Donec id elit non mi porta gravida at eget metus.
  • Nulla vitae elit libero, a pharetra augue.

Donec ullamcorper nulla non metus auctor fringilla. Nulla vitae elit libero, a pharetra augue.

  1. Vestibulum id ligula porta felis euismod semper.
  2. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus.
  3. Maecenas sed diam eget risus varius blandit sit amet non magna.

Cras mattis consectetur purus sit amet fermentum. Sed posuere consectetur est at lobortis.

HyperText Markup Language (HTML)
The language used to describe and define the content of a Web page
Cascading Style Sheets (CSS)
Used to describe the appearance of Web content
JavaScript (JS)
The programming language used to build advanced Web sites and applications

Integer posuere erat a ante venenatis dapibus posuere velit aliquet. Morbi leo risus, porta ac consectetur ac, vestibulum at eros. Nullam quis risus eget urna mollis ornare vel eu leo.

Tables

Aenean lacinia bibendum nulla sed consectetur. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Name Upvotes Downvotes
Totals 21 23
Alice 10 11
Bob 4 3
Charlie 7 9

Nullam id dolor id nibh ultricies vehicula ut id elit. Sed posuere consectetur est at lobortis. Nullam quis risus eget urna mollis ornare vel eu leo.


Want to see something else added? Open an issue.