Today I’ve attended to the ShipItCon 2019 conference hosted for a second year in Dublin. The speakers mainly talk about how they and their companies deliver software into production and what practices and tools they use.
I’d like to highlight the two talks I liked most. Although they don’t expose the newest tech I think is interesting how understanding a few principles could help you to prevent errors in production and help to deliver software with shorter feedback loops.
The first one is What Breaks Our Systems: A Taxonomy of Black Swans by Laura Nolan @lauralifts. Her presentation exposed a few major outages that big companies had and how they could have been avoided.
Soft and hard dependencies
She explained that most of the time when a new dependency is introduced in the software, we the developers, don’t think enough if that dependency should be hard or a soft one. For instance, she gave the example of a system that relies on an external dependency at initialization time could cause the unavailability of our software if measures are not put in place. Developers should take into account when adding new dependencies and thinking if the system can work without the dependency, and if it can, the software should not fail if the dependency is not available.
Normally when a system talks to other systems problems like connectivity and network issues might happen. A few alternatives might be retries with exponential backoff and the circuit breaker pattern. A lot of errors as oversized queues or exhausted systems might happen because retries are not well implemented. Ignoring problems in this area might lead to serious systems outages.
The second talk Immutable Deployments – Implementing a wicked fast deployment strategy by @grandazz is about practices to deploy software into production.
He talked about how the team he is part of uses feature flags for every single feature they put in production. How features flags have changed the way they make software and how their codebase is better than before having always that constraint in mind.
He categorised the feature flags in two categories, business and ops feature flags. The first one with a short duration in time and used by business people to decide what feature activate and when. The last is own by developers enabling them to make refactors or even deactivate features on demand.
Last but not least he showed how feature branches go directly to production without exposing the functionality to the final user. That allows them to make tests in production. They have also in place a way to rollback quickly because the application allows them to change between different versions in case of any problem.