Monthly Archives: October 2014

YADA (Yet Another DevOps Article)

yada-yada-yada-seinfeld-elaine-benes Yet another article about DevOps? When will it ever end? Well, I can say from experience, we are very far from a consensus on what it means. So, it will probably not end anytime soon. Much of that has to do with the fact that it is different for every company. Each company needs to determine it’s own definition. But, in order to reach that definition, you must have a framework to work from. When it works and works well, everybody is happy. When it doesn’t work, it can be very ugly. So, here is my attempt at some guidance as I see it from the trenches. Having been through several permutations of DevOps or DevOps-like migrations, I can just tell you what I have seen work and what hasn’t.

My Basic Tenets of DevOps

DevOps, or whatever you want to call it, is fundamentally a shift toward unifying the process of creating software across business units. To achieve this unification you must, absolutely MUST, have buy-in by all. There is no “throwing over the wall” in DevOps. From the time an idea begins to germinate, there should be representation from all parties. This is the place that my vision differs from many. I believe that more than just development and operations should be involved. I don’t see a world where some representative from product comes-up with an idea and hands it off to development and says “build-it” working in the new world of rapidly changing enduser expectations. I believe there needs to be a constant feedback loop. This is because, in the end, we are all the product owners. And must be aware of what is being delivered and what is being experienced by those that are interacting with what is being delivered.

Don’t get me wrong, I don’t believe the “everybody contributes equally to each part of the process” vision works. Especially once you start bringing-in non-technical representatives. The following are some basic tenets that I believe should be present on any team that wishes to carry-out this vision:

  • Product (or whomever is responsible for the curating of products in your organization) should be in close contact with development, architecture and operations from inception to…
  • Every member of the team should be empowered to say they see something going astray or they have concerns about some decision without any fear of shame or repercussions.
  • The technical members of the team should be capable of doing all critical pieces of the process. This is to facilitate an active self-support model. Not every member needs to know every aspect in depth. Just enough to diagnose a problem. They should then be empowered to take actions on that identified problem. Be it write a patch, test and deploy it. Or just raise it to someone on the team with more in-depth domain knowledge.
  • The team should be empowered to deploy their product to all environments. this needs further details as it is a touchy subject that must be evaluated on a case-by-case basis. But the fundamental concept must exist.
DevOps For Managers
 
I want to spend a little time on this subject. this can be a real hang-up to the adoption of DevOps in many organizations. But, I honestly believe that with the proper process and measures taken in advance, there should be no fear in giving a team the “keys to the kingdom” so to speak.
So, I do not advocate for most companies (many online start-ups may be exceptions to the rule) that you just allow all developers to check-in code and deploy it directly to production. You often hear developers and the like espousing “Continuous Deployment” and citing “Well, this is how Facebook does it”. [See their paper here] But, first I should say that because Facebook does this it doesn’t make it right for you. In fact, it makes it right for very few companies other than Facebook. Secondly, this is how Facebook deploys it’s front-end. Not the API that their front-end is built upon and that so many others have become dependent on being up. IF a user notices a glitch in the FB UI, they are not likely to squawk too loudly. If a large consumer of the FB auth API suddenly cannot authenticate their users, that will be noticed and may even make the news.
 
What I do, however, advocate, is empowering teams to do their own deployments. Don’t make this a silted process. This leads to a feeling of responsibility to “own your code”. Developers are much more likely to deliver quality code if they know they are going to be the ones deploying it and, ultimately accountable for any issues it may have. Empower your teams. Treat them like responsible adults that can make sound decisions about the fitness of a product for general release. While there may be some pitfalls along the way, I guarantee you they will return the favor by making you look very good.
 
DevOps For Team Members (The Flip Side)
 
So, up to now, I have been speaking generally. Or maybe more specifically to managers or team leaders. Now I want to speak to the teams directly. The implementers of the product. If you want the level of empowerment that DevOps requires, you had better care about what you put out there. There is no half-assing here. If you screw-up, there is no place to point the finger but right back at you. Here is some advice from the trenches to help you achieve that end-goal. Nothing too new. But way too often overlooked or brushed-aside.
  • Test Driven Development is your friend! Embrace it! Learn to love it! I, of all people, know full-well how hard this can be. And, to be completely honest, I still struggle with it to this day. But, I have developed my own rhythm for achieving TDD. It doesn’t fit anybody’s strict definition. But it works for me. Find what works for you. Just make sure you feel confident in them and their ability to flush-out issues.
  • Automate everything you can! Obviously, the tests mentioned above should be run on each check-in. If you broke it, fix it. Don’t ever go home with a broken build. (I know, all the mantras you’ve been hearing for years. But they’ve all become mantras for good reason.) But, go beyond this. Automate your deployments as well. Use tools like fabric, puppet, chef where available. Even better, look into containerizing your apps with a tool like docker. If you get to the point where you can deploy exactly the same code (or even better, the same container!) over-and-over, you will become more confident in your abilities to deploy to any environment any time. Make sure you also automate the rollback of these deployments. If something goes wrong you will be so grateful to have a quick and easy way to get back to the previous known good state. Also, until you are completely comfortable with your automation, practice in a development environment. I would expect, through the normal process of develop->test->develop you will get comfortable. this will also be a work in progress. But you will eventually get to a point where you feel comfortable. Comfortable enough to even deploy continuously. Or, more likely, much more frequently than you do now.
If you take the above steps. You will, with time, get to the point where the leaders in this arena are. And don’t wait for your organization to tell you to. Start now! there is no reason you can’t put everything in place. Get comfortable with it. Then present it to management. You can be the hero. Or, at the very least, you have prepared yourself to work at an organization that “gets it”.
 
A Benefit To All
 
The end result of all of this is that we will all benefit from this. Features, fixes and changes will get to production and in-front of clients sooner. this leads to a more immediate feedback loop that results in more changes that ultimately end in a better product.

Why am I 4everinbeta?

The_Leaning_Tower_of_Pisa_Small 4everinbeta is a nom de plume for me to express my thoughts on various subjects. Mostly technical, but not always. The title explains my philosophy of life in general. I will never be done. I’m constantly tweaking. Adding new features. Removing others that are no-longer useful. And, just like any beta product, I am chock-full of defects. This is reinforced everyday with my interaction with others. (My wife and child inform me of new defects everyday!) It is the other individuals in my life and my interactions with them both personally and virtually that uncover these defects. It’s my responsibility to track these defects and strive to fix them.

In my personal bug-tracking system I have several classifications as outlined here:

  • Sev1 – A complete show-stopper. If I encounter some aspect of my personality or behavior that intentionally or inadvertently results in physical, mental, emotional or other harm to another, I must take immediate action to resolve this. This is inexcusable.
  • Sev2 – If something that I do prevents myself or another from making progress toward a goal or aspiration, I must take corrective action as soon as possible. Blocking an individuals progress in life happens more than you may realize. When I realize I have perpetrated such an act, I take it upon myself to not only prevent it from happening again, but attempt to rectify any harm it may have already done. (Although this is often not possible)
  • Sev3 – This is some defect that may slow progress toward a goal or aspiration but does not cause that progress to come to a full-stop. Get to it as soon as you can reasonably. But keep your eyes on the goal.
  • Sev4/5 – Just annoying to myself or others. I’ll try to work on these because I really don’t want to annoy you. But, if I don’t fix it, deal with it. 😀

As you can tell from this list, “Do no harm” is an admirable goal in all aspects of life that I personally ascribe to. (And I feel the world would be a better place if that was just a part of the collective human psyche.) That said, you cannot focus entirely on your defects. You must keep growing. Pushing your boundaries beyond your comfort zone. Only by doing-so can an individual reach their full potential. As a direct result of this, you are inevitably going to make mistakes. And defects are going to be added to your tracker. But that is ok. Personally, I feel that at the end of my life I will be happy if my defect list is free of sev1 defects. And ecstatic if I have managed to eliminate all but the 4’s and 5’s. (Sorry folks!)