Ben Newton

In Defense of Rugged DevOps

Posted by Ben Newton May 31, 2012
Share: |


As I have been watching the Rugged DevOps discussions unfold over the last few months (and contributed my own small part), it has been interesting to see the different directions the discussion takes.


The need to consider security architecture and design in the agile release process is becoming clearer to the community. There is clearly a ways to go, however, and the discussion and evangelism should continue.


So, I find myself a bit confused with the "White Rabbit's" objections to the term "Rugged DevOps". The Rabbit, and others of like mind, object to the term "Rugged" or "Sec" (as in DevOpsSec) based on the belief that the phrase implies that security needs to be called out separately and even treated separately.


In the Rabbit's own words - "It perpetuates the notion that security principles are something that is apart from the development and deployment cycles and that security is something that is an add-on".


In short, I think the Rabbit is missing the point. I'll use an analogy to make my point - driving in your car. Your chance to die in a car crash is 1 in 88, so I'd say, car safety is a problem.


So, when I talked to my relatives about the craziness on I-95 over the Memorial Day weekend, the term "safe driving" immediately focused our minds on one particular part of the driving experience. Conversely, I might have used the terms like "reckless driving" or "driving in traffic" to focus on another part of the driving experience. Clearly my relatives recognized these terms as my attempt to focus the discussion on one aspect of driving, not the implication that "safety" is not intrinsic to driving overall.


So, back to the Rabbit. "Rugged DevOps" is a literary device that allows the community to focus some discussions around the importance (or lack thereof) of security in DevOps. If the majority of developers were developing with security in mind, the topic would be moot. That is clearly not the case. Also, the term obviously does not minimize the larger DevOps term. In reality, it elevates DevOps. If the concept of Rugged becomes an "assumption" within DevOps, then DevOps is better for the side-discussion.


So, my advice to the Rabbit. Don't worry about it :). DevOps still stands as the larger discussion, and "Rugged DevOps" only serves to strengthen DevOps, not weaken it.

Share: |


This is the fourth excerpt from the podcast with John Vincent.




Tom Parish:  What comes next after the current crop of tools?


John Vincent:  I think the big one is: we’re starting to see inefficiencies in traditional monitoring. That’s a big one.  You know we have traditionally operated under a polling-based mechanism for monitoring.


Tom Parish:  Right.



John Vincent:  We’re starting to go: when you have a 1,000 nodes, a poll doesn’t work. So we’re starting to see resources that are ephemeral come up and while they're up, send their state back.  But now we’ve got all of this information and as much as another buzzword is Big Data is we’re in that same kind of situation in our infrastructure.



We’ve got lots of data points and we need to make sense of those. And I think one of the things that’s going to be as an adjunct to improved monitoring efficiency is actual event correlation and sussing out the knowledge of: what do all these data points mean. 


We’re all going to become – a reference to Milton Freidman here, but you know we’re all data scientists now.



Tom Parish:  Yes.



John Vincent:  So we need a way to suss that out and figure out what does this data point mean.  Is it something we can ignore?  Is it indicative of a bigger problem?  And I think the other thing is, we’re going to start seeing, this is actually happening now, but a focus on orchestration.



That’s an overloaded term as well but we’re really talking about, it’s not important how many servers you have.  It’s not important what those servers are running. It’s really important about what the collective is doing as a service.  The end user doesn’t care, if you're talking about web operations, they don’t care that you have five out of six servers running. If the site’s slow, the site is slow.



Tom Parish:  Yes.



John Vincent:  And you’ve got these tendencies between different parts of your infrastructure, you know, not just traditional 3-tier level -- you know, web server, app server, database -- but now as people are revisiting or rediscovering what is called SOA, you’ve got internal stacks and tiers of applications that have dependencies on each other.  And while we were moving towards more resilient engineering, by adopting that route there is still a shim of a tendency between those and you need to coordinate that information.



I think right now we’re addressing the monitoring issue as an industry and once we get that data we’re going to see that we need to correlate that.  And then we’ve also as we move toward more of a services-based approach even internally, akin to what Amazon has done, you’ve got coordination and orchestration needs there.





Next excerpt from John is on security and DevOps: get everyone involved early. . .

Share: |

Kris Buytaert banner.jpg Buytaert is an accomplished open source jedi master, based in Europe, who is currently working for Inuits.  He has a deep history with the DevOps movement. [Curiously, I don't find he's all that well known in the states but he should (and eventually will) be.]


You can follow him on twitter (@KrisBuytaert).


He has a great sense of humor. Btw, the opening snippet which occurred very early in the podcast itself, is a tweak on Kris' "tagline" that "everything is a freaky DNS problem".


As usual, the format is: we work out a list of questions to run through in advance. This is designed to promote an open and planned discussion of DevOps issues with the leading minds on the topic. After we agree to a set of questions, Tom -- our official BMC podcast host -- goes through the list. Sometimes it is a testament to Tom's professionalism that he can segue from one question to the next...


So Tom spent some time and had a chat with Kris. This is my first excerpt from the transcript of that podcast. More to come!


[NB: The podcast interview with Kris Buytaert will be published separately. Last I checked, it's in the queue and should be available in a week or so. I will provide a link to it here.]











Tom:  Today, I’m talking to Kris Buytaert, co-founder of one of the leading open source consultancy groups in Europe.  He’s also one of the original instigators of the DevOps movement and the author of over 25 papers, articles and books.


Kris:  (interrupts Tom) So I have a question for you.  Do you know what DNS stands for?



Tom:   Domain Naming Service.



Kris:  No, actually, that’s wrong.  It means DevOps Needs Sushi.



Tom:   Oh, you got me there.  Of course, it would be DevOps, right. 



Kris:  Yes.





Tom:   You did that on purpose, didn’t you?



Kris:  I did that on purpose.




Tom:  So Kris, where do you see DevOps going over the next two, three years?



Kris:   So in order to see where it’s going, I think you first need to look at where it’s standing now.  And if I look at where DevOps is today, it reminds me a lot of when I started doing open source consultancy over a decade ago. 



There was a lot of grassroots people doing it.  There was a lot of people talking about it.  It was a really cool thing to do.  But there weren’t that many organizations that were actually implementing it, except for the really original ones. 



It’s to me, kind of that – well, what do you call it, on the edge of going really mainstream?



Tom:   Yes. 



Kris:  Where the big consultancy groups, the big analysts.  They’ve all started writing about it.  And you feel there’s a lot of people doing stuff around it. And you also feel that it’s – it’s grown beyond the initial couple hundred people that were talking about it. It’s grown to a level where there’s now new, bigger organizations saying, “Hey.  This is cool.  We’re doing this too.”



And it’s also grown to a point where, just like open source six or seven years ago, people are abusing it as a marketing terminology.  So they try to attach the DevOps terminology to everything they sell and hope it’s going to help them in visibility in markets.



So it’s – it’s to the point where it’s becoming really interesting. 



If I look where it’s going, it’s going to grow.  It’s going to grow fast.  I’m already hearing of customers thinking about spending multi-million dollar in euro projects on improving their DevOps ideas in their organizations.  But I also think it’s going to be a commonality in a couple of years.  Basically, DevOps is – it’s nothing new. 


It’s just a lot of things that lots of us people have been doing already.  It’s common sense.  It’s being good at your job.  So it’s not really something that’s going to be standing out like everybody’s going to talk about this DevOps thing.  But it’s going to be there. 



Tom:   Yes.  Well. So I have a classic question. I've asked some of the other DevOps guys this kind of thing in variousways.  I'm real curious how you answerit.  If all developers were sysadmins and all the sysadmins were developers, would we even have a need for DevOps at this point?



Kris:  Well, it's not just sysadmins.  It's much more whole organizations.  It's about breaking down barriers and tearing down walls between different departments.



Tom:   Yes.



Kris:  And developers as far as operations, that's where probably most visible. Those are the traditional types that are talking to the developers and they write so-called code. 



But it's also about guys with DBAs. It's about marketing people versus project management – the product managers.  And it's about an organization improving itself and increasing the level of communication between the different groups and basically breaking down all the walls so that when you go to market, when you go to production, everybody in the organization is looking in the same direction, knowing which metric they are talking about. And they have common topics to discuss. 



So it's not just developers versus operations.  It's much more. But not everybody can be good at all things. 



So it wouldn't be just developers doing also operations work and operations people doing also something good at development, then we'd be lacking stuff like graphical people. We'd be missing out on the security and the network people. 



It's about everybody communicating with each other.  Not just developers and operations.



Tom:   Yes.  Yes. That makes sense.  All right.  So let's step back for a little bit.  How old is DevOps?



Kris:  Pretty old.  I think it's really old.  The name of course, DevOps I think – the DevOps name, which started getting popular I guess somewhere back in – summer in 2009, when Patrick announced DevOps Days as a conference and when the word was being used by people like Charles Fall and Andrew Clay Schafer and John Willis was talking about it. 



And then the name was out there. But basically it was already before that.  I've been doing DevOps-like stuff for probably somewhere 2003, 4, 5.  So that's probably almost 10 years. 



And the actual ideas – if you look at one of my slides in my presentations, I give a set of values.  And it's page full of key ideas, how organizations should work.  And it's also stuff we've already been discussing of DevOps Days on how the philosophy should work.  It could be looked at as the DevOps manifesto.  And I trick people into thinking that.  And it's actually a summary of out of the classic book written by Deming somewhere in 1986. 



So that's – that's ages ago. That's decades ago.  There's really nothing new.  But with the terminology DevOps coming into mainstream, it now gives us a common place and a common denominator to talk about the topics we care about.  About how do you do deployments.  How do you care about operations.  How do you scale environments.  Those topics now have a common name to tackle.  So we now have an umbrella under which we can talk about lots of those technologies and lots of things culturaland ideas. 



So the name itself?  About three years, going on four.  .  .





Coming up next is our second excerpt from the podcast with Kris, he talks about Infrastructure as Code, Monitoring, and DevOps Resonance.


Other related on-line materials:

When the Kris Buytaert podcast is published, it will be here.


Excerpt A: DevOps Needs Sushi & Mainstream DevOps

Excerpt B: Infrastructure as Code, Monitoring, and Resonance

Excerpt C: Packaging Religious Wars, Maturity & the End of Tradition

Share: |


This is the third excerpt from the podcast interview with John E Vincent in the on-going BMC DevOps Leadership Series. John makes a couple of important conceptual points in this excerpt IMO.





Tom Parish: OK, so let's look ahead, where do you see DevOps going over the next two to three years?



John Vincent:  I think the biggest thing is people are going to realize that DevOps is not about development and operations.



Tom Parish:  Yes. Despite the name.



John Vincent:  Yes, despite the name. It's a catchy name. It works.  And right now the biggest roadblock that organizations face is their interface with the operations team.  But as that gets smoothed out, I think the next big one you're going to run into is security.  Your SecOps team is going to have its own concerns.



And I think eventually people are going to realize the issue is not necessarily the interface between developer and operations, it is the interface between IT and the business.  And then trying to possibly realize the goals that sort of agile started in aligning IT with business. 



I think another one is that you're going to start seeing more hardware become programmable, exposing things that were APIs. And that's going to cut further the delay between provisioning and resources.  You know one of the things that people say “DevOps is sort of born in the Cloud” and there's a bit of truth to that in that when the artificial barrier to getting resources online is removed -- the one around six weeks provision and purchasing and all that -- there's no more excuse for saying, “I can't do this now.”  Well, yes, you can. It’s ready now.  You just need to get it ready.  So, people had to become more efficient.



And I think you have some companies that are coming up with programmable hardware, you've got people doing private Clouds that have programmable networking and that type of thing is going to expose even more inefficiencies that either automation or those kinds of practices will assault.



Tom Parish:  So what would be an example of that and it sounds like you were talking about something below -- along further down the stack?



John Vincent:  So, well yes, right now one of the benefits of an Amazon EC2 or, you know, installing open stack or Cloudstack to your private infrastructure is that you abstract out the configuration of the infrastructure.  There's an API on top of it.



Tom Parish:  Exactly.



JohnVincent:  But obviously people still have real hardware and in cases they have real network switches and they have real routers and firewalls.



Tom Parish:  You bet.



John Vincent:  So what we’re talking about now is there are vendors coming out with modified versions of the traditional networking hardware that is programmable over an API.  So now you can write code to actually make those changes as opposed to a manual, go in and configure and sync, kind of situation.



The same kind of evolution you see with the servers and infrastructure as code.  I mean, we’re really moving – IT is a whole – I feel like it’s going to go from the same kind of shift that was agrarian to industrial.



Tom Parish:  OK, very interesting metaphor, never thought about it quite like that. So that sort of further accelerates the opportunity for being agile and quicker in the way that you would…



John Vincent:  Right. It’s the efficiency thing.




The next excerpt from John's interview is coming up.

Share: |



This is another excerpt from John’s podcast.


It reminds me a little of Jez Humble's segment where he was concerned that he sounded a bit too "kumbaya" but listen to what these practitioners are saying, more than how they are saying it. There's an important message.





Tom Parish:  What do you figure are the top three takeaways about DevOps and why?


John Vincent:  You know I think the biggest takeaway is that it will -- I hate it being worded this way -- but it's going to optimize things.  I know that sounds a lot like a marketing pitch, which is not intended …



Tom Parish:  That's the end point.



John Vincent:  That's the end point --  you know the biggest take away for me is that you don't have to keep doing things the way you're doing them.  You don't have to keep dealing with passing the buck and passing blame and, you know, CYA.  It's more of a “your company can operate better”.  I mean, in the end we all have a responsibility to help the company we work for or the business achieves its goal.  And for me that's one of the key takeaways.



Another one is that things are going to change without you anyway.  So, you kind of have to get on board with what's happening. I argue that the current crop of what is operations or developers people that are coming up, won't be tolerant of the current way things are done.  They want to succeed and they want to get things done and that they're not going to tolerate these walls because it's not an environment they're used to working in.



And I think that was the main three ones. That is, DevOps is having the same impact on IT as open source did.  Some people might consider that a bit of an extreme but really …



Tom Parish:  I like that.  Talk to me a little bit more.  What do you mean by that?



John Vincent:  Well you know when Linux started making headway it used to be the situation where “OK, we're going to run this, this one Linux server over here and kind of use it to do one thing.”



Tom Parish:  Yes, quarantine over there and if it’s a good boy, that's where we'll leave it.



John Vincent:  Yes, nobody knew how it worked. Or the guys who were, again, practitioners knew this was what they needed.  And it short-circuited so much of the process to do it that way.  And I've been at companies where what you would call DevOps was done as a like a “dark Ops” kind of operation.  One night somebody said, “look, this is painful,” and the next day came in and said “hey look, we just installed this.  And we're going to start using that and this is how we're going to use it.”



Tom Parish:  Yeah, I’ve been there.



John Vincent:  Sometimes those kinds of changes have to be done, what is it -- ask forgiveness later rather than ask permission now?




The next excerpt on John discussing how the issue is with the interface between business and IT and the post agragrian IT future, is coming up.

Share: |

Jez Humble banner.jpg


The last segment of the interview with Jez Humble we discuss: what does he think is coming?




Me: what do you see or anticipate for DevOps, say, 3 to 5 years out?


Jez: Of course the term DevOps - which never had a snappy definition to begin with - is already undergoing semantic diffusion and everyone is jumping on it and adapting it to their purposes. That's to be expected.


I think the best that we can hope for is that as many people as possible get some leverage out of the movement to try to make things better. But if DevOps and Continuous Delivery are accepted as part of the canon of How Things Should Be, that would be a good outcome.


Realistically I don't expect that most enterprises will fix the problems DevOps addresses in only 3-5 years. Inevitably there will be a backlash - like the one we've seen recently with"Agile" - some people think that's already happening. But that's the nature of movements. What really matters is what happens on the ground.


Share: |



Anyone who’s been around the DevOps community for awhile knows of “lusis”, otherwise known as John E. Vincent. He gets quite an introduction on this latest DevOps Leadership Series podcast. John's well-known throughout the DevOps community. You get a vivid reminder why from this active practitioner during this podcast. [A quick recap on the format: John and I sort of work out a theme or issue or two to discuss, a rough set of the questions and then we let loose with an interview by the BMC podcast pro, Tom Parish.]


John runs a great twitter feed (@lusis) you need to follow. It's the feed where he’s known to be quiet, shy guy who is slowly learning how to express himself.




This is the first in a series of excerpts. I’ll provide the highlights I like and the formal podcast  is already out for those of you who prefer to consume your info via speakers and podcast streams.




Tom Parish: [Is DevOps] just another one those management things you know like a “OK, we’re going to get more with less people” sort of thing? I think there's more to that because you and I talked about this beforehand, and I really like your perspective on it.  So what exactly is DevOps from your perspective?


John Vincent:  [laughs] DevOps -- I guess you're partially right in that it's doing more with less.  But, really, DevOps is a cultural thing.


            What's amazing about this is that it wasn't born out of any sort of corporate plan or some sort of marketing type situation, it’s really a practitioner-driven thing.  And it's basically, the key thing is: it’s communication.  It's breaking down walls between people, departments, responsibilities and roles. It's not saying you are no longer primarily responsible for X, but it's more saying that while you were responsible for X, you aren't the only one responsible.


Tom Parish:  When you say walls, what do you mean exactly?  I think everyone kind of has a feel for that, at least from what I know. You know, Group A starts out doing what they're doing, they have a manager and a team, and they start doing things their way and then you know the hardware admin guys, they're doing their way. And so it's they just don't talk to one another?


John Vincent:  Mostly that is the problem.  Most IT organizations have a very strong CYA and culture of blame.



Tom Parish: Little tribes.



John Vincent:  Exactly. And there's legitimate reasons for that in terms of who the ultimate person who is responsible for something is.  There's also just the standard human nature of, again, wanting to make sure you've got your bases covered and if no one can blame you, then you're clear.



So part of this is also born out of this particular silo problem because of, I think, a misguided approach to due diligence. In that if there are clear defined lines of where things are handed off, then supposedly the idea was that when something was handed off, the person would do their due diligence and make sure that everything was there before it would go on. But the problem is that disconnect, that even the smallest of gaps there, caused more problems than it helped.



That’s the end of my first excerpt from John’s podcast. The next excerpt is coming . . .

Share: |

Jez Humble banner.jpg


And our third segment from the interview with Jez.


You need to create organizations that are organized around outcomes, not function, and think about how to organize your teams - and your systems architecture - around the idea that everyone has something to contribute.


Chris: is DevOps an enterprise-ready notion?


Jez: Absolutely. One of my current jobs is making it absolutely clear that DevOps and Continuous Delivery apply at scale and in an enterprise context. I've written articles in the last year which describe how to make Continuous Delivery and DevOps work with ITIL change management and controls such as segregation of duties.


There's no reason you can't implement these principles and practices and still be compliant with regulation like Sarbanes-Oxley or PCI-DSS - indeed my co-author, Dave Farley, works for a financial exchange that is subject to both of these regulations. They found that their deployment pipeline actually made it much easier to do auditing - they got it for free, as a by-product.



They had an old-school ops person who was initially skeptical (having heard fairy stories from developers many times before), but once he saw it all working he was thrilled and helped to convince the external auditors that this stuff was actually superior to the "traditional" implementations of these kinds of controls, which often (ironically) increase rather than decrease risk.



Chris: [I gave a recent customer visit example and then went into asking about metrics and what to look for in those situations.] How do you as an executive, look out and navigate the entire business to this place of integrated, efficacious accomplishment? How do you adopt to modern customer requirements for responsiveness, speed, improved velocity, etc., before the reality of your current Waterfall situation puts the dinosaur in your company DNA out of business and takes your company along with it?



Jez: Well, I'm not going to be able to answer these questions in a few paragraphs. But they're very common - so common in fact that I am currently writing a book to address them.



But to sketch out a short answer, I think that senior management has to really pay attention to IT. You can't treat it as separate from the line of business. For many organizations - more than currently realize it - software is the business, and so you have to pay attention to the details.



You need to create organizations that are organized around outcomes, not function, and think about how to organize your teams - and your systems architecture - around the idea that everyone has something to contribute.



There's this idea in lean that people optimizing locally can actually create a sub-optimal system. Everyone in your organization has to have a macro-level vision of what the organization is trying to achieve, and think about how they can work together to achieve that.



We've known this forever - Peter Drucker was talking about how to manage knowledge workers in the 1950s - but recent work by people like Dan Pink and Donald Reinertsen, movements like Design Thinking and Beyond Budgeting and Lean Startup - they're all pointing in the same direction. Companies like Amazon and Apple have already adopted some these ideas, and my personal revelation of the last 18 months is that everyone wants a piece of this.



Unfortunately though you can't just wave your magic wand and make it happen. It sounds obvious, but I work for an IT consultancy, and we've had people hire us in the expectation that we can somehow painlessly transport them into this magic world of Continuous Delivery and DevOps. It's like the Underpants Gnomes in South Park.



But guess what. It's going to hurt, it's going to be painful, it's going to change the way you manage and lead your organization, the way you collaborate, the way you think about everything. Most people don't like change, even if they like the idea of change.



And when you grow in size, it's incredibly easy to become sclerotic and lumbering and descend into political deadlock and lose touch with any sense of passion about what you do. It takes constant hard work from everyone in your organization to avoid that fate.



So ultimately you have to find a way to drag the people in your organization out of the tar pit and harness the great ideas they have, and fight the people who are - with perhaps the best of intentions - preventing you from changing. That requires being hands-on, making trade-off decisions and managing risk without enough information, and generally (as my colleague Jim Highsmith puts it) riding paradox.



That sounds utterly indigestible. But the goodnews - as I said earlier - is that everybody can make a difference provided there is some slack. The fear of local optimization shouldn't paralyze people from trying things - that's one of the paradoxes we need to work with.




Next Jez discusses what he sees 3 to 5 years out.

Filter Blog

By date:
By tag:
It's amazing what I.T. was meant to be.