Share: |


This is my 3rd excerpt from the interview with Gene Kim.



Gene Kim: You know, incidentally, just thinking about your question ‘how important is DevOps to business?’



One of my colleagues and I actually did this massive calculation how much value is there to recapture.  So we did this massive calculation, how much would be saved from all this DevOps -- like long lead times, poor hand-offs, unplanned work and rework. You know, we said, if you could just halve the waste. So the calculation of the value, the value of that waste, was about $980 billion



So we said if we could probably just get half the value back. If we deploy those dollars in a way that would get you know five times the value.  Anywhere. Where you can just get five times the value.  We're talking like $3 trillion dollars of value that is there for us at the table to grab, per year. 


So, you know, that's for some perspective: that [the potential global DevOps benefits] is more than the entire economic output of Germany.


So that is an incredible amount of value to the business.



Tom:  Wow.



Gene Kim:  Oh and by the way, I heard a fascinating stat the other day.



Tom:  Yes.



Gene Kim: Fifty percent of all capital dollars being spent in US corporations is on IT. So it looks like an improvement in IT has a global effect on the organization.




Next we get some longer excerpts from the transcript that start providing some real meat. Gene provides clear, concise acitonable advice on improving your DevOps kung-fu.  First up, the question most of us think we know the answer to, Gene puts some studied science on: what do high performance IT Ops organziations look like?





All excerpts from Gene's interview and the link to the podcast itself are here:


Gene Kim's DevOps Leadership podcast is available here.


Share: |



This is the second excerpt from the podcast with Gene. Gene is an articulate, passionate speaker and jedi master on things DevOps and these excerpts capture a few tidbits from his great podcast, which will be published soon. 



Tom: How is DevOps important to business from your perspective?

Gene Kim:  Yes, I think it's very important.  One of your previous guests, who I’m a big fan of -- Clyde Logue -- said it really, really well.  And he said – and I love this, I wrote this down -- he said “Agile helps Dev regain the trust of the business.” And I think what was left unsaid was that in many ways Agile left IT operations behind. 



So just for a variety of reasons, you know, IT operations is often the bottleneck for most organizations because all work will funnel from the ten, fifteen, developer groups into one IT operations group.  And so one of the side-effects of Agile helping development go faster is now we have more and more work piling up in IT operations.



Developers merely committing code into version control doesn’t achieve the goal.  You know, only when it's in production and it's delivering intended value to the customer, only then do you  have a shot at achieving the goal.



DevOps is as important as Agile was to development but DevOps is going to be important for all of IT.






More coming on Wednesday this week. In the next excerpt, Gene will talk about a study he did that showed the aggregate potential business value of DevOps of $3T. Yes, that's trillion...

Share: |


Since 1999, Gene Kim has been studying and benchmarking high performing IT operations and information security organizations. 


For 13 years, he was the CTO and founder of Tripwire, a security software company that commercialized the open source software he wrote in 1992.  While at Tripwire, he also co-authored the “Visible Ops Handbook,” which has become widely recognized as the “good to great book for IT,” which has sold over 200K copies to date.



In 2007, ComputerWorld added Gene to the “40 Innovative IT People Under The Age Of 40” list.  He has helped define IT compliance standards, including SOX-404 and the Payment Card Industry Data Security Standard (PCI DSS).

Gene is currently working on his third and fourth books, “When IT Fails: A Novel” that describes how to fix the dysfunctional relationship between IT and the business, and “The DevOps Cookbook,” a prescriptive guide on how to start and finish DevOps style transformations.


The podcast with Gene is marvelous. Going through it and then the transcription to select out excerpts I found appealing or insightful or funny, was a pleasure. Some of these excerpts are going to be solid for your DevOps kung fu and the up-coming DevOps Cookbook (more to come) is obviously going to be your step from padawan learner to full master.


Tom: Now I've asked some of the other folks that we've been interviewing on DevOps about their definition about what DevOps actually is. And I'm curious – is it important that DevOps largely be left without a crisp definition? 



Gene Kim:  You know, it's an interesting question. In my mind, the fact that there is no crisp definition is sort of good and bad news.



Tom:  Yes. 



Gene Kim:  On the one hand, you could say that DevOps has articulated an incredible philosophy, a set of values and cultural norms that have appealed to a broad range of people. It's helped gain the hearts and minds of some of the best practitioners and thinkers in development and operations and even information security. 



So that’s a phenomenal thing.  In fact, one of my coauthors of the DevOps cookbook, his name is John Willis.  He wrote this amazing blog post called “The Convergence of DevOps.” And he starts it with four years of history of where DevOps came from and to how it's really a convergence of at three large movements cometogether. 



And one of them is like the 'infrastructure as code' people. Yes.  So this - this includes all the Puppets and Chefs of the world who automating the infrastructure andprovisioning processes.  Which has brought people like Patrick Debois, one of your previous guests.  Andrew Shafer and the like.



Another thread was the Velocity thread where, you know, John Allspaw and Paul Hammond gave a talk in 2009 about how they were doing 10 deploys a day at Flickr, which sort of shocked the world.  That created a whole movement inside of the VELOCITY community.



Then there's this other thread that's coming from Jez Humble from the seminal work that he's been doing in continuous integrations and release. So, you know, I think there are even more forces than that converging including the fact that so many of the DevOps patterns that you see I believe are actually the inevitable practices that you'd get as you would apply the Lean principles to the IT values stream.  So to make my point here is that, you know, it's fantastic that DevOps has cast a large enough umbrella, you know, that’s wide enough that it that can bring all these people from different backgrounds together. 



And I think the bad news is that DevOps is not yet prescriptive enough where it’s not a collection of practices or processes.  You know whether descriptive or prescriptive.  And so organizations that actually want to adopt DevOps-like practices don't have much to go on.  So the intent of the DevOpsCook book project is to really catalogue what the high-performing DevOps organizations are doing and provide some prescriptive guidance so that other organizations can replicate the results that we're seeing already.



So – I think the good news is that it describes fantastically a set of philosophies and values.  The downside to that is, ideally, in orderfor DevOps to become mainstream and to be adopted more widely, we do need something more prescriptive.  That's where the DevOps Cookbook is coming in. 





There is much more goodness to come. Next up is the excerpt from Gene's interview where he talks about how "DevOps is for ALL of IT"



Share: |

Kris Buytaert banner.jpg


Another excerpt from the BMC podcast with Kris. Kris makes some great points in this, IMO.




Tom: I'm curious, why is packaging such a religious issue?  Do you think there'll ever be a packaging standard?



Kris:  No, there never will be a standard. Just as there never will be a standard in distribution or a standard operating system.  Every operating system, every distribution comes with its set of packaging guidelines.  So there's – there's two things to packaging.


          One is some people package, some people don't package.  And there is the discussion of should you package or not? And the other thing is, which packaging tool you should use.



So to me, the thing is, I don't care which packaging tool you use.  But I do care about being able to track where files are coming from.



I want to be able to figure out if a file has been modified on my platform.  I want to be able to see who made themodifications and why.  And I want to be able to deploy on each and every platform I have the same version of software over and over again – whether I have to do it over a satellite link or whether I have to ship the image or the software on a CD to a customer.



Tom:   Right.



Kris:  And packaging helps me doing this.  It gives me all of those things for free. 



So it's not about do you package or do you not package?  To me, it's a culture issue.  It's 'am I going to have something in there'?  Am I going to just fiddle around with the platform?  Or am I really going to build something which I can trace back. Which I can know where stuff is coming from.  Where I can identify the source of something.  And we're actually building an enterprise environment. 



To me, it's maturity.  It's about how do you deal with software distribution? Am I just going to throw stuff in a pile and hope it works? 



Tom:   Yes.  So DevOps has been focused mostly on obvious ops changes.  What changes need to occur on the development side?  In the dev side?



Kris:  I don't think it should be focused on ops changes.  I think the ops changes have been – have been the most visible, because over the last couple of years there have been so many tools put in the Ops area that have been really growing so much. Whereas on the developer's side, there's not really new tools. 



There's not really – well there's old tools that are revising.  But it's more of a mentality shift.  For operations people, it's much more visible because they start doing infrastructure as code.  They start building system configuration tool change for the developers.  And for the developers it's more of a mindset of realizing that software is not worth managing until it's actually in production generating revenue.  It's realizing that maybe they also should bethe guys on call because they wrote the code, they should fix it.  And that's a change to them. 



It's more of a practical, how do they deal with the traditional mind of software developer.  That's something that's going away.  If you build software, you're also responsible to make it work.  And some people say, “Look, a software project ends when your last living computer dies.”  And that's a total different point of view than five or six years ago when a software project ended when, well, you went home at 5:00 on Friday afternoon.  You were done with it.


Tom:   Well, so, How far back into the productlife cycle should DevOps be involved?


Kris:  So imagine you're thinking about a fancy new feature. You're thinking about building a start up tomorrow.  In that case, we're already too late. 



The idea is that as soon as possible – as soon as there is the idea of a project, the idea of a feature, the idea of some data software that needs to be deployed eventually -- infrastructure people, security people, relevant peer parties, should be in a discussion.  



They should be thinking about which requirements are going in.  They should be thinking about the non-essential requirements as well as essential requirements as well as all the dependencies in between. 



A decade ago, when you were thinking about we have to launch a new mobile provider, you needed to start thinking about buying new hardware, buying stuff.  And it was going to take you up to six months before everything was in place. Just procurement-wise. 



Today you can build infrastructure or deploy virtual machines in five to ten minutes, and then you have an application which is running.  But it's not an application that's also scalable, that's also something that can be exported.  Can be monitored.  That can be running and serving a lot of computers. 



So getting people involved, into your ideas.  Getting people into “we want to have this thing.  And we want to have it stable, it needs to scale.”  That's where you actually want to have people involved that have experience on the topic and talk to them and get them involved as soon as possible.  So you don't have to call them 10 minutes before the weekend saying, “Look.  We want to launch this new project.  And we're going to do it on Saturday. Can you deploy this tonight?” Because people don't want to do it on Friday evening.



Planning is still the most important thing.  You really want to have people involved in projects as soon as possible.  So they – both systems people, developers, security people, DBAs -- they should be involved as soon as possible in the life cycle of the project.






That's my last transcript excerpt -- be sure to listen tothe podcast.


Other related on-line materials:

When the Kris Buytaert podcast is published, it will be here.


Excerpt A: DevOps Needs Sushi & Mainstream DevOps

Excerpt B: Infrastructure as Code, Monitoring, and Resonance

Excerpt C: Packaging Religious Wars, Maturity & the End of Tradition






Share: |


This is my last excerpt from John's podcast. Some good stuff!




Tom Parish: What about security – where does that fit in with DevOps – my mind was just going crazy when I heard you talking about APIs that talk down into hardware levels and more. You got to know, that’s got to give some people some sweaty palms, you know.



John Vincent:  Well, it does and here is what I think people are starting to realize: I know some security professionals who I would say get it, and that is by automating and codifying how things should be in some sort of programmable way, you take the risk of some misconfiguration opening you up to security vulnerability, or making an attack vector, out of the picture.



Tom Parish:  Now, how would that be?



John Vincent:  So, here’s the thing: if you know that every server needs to have XYZ and it needs to be configured this way to be secure, if you're doing it manually, you’ve introduced an issue where someone might misconfigure and now you have security risk.



So my argument is that things like configuration management actually makes things more secure and by proxy you remove the need for people to actually be on a system.  Which is, you know, closing loopholes that exist and that needed to exist because people needed to do certain things.  And we as approach this concept of ephemerality, if that’s even a word, of resources when you have a compromised server you can easily replace it and go do your investigation.



It’s not a "stop the world" kind of scenario.



Tom Parish:  I see.



John Vincent:  You know, obviously there are some best practices you want to follow there…



Tom Parish:  Let's move into that.  I mean let's talk about that: DevOps best practices.



John Vincent:  Well, they're rarely best or practiced!



            So I think the primary best practice of DevOps is not one that’s specifically DevOps and that’s the whole “Mr.Gorbachev, tear down this wall!”  but, it really is, the biggest best practice you can take away from DevOps, is that you need to have your teams communicate better. Regardless of what it is [they need to communicate].


So early involvement in any initiative, you know I talked about the gap when you hand it off something and you start to discover new things and it kills agility but early involvement can actually mitigate risks. If you have your operation team involved from the get-go of an initiative and communicating regularly with developers, it kind of ties into the security question. If your security guys are involved from the get-go, they instill and they can address needs before they become a systemic problem.  And the same goes for operations.  It goes for any other department.  The earlier you get involved, the quicker problems come to life before they're actually released.



That’s the best practice and it’s not even DevOps specific. I mean some people rightly argue that people have been doing what we have kind lumped together as DevOps, for years. And they have. It’s been primarily contained to the situation where you don’t have as many people in the company or you have such scale that you have to do it.



I think there is this middle ground now.



Tom Parish:  How much of the best practices for DevOps is the definition of DevOps.



John Vincent:  Right and it really is.  It’s you know a definition of DevOps -- even just the name “DevOps” -- it implies this inter-team approach to dealing with IT.  And it brings to mind cooperation.  You know that’s what I think of and I think it’s no mistake that the first sort of -- I won't call it a tenet --  but you know there is a term called CAMS that people like to use that John Willis came up with.  It’s “culture automation metrics and sharing.” And that’s kind of what DevOps is but the first one is “culture.”


TomParish:  What would change in the daily role of a sysadmin or developer.  It’s kind of full circle from where we started but let's talk about it from that perspective.



John Vincent:  So my background is obviously system administration and operations and I think I’m not going to word it exactly right but we go from being, I guess, assembly line workers to managing the machines or managing the code that runs the applications.  That’s what you see with the system administration field lately or at least in terms of how I see it shifting is: that you step up a level.



And it’s not a one-man show or you’re doing this one task. You're codifying your knowledge and your skills in a way that things can do that for you.  So you become more of an overseer.  I don’t know if that’s quite the right word but as far as like say QA, it’s really the same thing.  People have already gone from manual testing to automated testing and multiple tiers of testing.



A QA professional will tell you that it's not just QA.  And you know so they went from – they made this shift from being guys who would manually test an application to using […] programmable tools that automate that.



And they actually had to do some programming around randomizing data for inserting and things like that.  So, both of those department sQA has sort of already gotten there [and] systems administration as well. They're moving more into being programmers in the generic sense, as opposed to being developers of specific product, of their environment. 



And for developers? It really depends on the industry. WebOps -- web development is completely different than desktop tools development. There are artificial barriers there that will never go away if you're talking about off the shelf kind of software versus a Web site. I think the developers are going to start coming back to [system administration].



It's really interesting developers and system administrators diverge from the same person, in my mind, historically. Where the original folks who managed the system, were the folks who wrote the code that ran on the system.



And then somehow we had this split.  And it was “OK we need more stuff to run on top of [that stack]” but this one person can't be expected to know it all.  I think that developers are going to start revisiting closer ties to how the code is running at a more micro-level with the system and that requires some operations knowledge.



I think they're getting better full stack awareness.  In the end I think that regardless of what the specific department is, everybody is shifting up the stack a bit.  You know there's still the underlying stuff and that becomes less and less of a burden and you still need to know how it works but you know people start shifting up – their concerns start taking into account more than just their domain. 


They're paying attention to things that they might not have cared about before.



Tom:  It seems like very much a natural evolution.



John Vincent:  And you can look at any industry and see this.  I mean it's not just IT.  We're just taking – we're almost re-inventing the wheel or rediscovering stuff that other industries have done before whether it's resilient engineering. The airline industry has this problem and they've investigated what causes human error and we're doing the same thing on the IT side.  And we're leveraging that existing knowledge base. It's changing, whether we look at biology or manufacturing.  You know we're taking our cues from things that are somewhat of a solved problem.

Share: |

Kris Buytaert banner.jpg

We continue our excerpts from the podcast with Kris Buytaert.


This is my second sampling from Kris on 'Infrastructure as code', a core DevOps tenet, and how DevOps is a truly practitioner-led movement.





Tom:   So how important is infrastructure as a code to DevOps?  And you might just take astep back and explain to people who aren't as into the details exactly what you mean by infrastructure as code.


Kris: So infrastructure as code is the idea today, you don't go into a datacenter anymore, pop in a CD, click to enter a couple of times and then move onto the next machine.  Systems people don't do stuff manually anymore.  We are basically writing codes to managing infrastructure.  We are writing in languages like CFengine. And we're alternating our whole infrastructure up to a point where you pop in your disc in a data server and it totally configures itself. 



That's what machines have been built for.  To automate itself and not making human servers.  And the evolution of automating this is something that the tools have been around – the tools have been around for ages. So because of virtualization, the scale of the number of machines we have to manage by changing the cloud even has forced that a bit. 



And then we needed to figure out different ways of installing machines, different ways of managing machines. Those were kind of accelerators to start infrastructure as code movement. 



And if you then put that into DevOps, if you think about testing.  If you think about scale.  If you think about orchestration.  Imagine that you have to go to a test engineer, “Look here's a bunch of servers we built.  Can you please try to destroy them?  See how you can break them.” And then, after you've broke them, you have to go back and say, “Well, that's very nice.  But now we have a problem, because we cannot give you your test environment again.  We cannot help you to do your test.”



So, for example, testing is one of the reasons why you really want to have infrastructure as code.  You may even want to give it to them the way that they can say, “Oh.  I just want to roll back to a previous date and see what happens.”



Now, if you were a small shop and you have to configure two or three machines.  That's fine.  You could go there manually.  But once you grow into a couple of hundred, a couple of thousand, it starts to become really difficult.  So people really need to automate there. 



On the other hand, if you are a small shop and if you really are doing specialized things, you also want to automate stuff, because just – just those little one-offs.  Those little, “Oh I made this really special setup, and it's real difficult to do.”



Tom:   Yes.



Kris:  That's also something you want to have automatic so when it breaks, you can actually do it again.  So this whole idea helps people think about how they should deploy stuff, how they should deal with infrastructure. But tools are getting there and up until five or six years ago, tools were a bit more difficult to use than they are today.



Tom:   Yes.  Yes. Well, let's move on to monitoring. How important is monitoring for DevOps?



Kris:  So John Willis and Damon Edwards, they coined a good summary of DevOps being CAMS.  C-A-M-S, which stands for culture , automation, measurement and sharing.



Well, I think the previous question pretty much tackled the automation part.  But the measurement part, that's where monitoring comes in.



You want to know what's going on in your infrastructure.  If you want to improve your environment.  If you want to improve the way you deal with software.  If you want to improve scale.  If you want to improve the meantime to recovery.  If you want to basically

improve the sleep of your system engineers, you want to measure what's going on. 



You want to monitor your infrastructure. So you want to build systems that are stable.  You want to make sure you know how stable they are.  Otherwise, you cannot see how much progress you made.  People want to know how many times you deploy a day.  So you need to monitor how many times you deploy a day.  They want to know what changes happened.  They want to know why changes happened.



Tom:   Yes.



Kris:  And you really need to build in those kind of memory things into your infrastructure.  And it just makes stuff a lot easier.  If you don't have any monitoring in your environment, then you're pretty much blind and just trying to grab somewhere in the dark.  Monitoring shines light on your infrastructure.  And it's really important to know what you're doing so you can actually improve on what you're doing.



Tom:   Right.  Right. And that even makes a lot of sense. Well, stepping back again, I know we kind of keep coming at this topic.  But it's new enough to kind of help people fully realize, you know, what this is going on here.  From your perspective, why does the term DevOps resonate?



Kris:  I think it's for a lot of people, they see the problem.  They see how difficult it is to keep an environment up and running.  They see development groups thinking about building software.  Building new features.  And then having the operations people with totally different markets having to keep systems stable and running. 



Now the whole DevOps discussion and whole DevOps movement have been furthered by people who have been working in the industry for – for a while, who have built good reputations in the communities.  Who have built really good track records.  And they started talking about the experiences they had on how to improve the way other people work. 



And it's probably pretty much because there were a lot of people involved in the movement in the early days that we're pretty visible in different communities.  That we're pretty visible in – in start ups that were doing this kind of stuff.  That most people want to be edgy.  They want to look at how operations is done.  And they look at the big example of "wow, this is how they run operations in such an environment."  And people want to be the same.



So it's – it's a combination of a bunch of good examples.  Good track records on “this is how we can do it.”  Combined with experience and well-recognized people explaining how they actually did this stuff.  And to me, that's I think one of the reasons why DevOps is probably much faster than other stuff.  It's not backed by commercial vendors.  It's actually people talking about how to do stuff with tools.  And all about pushing a product that's really solving solutions with tools that are widely available. 






Coming next with Kris: Packaging Religious Wars, Traceability And Maturity.



Other related on-line materials:

When the Kris Buytaert podcast is published, it will be here.


Excerpt A: DevOps Needs Sushi & Mainstream DevOps

Excerpt B: Infrastructure as Code, Monitoring, and Resonance

Excerpt C: Packaging Religious Wars, Maturity & the End of Tradition

Filter Blog

By date:
By tag:
It's amazing what I.T. was meant to be.