http://vmlmat.wiki.sourceforge.net

Background, in case this is the first post you have ever read about VMLMAT. VMLMAT is not a product of BMC's R&D organization. As we are organized internally, R&D Support is actually in the IT department, and VMLMAT was designed and built by Ron Michael, the man charged internally to support all of R&D Linux images on the mainframe. It expressly is written for the needs of our internal environment, and some of those parameters are:

• All Mainframe Linux images run under z/VM. While we have      LPAR's, we do not use them for Linux.

• We made an arbitrary design decision to use Active Directory      as our repository for users and passwords. VMLMAT stores the      metadata about what each user can do, and which Linux images they      own, but not the actual userids / passwords themselves

• Our R&D community was needing to treat MainFrame Linux as      "Just Another Linux": the same as the Linux on the X86      server, the one in VMware, or the one on their desktop or laptop. As      developers, they know Linux, but they may not know much or      even anything about z/VM or the mainframe.

• Related to that: We had no idea what platform the R&D      people would be using when they were managing their Linux images.      Could be AIX, Solaris, Linux, HP-UX, OS.X, Tru64, VMS, MS Windows      from 2000 thru 2008...

• With terrific z/VM system programmers in the support      organization, we were not looking to do anything that either got in      their way, or add any system z/VM management features.

• We have no MF attached SAN devices, therefore no MF Linux      running on FCP devices.

Because of Whurley and the new direction he represented for BMC, and discussions with same, we knew from a fairly early point that VMLMAT might become one of BMC's first Open Source projects. However, that did not translate into making VMLMAT the be-all, end-all of Linux on the mainframe management tools. We needed VMLMAT to do certain things internally, and that was all we could justify from a time / cost point of view of Ron's efforts. All of VMLMAT's features are funded by IT, therefore they had to meet IT's needs, and in this case, that meant R&D's needs. The good news is that Ron knew this might go Open Source, and so he was very judicious about the codes documentation both inline and via the Wiki. This serves us well internally for anyone that might come along behind Ron, and it serves the Open Source community as well, since they were going to get clean, well documented code from the start.

Being well documented was key: We knew that there were missing features, and we knew we might not be the ones adding them. We made the leap of faith that any submissions back to the project from the community at large that were based off the codebase stood a better chance of being equally well documented and written. We will not accept anything into the main code tree that is not so (or that we can easily make so).

Taking my list above in order then, here are the implications about what VMLMAT could be made to do in future:

• VMLMAT can be thought of, at least in part, as a cold      provisioning tool. It takes Virtual Machines and makes them into any      version of Linux in the archive. It can also be used to store      particular configurations so that they can be promoted from Alpha to      Beta to pre-GA / release candidate, then to to production. But if      Production is in an LPAR... not so much. Not yet anyway. We think      adding LPAR support is very do-able though.

  • Related to that: VMLMAT is a stand alone tool right now, but           what if you already have a provisioning tool in house? Something           like BladeLogic? Could VMLMAT be made to interoperate? Again, we           think the answer is yes.

• David      Boyes, over on the VM      Linux mailing list made a great suggestion, that VMLMAT use PAM      rather than our Samba code, so that authentication could be made      pluggable, and therefore it would not matter what the target shop      has for userid / password authentication. LDAP, NIS, NIS+, AD...      whatever.

• In a production environment, not everyone and their cat      should have full access to the production versions of Linux. VMLMAT      already has a pretty good security model for this, since it allows      one to many, many to one, and many to many groupings of users to      Linux VM's. But I can see how some shops might want to tie this is      to something like RACF or VM:Secure. For our part, we would just      like to store all this is something like PostGres or MySQL in the      future, rather than in locked down flat files.

• Our choice about the user interface was Web 2.0ish (even      though it is pure HTML right now). We assumed everyone would have a      web browser, even if we did not know what the web browser might be.      Other shops may prefer a Curses or an X interface. We have no      internal demand for that, and I can not see us adding those other      interfaces right now. What I can see us doing is getting far more      sophisticated over time with the web interface. It is currently      designed to be about as fast and simple as it can be: Real Web 2.0      technology Open Standards (such as AJAX) could make it much more      active in nature. Whatever is done here, Open Standards will be at      the core though. If we learned nothing from the web browser wars of      the 1990's it is that using proprietary browser extensions is a      "very bad idea" (tm).

• There is an API in z/VM called System Management API, or      SMAPI, and not to be confused with Microsoft's Simple Messaging API,      also called SMAPI. Other than name, and being an API, not even      similar. Via VM's SMAPI, VMLMAT could be extended to "talk"      to the host OS, and have it do things like create and destroy VM's,      add mini-disks, change memory parameters, add Diag permissions...      whatever is needed. While VMLMAT has steered pretty clear of most of      the features of BMC's original cloning tool, such features would put      it squarely back in that realm. We don't need such things, so they      would more than likely not be added unless something extraordinary      happened, or they were added by the community at large.

  • Such features could also be added by having VMLMAT make           calls to whatever the shops systems management tool is. This is the           way the the BMC Cloning tool went about it. VMLMAT shares no code           with that tool though, so adding such a feature set would more than           likely be a community endeavor.

• A key design point of VMLMAT was device and file system      independence. The way that the systems are archived and restored      does not care about what the underlying device is. It does not care      if it is EXT2, EXT3, or (probably, but not tested) XFS, JFS, or      Reiser. We take whatever device VM gives us, and we are glad to have      it. Thus should it ever be. While we have never tested it, adding      SAN devices should not be that hard: Mostly a matter of dealing with      the device names themselves.

  • Related to that: We built the archive feature to match a           resource that we had: that being our spiffy Linux NAS servers.           Other shops might prefer to archive to FCP devices, CIFS, or even           to MF DASD, leveraging at least the fact that the archives are           gzipped and still saving space. All of this should be easy to add.

  • Further related to that: We internally use only VM presented           Mini-disks as "raw" devices for Linux. No LVM or EVMS           layer. Since VMLMAT is device independent, adding support for a           Logical Volume Manager(s) should not be that difficult. In fact,           installing Redhat without LVM has gotten more and more difficult,           so this is something we may have to add for internal reasons.

Those are just some of the obvious things we have talked about (and we have talked about other, more esoteric things still), and to some degree they are based purely on our projections of what others needs might be. I freely admit that I have no idea all the possible things others might need or want from a tool such as VMLMAT. We tend to have a very BSM mindset internally for some reason, so that may color what we think about when we consider what we might do to VMLMAT.

In its current form, it has more than paid us back for the time Ron spent creating it. How it might be extended to be of similar help to others is an "Open" question.

http://vmlmat.wiki.sourceforge.net

I am going to go out on a limb and guess that if anyone in your shop is using Linux, they started using it on a PC of some kind. Probably an X86 box like a laptop or desktop. If Linux was studied in school for something like a compiler of OS architecture course, it was probably taught on small, desktop class gear. Point being, they are used to having control of their system.

In a small development group there is probably a re-purposed PC or small rack mounted server or a VMware Virtual Machine, and everyone accesses the Linux server via X or SSH from the Linux laptop or desktop in their office. Someone wants to reboot, they send out an email, IM, twitter, or just shout up and down the hall that the server is going down.

The traditional MF shop is the opposite of this. It only makes sense: the MF is too expensive to not have it be centrally managed. I often think that ITIL was developed by people that had MF background because a large number of the disciplines it discussed are things we had to do in the MF world long before they had formal names. Change Control for example: You make a change to the MF, you have the potential to blow up a whole bunch of people or a whole lot of batch work, all at one time. This tends to make people grumpy.

As one who has lived in both worlds, I have seen the culture shock of someone coming from the small Linux platform to the MF. Something like: "What do you mean I have to call the data center to get my VM recycled. Are you nuts? Just get me a 1U server and let me control it. I don't need these people in my way!" It is just not intuitive that they should not have control of their own Linux, unless it is the production web / app server or something. There is a sense of ownership: That is my Linux server.

This is compounded of course by the fact that with the general Hossness (TM) of the MF and the best OS on the planet, z/VM, there can be hundreds of Linux VM's. Thousands even. Now we get a real clash of cultures (the folks who want to own their compute resources vs. the central data center), and a real stress on time and resources (all the behind the scenes work it takes to keep the shop running and up to date vs. being responsive to the end user). If one runs this Linux shop the way that the MF has done it in the past, then a system programmer makes all the changes to the guest operating systems, and a system operator brings up and shuts down the same Guest OS's and the "owner" just uses it for whatever they need to do. And oddly, no one is happy.

• The Sysprog or Sysprog team is/are stress cadets because      someone forgot to add more hours to the day to do all the work.

• The Operators are really really tired of all the reboot phone      calls and emails and equally unhappy when they get a complaint that      the reboot did not happen in five minutes, even though they were      running backups over on MVS and mounting a tape, retrieving an      archive from the vault, or something similar at the time.

• The end user is not happy because this is not nearly as good      as if they just had their own computer. They could install what they      wanted when they wanted and reboot all day long and no one would say      anything about it.

Part of the problem here is that, for the most part and to the end user, Linux is Linux regardless of the platform it is running on. If it does not look different, why must their behavior towards it be different.

Why Indeed?

VMLMAT solves this culture clash by returning control of the individual Linux VM's to the end user that "owns" them. It is not the exact same thing as having a physical big red button, or the server tucked under the desk, but via a standards compliant, any browser should work, web page they can shutdown, restart, or rebuild their Linux to any version of Linux in the repository. This is something like Vmware's VI Client, except that it adds provisioning to the mix, and where Vmware has snapshots, VMLMAT has archives. Not the same thing, though they can function in the same role more or less. Now the Operators are focused on the things that need their physical presence, the sysprogs are working on more technical things, such as putting the latest version of Linux into the archive so that all can benefit from it.

Install once, run many (tm).

And of course, the reboots are happening right when they are requested. Everyone is much happier

VMLMAT takes the paradigm of real machine ownership a bit further: In the real world, one person may have several Linux machines, or a group or people working together might have just one... or a group of people have a group of machines. In database-speak, it is a one to many, many to one, or many to many relationship. VMLMAT implements this so that there can be individual machine ownership, or group machine ownership, except that the 'machine(s)' is/are really Linux Virtual Machine(s).

The time saving are real too: Ron noted in his "Beyond Linux Cloning..." post that VMLMAT returned 30 hours a week to his time. Time that he used to spend working on tickets for the users to make various changes to their Linux VM's to match their current requirements he got back, plus now the end user was not waiting on him to have time to get to their tickets either. Everyone was more productive. A huge win-win.

Our savings may be a bit overstated: We are an R&D environment, and we make changes all the time. We have test versions and test versions and test versions. We need something at base level, another with a particular set of patches installed, another with a different set of patches, another totally bleeding edge current, plus Alphas and Betas... Still, I think that the savings are there for everyone.

I assume any reader of a technical blog has come across the "Law of Unintended Consequences". A decision, or path that leads to unexpected places. It usually has negative connotations, as in

"We (the generic we) decided not to spend any money on a new firewall, and with our savings we fought off a series of virus and malware incursions that cost us 100 times what the new firewall would have"

or

"We (still the generic we) designed the car to be light weight so it would get good fuel economy, and with the money we saved by not using higher strength materials we paid for all the lawsuits we were served with when it turned out the cars fold up like lawn chairs when hit by a Vespa.".

There are lessor and greater examples of the law than this, and I totally made those two up (although I am willing to bet my first example has happened someplace to someone). The law is well known though: It has a Wikipedia entry even, and as noted there, when there is an upside to the law, it is called a serendipity or a windfall. VMLMAT provided us with a windfall recently.

Rote Work

Computers are supposed to help us out with repetitive tasks. That is what they are good at, and we... at least I.. am not.

We had a situation not that long ago where we were re-doing some internal TCP/IP addresses, and all the Linux guest on the mainframe needed to have new IP addresses and new subnets. We were going to have to go manually through over 100 VM guests and change their IP settings.

The chances for error were high, and boredom even higher. Ron and I talked about it, and a little light went on for us. VMLMAT could be made to do this in very short order. Not only was it Open Source, it was *our* Open Source. VMLMAT had been written to be easy to extend. Ron thought he could add the feature in a fraction of what it would take to re-IP all the VM's manually.

The secret sauce was in the module Ron had already written that preserves  the TCP/IP identity of each Virtual Machine, regardless of which actual version of Linux was running at the time. VMLMAT had allowed us to treat VM's Virtual machines more like they were group or personally owned assets. Once we had set up the VM in the directory, VMLMAT updated to know whose VM this was, then we were pretty much done. The end user, or group of people could go into the web page and make the virtual machine into any version of Linux in our archive. No matter what they did, the identity code would chain through all the right places and set up the new Linux VM as having the same IP address, Subnet, Default Route, and DNS settings as it did before it was changed.

Since VMLMAT could do that, why couldn't we slightly change it so that it could be handed a list of VM names, IP addresses, and Subnets, and have it chain through *all* the guests, and reset them to the new IP? This module already knows all the different places the information is kept for each distro, so all it had to do was detect the distro, change the IP to the *new* one, and then run on to the next VM. VMLMAT could do in minutes what would take us days to do by hand.

And so it came to pass that we could schedule one short outage, pull the trigger on the re-IP mod, and be done. No mistakes. No long outages. R&D went home Friday night and their VM's had one IP address, came back Monday and they had another, and never knew anything had changed.

VMLMAT already had the habit of returning over 80% of Ron's time versus pre-VMLMAT days.

This was a bonus.

UPDATE: Ron has now posted an in depth look at the re-IP module over in his blog:  Open for Mainframe. Have a look!

http://vmlmat.wiki.sourceforge.net/

If you have been following this blog, you know that my background is the mainframe. I started as a VM system programmer, and VM is still near and dear to my heart. When MF Linux became a reality, it was, to me, a natural thing that it would be a guest OS on VM the same way VM has hosted MF operating systems since it was invented over four decades ago (and the research that went into creating it stretches back even further).

Linux on the MF is just a logical progression of all the things that came before it on the mainframe... things like UTS and AIX/370. It is also natural for Linux: The ultimate in multi-platform OS's.

However...

Linux on the mainframe, like any other technology deployment of human kind, is not without considerations and issues. If you have worked with the X86 spiritual baby brother of VM, VMware, you might know what some of those issues for Linux on the mainframe are: Things like server sprawl, and the tendency for many end users to treat the resources as essentially infinite.

There is also a cultural issue. If you have programmers working on Linux, be it web apps or anything else, and they are less than four decades of age, they probably learned Linux on their Laptop / Desktop / small-server-in-the-cube-next-to-them (shades of "departmental computing") kind of thing. To them, for the most part, Linux is Linux, and it does not really matter where it is, and they most certainly do *not* want to learn anything about the MF in order to access their Linux there.

That means that they don't want to call the data center and have the operator autolog their Linux VM, they don't want to learn how to log into VM, IPL the Linux system, and then do a #CP SET RUN ON#CP DISC. No TN3270 stuff. No green screen. It is not the way of the GUI world, and very much not the way of the Web 2.0 world. CMS is only for us VM'ers it usually seems. I know my track record in teaching people how to use the mainframe who started out in computing on Linux, UNIX, or MS Windows is not good. It is not zero, but it is not the next generation of MF people either. The MF is just too different, and its attractions are often not obvious at first glance. Sure XEDIT is the best text editor *ever*, but it takes a while to come to appreciate it....

In general, Linux users are used to owning the resource, and being able to boot it whenever they want (during development anyway). There is a sort of security in owning the "Big Red Button" so that one is the master of their own computers destiny.

This all stands at odds with a great deal of the culture of the MF: The ultimate in data center glass houses. To solve the problems above, often all the Linux VM's are just autologged when the MF is IPL'ed , and they run whether they are needed or not. This would not be a problem (at least not nearly as much of one) if these were all CMS VM's, but Linux in a VM turns out to have a few design points that operate against the "Just IPL and let it run" way of working. The main one is memory management. Think of the way Linux allocates memory, which is more or less "use everything I can". What is not programs is in-memory cache. It is a great design, and it makes perfect sense if you are running natively on real hardware: Why let the extra memory go to waste? Why not use it to speed things up? The idea is not even original to Linux. UNIX before it had it as a central design precept. The idea was that disks are way slow, and so if there was spare memory laying about, use it to cache the I/O, and speed up the programs. Let the I/O happen when it could. This design point is still valid today, as disks are still extremely slow relative to RAM.

As a guest on the MF though, that means to the host OS...to VM, it looks like the guest memory is 100% busy all the time, and the way the VM likes to page out unused memory on guests so that only active memory is in core is violated. The solution is to keep virtual memory trimmed to just what the guest actually needs to do it's job... At least it was before VMLMAT.

We do R&D on MF Linux. in the pre-VMLMAT days, at any given time, using the above autologging method, we had over 100 Linux VM's running. VM was running out of real memory. VM would look around, find the least recently referenced pages, and page / swap them out to DASD. But Linux would then reference the page, and back in they would come. Paging / swapping is fine when what is being moved is rarely referenced, but it is called thrashing when it goes out and then comes right back in over and over. There are limits to what you can do to tune this, and we were at them.

Many of the VM's were idle in reality, but we in the support group had no way of knowing what was being actively used, what was up as reference, and what was up because it was autologged.

Sure, things like the Build systems need to be up all the time. That was just a handful of the total systems we are talking about here though. We needed a way to make it so that end users could, without knowing *anything* about VM, bring up and down their own Linux VM's.

Enter VMLMAT.

VMLMAT

Virtual Machine / Linux Management and Archiving Tool.

As system's programmers, we are of course not marketers. Our name for our tool is descriptive, not beautiful. The VM part tells you right away this has nothing to do with LPARS or MVS of DOS/VSE. This is a VM tool, and that only makes sense: where else but the best hypervisor on the planet can you manipulate the guests without them knowing what you are up to?

The Linux part lets one know we are not dealing with CMS here. The Management and Archiving part is descriptive of function. Thats just the way we roll. Since this is Open Source, maybe someone can contribute a snazzier name some day.

VMLMAT runs a standards compliant bit of HTML under an Apache web server: This is the way that the Linux users interact with the program. The system programmer has a few things to do on the install that do not relate to the Web interface, but the idea is that Linux users are used to doing stuff via the web browser. Moreover, we did not want to assume anything about where the MF Linux user was starting from. Could be AIX, Solaris, Linux, or MS Windows (perhaps with some sort of X loaded up on it). Given the platform diversity, standard HTML was a requirement even if we were not already a pretty standards driven group.

Now when we IPL VM, just the Build and Packaging Linux guests are autologged, and the end users can go to the web interface and IPL their Linux whenever they want. Or bring it down. It seems a simple thing, but that one feature saved us an MF upgrade. Only the Linuxii we need... that we are actually actively using... are up at any given time. 100+ Linux VM's up and running at the same time dropped to one fifth that number.

That is just the tip of the VMLMAT ice burg though. Here is another nifty feature: Disk space savings

VMLMAT and DASD

MF disks are very expensive things. One of the primary criticisms of Linux on the MF is that Linux was normally run on commodity priced hardware, and now by running it on MF gear all the price advantage was lost. Many saw of course that there were advantages: I/O pathing and monster transaction capabilities, best in the business HA, and so forth: Perfect for the production Linux environment. Not so perfect for all the possible iterations of a development environment.

Since we in R&D Support do care and feeding of all sorts of Linuxii internally on the MF, all the way back to the very first bootable kernels, before RedHat or SUSE were making MF versions of their distros, up to the latest and greatest stuff, we had a real diversity issue. All those VM's were sitting around.

Everyone needed a separate VM for each version they were  working on, plus a set of their minus one versions of code, plus the betas and alphas for announced code: This was not server sprawl from the "Its all free" point of view i mentioned earlier, but it was expensive server sprawl nonetheless.

VMLMAT takes a different route to this. We do not store all the versions on the MF DASD at all. Everything is archived to inexpensive NAS. If you have been following "Adventures", I have been writing about that inexpensive NAS for a while now. Now you know one of the things it holds: Our MF Linux images.

VMLMAT can package up via TAR any VM, and store it off to the NAS. Even better, it can restore that archive to ANY OTHER Virtual Machine. VMLMAT unrolls the archive, and then personalizes the archive to match the VM it is being restored to.

We then leverage that in many ways:

1. When a new release comes out, we install it for the first time and then archive it. Now everyone can use that new version without anyone in tech support ever getting involved.

2. If applications need to be added, the base archive can be installed, updated with, say for instance, Oracle. Now it can be archived again, and anyone who needs that release of Linux with that release of Oracle can leverage the work.

3. Business Continuance / Disaster Recovery: Now we can take the NAS archives and replicate them wherever and however we like, and get back all this work, and it is simple and easy.

Only the currently being used copy of Linux is up on the MF DASD. The rest is spinning far less expensively out on the NAS. Multiply by the number of Linux VM's and the number of Linux versions and the number of application setups and that is a bunch of DASD being saved.

With the Web interface to VMLMAT, the end user is now an empowered individual. They can bring up their VM, shut it down, change it, install stuff, archive their changes, share their changes with other teams, and leverage other teams work. Say they are running RH AS 5 with Orcale, and they want SUSE with MySQL for their next test.  They can archive the work, retrieve SUSE with MySQL from archive, and start testing. The whole backup and restore take less then 30 minutes in our shop. The new SUSE VM has the same name and same IP as it did before: VMLMAT took care of editing all the stuff in /etc so that the VM name never changes. Just the Linux version.

At no point did a system programmer get involved in that transaction: We have seen a workload that was frankly swamping Ron drop to the place where rather than him needing to work crazy hours to keep up, he puts in a few hours a week on MF Linux maintenance (like creating new archives of new releases of Linux when they come out) and then moves on to other things. Not only was the end user enabled but we got back most of an employee for other tasks.

There is even more, and I'll write about that in future posts, but for now I want to tell you how anyone can get VMLMAT if they want it. Did I mention that BMC is Open Sourcing it yet?

Sourceforge and BSD

VMLMAT is available to anyone who is interested at Sourceforge:

http://vmlmat.wiki.sourceforge.net/

VMLMAT is licensed under BSD, one of the most open and permissive of open source licenses, and the one that we at BMC have chosen to use when we release Open Source projects. Our little site was just set up yesterday, and we are still knocking around learning how to use Sourceforge, so bear with us. We are loading up the documentation on the Wiki, and the tarball for the current 1.1.0 version is there. We'll be checking it all into SVN soon, but till then this is the way to get it.

VMLMAT is created entirely out of Open Source projects like Apache, PHP, and Samba, and the VM portions are the very VM standard REXX. No C or assembler was harmed in the creation of the tool. The  HTML is scanned and certified as being 100% open standards compliant. As we have it written, it leverages NFS to archive Linux images to NAS.

Ron did a slideshow style presentation to walk through some of the features of VMLMAT, and it is loaded to Sourceforge as well.

I'll be your host on the project, along with the BMC internal author of VMLMAT, Ron Michael.

Ron also has a blog at TalkBMC called "Open for Mainframe", and he has a bunch of posts in various stages of readiness for posting there about VMLMAT. We'll also start cross-posting with the blogs over at Sourceforge soon, so stay tuned for that.

In the meantime, I am jazzed. VMLMAT is a simple concept, and an amazing tool, and it has been saving us all sorts of time and money. I am thrilled to be able to share it with anyone else who is interested out there in two of my great geek-loves: VM and Linux. Further, we knew going in that VMLMAT was created to meet our real world requirements, but it also matches our R&D Support environment. Knowing that it might be Open Sourced, Ron created it so that it can be easily enhanced to add new features. For example, we chose to use Active Directory for a end user authentication (via Samba) rather than maintaining a separate user / password file or perhaps having it in LDAP. The code is modular so that anyone can come in an write their own module to match their internal needs, and hopefully they will contribute that back so the VMLMAT can grow to meet a broader set of real world, Linux on the mainframe management challenges.

As Rachel Maddow says: "One More Thing:" Please do not confuse VMLMAT with BMC's VM Cloning Tool from a number of years ago. It shares no design and no code with that tool.