- by Joe Goldberg, Lead Technical Marketing Consultant, BMC Software Inc.


The truth is, it already is and has been for quite some time.


Regardless of how you categorize it, moving data securely and correctly is extremely common, hugely important and will become even more so. Specifically this post discusses automating and securing the movement of data across the enterprise and beyond.

When speaking to customers, I used to ask for a show of hands for anyone that did NOT include file transfer or data movement within their batch scheduling environment. For the last five years or more, I have never encountered a single raised hand and have now stopped asking the question.

Of course it’s not surprising. After all, however we describe ourselves, whether it is IT or IS (Information Technology or Information Systems) professionals or working in MIS (Management Information Systems), or EDP (electronic Data Processing), “data” and “information” is what we process, manage, store, secure or transfer. As collaboration among departments and companies increases and as society at large becomes more ”digital”, we realize we live in the Information Age and we are Knowledge Workers. The tangible manifestation of information or knowledge is data and it is now the input and the result of almost everything we do.


By the way, if you haven’t heard, data is increasing exponentially and we have already passed beyond the point where there is more data than we could possibly store, assuming we wanted to do that, even if we used every single storage device on this planet.


So let’s get back to the relatively simple challenge of moving data securely to support enterprise applications.


Because we have been doing “data processing” for a long time, lots of the data transfer processes we have today were built in simpler times when security, accuracy, governance and even speed were not nearly as critical as they are today. Sometimes, the transfers were initiated manually and verified manually. It is still not uncommon to find a process where one person calls or emails another person as the mechanism for starting a transfer. Upon completion, it is still common to have people manually check for success and to perform manual restart if there was a failure.


The transfers themselves are still frequently implemented using conventional file transfer protocol (ftp). This approach almost mandates weak management of security credentials and the need for either extensive scripting or manual confirmation.

Even newer, more reliable transfer facilities still fail to deliver out-of the box integration with workload automation tools thus necessitating additional scripting with all the negative characteristics of that approach.


The ideal solution for managing data movement within an automated workload automation environment should have the following characteristics:

  • Out of the box integration so that success or failure are authoritatively communicated to the workload automation solution
  • Secure credentials that are not exposed either during the administrative process (when jobs are built) or during execution
  • Support for popular, standards-based protocols
  • Support for popular encryption such as PGP
  • Simple to operate for users of the workload automation solution so that no scripting or programming is required
  • Support for all platforms used by the organization
  • Fully integrated into the workload automation solution so that functions such as SLA specifications, forecasting , auditing and reporting should operate seamlessly with the file transfer mechanism
  • Support for automated restart from point of transmission failure or from the failed file when multiple files are transferred by single job

A solution that does anything less, is likely to sooner or later result in processing incomplete or duplicate data or in embarrassing or costly data breaches.

Today’s IT organizations are constantly seeking to reduce costs while seemingly paradoxically increase the quality of services they deliver to the business. With the importance and volume of data only increasing, I believe it is impossible to achieve the goals of IT when hamstrung with a file transfer solution that does not deliver these capabilities.


The postings in this blog are my own and do not necessarily reflect the opinion or position of BMC Software Inc.