Problem solve Get help with specific problems with your technologies, process and projects.

Sys admins: Are your config files and scripts hacks or products?

Systems administrators often need to write scripts to fill the gaps in systems management tools. But often those scripts are just hacks -- "one off" pieces of code that don't scale or have a future. Instead, systems administrators can apply best practices to create viable, documented products.

Systems administration is a job with many facets, but I think it can most succinctly be generalized as one in which we "glue" products together to create solutions. We generally don't write applications or design computers in the traditional sense; rather, we create an environment of harmony and fill in the gaps between vendor-created technologies. That is to say, we make it all work.

There isn't an abundance of appreciation for all this glue work. Try telling your boss that you wrote a new application and see if he doesn't call it a "script" in a derogatory tone, as though the same program in Java or C is more honorable than using Perl or Python. The result is a collection of dreaded "one offs," scripts and lack of documentation, because the perceived value is in the result, not in the process.

Let's take an example. Monitoring products such as Zabbix, Zenoss or Nagios can provide a useful framework for your monitoring infrastructure but they rarely fill every need. You inevitably create custom monitors, configurations and plug-ins to fill the holes and get the job done. The question is, when you do that work, do you simply write the plug-in and get it running or do you create, polish, document and package it -- and perhaps create a roadmap for future improvements? Put another way, are you engineering hacks or products?

The required effort to transform a hack into a product isn't significant, but it does require a different mindset. Let's look at the steps involved in both approaches to see how they differ.

Creating a hack:
  1. Identify the problem.
  2. Research methods to solve the problem and choose the best (or most expedient) solution.
  3. Implement the solution.
  4. Test the solution.
  5. Modify or improve the solution until it solves the original problem.
  6. Done.

Here we see the most common approach. Figure it out, slap it together, keep hacking on it until it works, call it done and move on.

You solved the problem, people are happy, you can go home and have a beer, but what happens when you need to scale this? What do you do when six months later it breaks and you forgot what you did? I don't know a single technologist who hasn't at some point looked at code and wondered who wrote it, only to realize he was the one who wrote it. This is the essence of a "one off" solution, and it's poor practice.

Now let's look at the same process, but with a product mindset.

Creating a product:

  1. Identify the problem.
  2. Research methods to solve the problem and choose the best (or most expedient) solution.
  3. Create a new directory/project in a revision control system.
  4. Create a README and include the problem you are attempting to solve, the solution you've chosen and why.
  5. Implement the solution, document both inline and update the READ ME with your thought process.
  6. Test a rollout of the solution.
  7. Revise the solution, checking changes between each test.
  8. When complete, shape into an easy-to-install form as appropriate (RPM, tarball, installer script, etc.)
  9. Test deployment and revise as required.
  10. Tag the repository and move the easy-to-install package to a central repository (FTP server, Web server, etc.) with your other tools.
  11. Done.

At the outset this may appear far more burdensome, but if you have the appropriate mindset it's just common sense. Even if this is just a configuration file or 20-line shell script, you've produced something with long-term value. If you ever need to modify the solution or implement something similar you can quickly come up to speed and complete the task, producing either a new version of the product or a fork thereof. With the little extra effort you can go from churning out dirty hacks to engineering professional and re-usable products with increased value to your employer and other employees who may inherit your responsibilities.

And if you think I'm kidding about doing this for something as simple as a configuration file, think again. Its very easy to fall into the delusion that "everyone knows this" or "this is how everyone does it" or "this isn't that important." When you sit down to solve a problem for the first time you have an entirely different perspective on the problem than you do when you later replicate or tweak it. More time, research and thought is involved in that initial phase than later, when you simply zone in on something specific or duplicate. Even if its just a MySQL configuration, adding liberal comments to the my.cnf and then putting it in a tarball with a READ ME will add tremendous value.

What happens when you don't put in place a process like this? More often than not you simply go back to the drawing board and re-implement the solution from scratch, because it puts you back in that initial perspective we talked about. It's easier to do it again rather than understand the original solution, so you -- yet again -- re-invent the wheel.

Every product needs a process, but a good product also requires something else: a product manager. If you work in an environment with other administrators, it's important that you not become overly distributed. While everyone may share in a particular duty, such as our monitoring example, it's most effective to have one person with a broader perspective. How do multiple products and solutions fit together? How do they improve over time? This person doesn't need to do all the work, but he needs to see the big picture and keep everyone fighting in the right direction.

If the idea of a product manager seems completely crazy, ask yourself, "Who really knows what's going on with X?" In a lot of companies, if you simply ask, "Who's in charge of monitoring?" you'll get a variety of conflicting answers or shrugs. If that's your organization, sit down and list your department's responsibilities on paper and start associating names with each one. Like I said, that person doesn't need to do all the work, but he represents the go-to guy (or gal) for that particular responsibility. It can (and will) change over time, but always have a name. Once you do, trust that person, listen to his opinion and give him the authority to be successful.

In your READ MEs, include this information: Who originally designed and implemented the solution? Who is the current maintainer? Who is the product manager with the big-picture vision of the integrated solutions? These are the details that are commonly overlooked or omitted because we assume it's obvious or won't change. That kind of thinking inevitably bites us in the butt, as people come and go and years roll by.

Everything is a product, or at least I hope I've provided some compelling reasons why you should think everything is. Properly implemented, this mentality can make your organization more professional, scalable and efficient, and can even improve moral by giving you a clear sense of completion. Give it a try, always listen to the opinions of those around you for ways to hone the process, and whenever possible, contribute the results with the community at large.

ABOUT THE AUTHOR: Ben Rockwood is the Director of Systems Engineering at cloud computing infrastructure company Joyent Inc.. A Solaris expert and Sun evangelist, he lives just outside of Silicon Valley, Calif., with his smokin' hot wife Tamarah and their three children. Read his blog at

What did you think of this feature? Write to's Matt Stansberry about your data center concerns at

Dig Deeper on Real-Time Performance Monitoring and Management

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.