These are things that I have found most developers of system administration tools and configurations don't get. Looking at some of the insane configuration tools from AIX SMIT to HPUX SAM, we end up with configuation tools that you require about 6 months of deep training to be able to get it to boot and stay up reasonably well. Yes it does inspire a certain kind of brand loyalty.. a sort of passive aggressive one that is also something BOFH's and SA's are known for. For my part I try to evaluate any tool or product by what I call the 2am pager incident:
Take a product and get it running. Then spend 24-48 hours awake working on some other hot project that has to be done by X. After you finally get to sleep, set up the pager to wake you at 2am. If you can rebuild, configure and get the product working by 6am it can be used in production. If it can't be it is too complex or unreliable to be useful in any server environment. I came up with this test after having this happen multiple times at both my first consulting job and then my first job at a startup. I would never deal with Digital OSF or HPUX again because of this complexity, but found Solaris 2.4 and Red Hat Linux 3 to be perfectly workable (AIX was just fun to have smitty run and fall over.. at 6 am its always funny).
The second test is to get the new administrator to get it working. Always give them a 24-48 hour deadline and then 'break' it just after it has been put up. If the new administrator can fix the system without having to call technical support its a good sign. On the other hand if the sysadmin quits or sets the box on fire... its probably not something you want in production.
While I have been using these tests for 10+ years now to good results, I have found that most senior sysadmins have similar rules (though the can you configure it after a quart of jagermeister is just too extreme for me). In the end they come down to the following:
- Assume that your customer is under-trained, tired, but does not want too much to get in their way of finding and fixing a problem. Clippy should only be there if you can set him on fire at 6am.
- Put configuration files in obvious places. Putting some configuration files in /etc and others in /var/lib/moo and some elsewhere is definitely a killer."
- Document what flags do. No one wants to find out right before the presentation that '--clean' cleans up bad data but '--clean --clean' reformats the whole Database.
- Make sure that commands are simple, easy to remember and
- "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" - Brian W. Kernighan and P.J. Plauger, The Elements of Programming Style, Second Edition
- "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction." -- Albert Einstein
- "Everything should be made as simple as possible, but not simpler." -- [possibly] Albert Einstein