2020-12-18

Reading: Peopleware: Productive Projects and Team by Tom DeMarco and Timothy Lister

So last month I finished a re-read of "The Mythical Man-Month" by Fred Brooks.. and saw in the back of the book a reference to "Peopleware:..." by Tom DeMarco and Timothy Lister. I had heard of the book in the late 1990's (right before I joined Red Hat for the first time) from a fellow coder who was moving  into management and had wanted to do it right. Marshall had read the book, tried to put the ideas into place and was promptly let go by the company. [Within about 6 months, we were all let go also... for a reason given in the book..] In the years since I have seen the book brought  up several times, but I had never actually read it.

First off, the book is very ranty. Very very ranty in parts about open floor plans (and the Furniture police), open organizations, and a couple of other areas of middle management. It is not that I found their comments off the mark.. but you may have to soldier onto the next chapter through a couple of them. That said there are a lot of gems in the book which 'spoke to me'.

Parkinson's Law Doesn't Exist

Work expands to fill the time allocated to it. - C.  NorthCote Parkinson

I have had several managers quote this to teams I have been on in the last 30 years. People would bring up Parkinson's law about why we shouldn't add more time to a schedule but should cut it back. I, myself took it for certain that this was some sociological experiment which proved things about scheduling. I never realized until reading this book that a Mr. Parkinson was not a scientist, but a satirist and the things people have 'quoted' as being proven by it originally were all made up for a humorous article and books he wrote. 

In the extent that there may be places where people come up with busy work to meet a schedule... it usually is where people are not finding satisfaction in their current job. [chapter 5]

Open Floor Plans Don't Help Programmers Flow

This book was originally written in the 1980's when open floor plans were having 6 foot high cubicles versus offices with doors. Back then there was a lot of recorded data about how various 'knowledge' workers can't get into a flow state of thought because there are too many distractions. Today's open floor plans of no cubicle walls and brutalist concrete structures in order to get some sort of energy efficiency put those to shame. [This is where the rants on furniture police start and go on for quite some time.. ] The useful information I took from this is that one should measure their uninterrupted time and not their hours. Doing this can show where productivity is actually going and work out better planning on how efficient a project can meet its goals.

Hiring The Right People to Teamicide

This section of the book has a lot of useful parts, but one that stood out to me was the chapter "Happy to Be Here" with an excerpt from "Up the Organization" by Robert Townsend. It basically explains a method for dealing with an underperforming organization by moving it to a new location. The purported guaranteed results:
  1. the best people will move to the new location.
  2. the rest don't have to face the fact they have been fired. Instead they can say "The company moved town."
  3. your competition will move in and hire these less desirables thinking they are getting a bargain.
  4. the new people at your new site are infused with enthusiasm because all they know are your 'best' people. 
This was basically the strategy used by that company I worked for years before Red Hat. We were all told the company was moving town and we could reapply for our jobs at a different location. The problem they ran into is the one mentioned in the book.. most of the best people didn't go (they ended up being hired as contractors and paid 2.5x what they had been paid before so that deadlines could be met), and a lot of the people who did weren't the ones they wanted. Those of the best  people who did go left shortly after their moves. The psychological loyalty they had with the company had been broken so any problem at the new place was enough to send them packing. 

The next sections cover more about how to make a team of people "jell" which ends up being harder to describe than the ways to make a team break down. 
  1.  Defensive Management
  2. Bureaucracy
  3. Physical Seperation
  4. Fragmentation of people's time
  5. Quality of reduction of the product
  6. Phony deadlines
  7. Clique Control
  8. "Motivational" posters and awards
  9. Overtime causing undertime
  10. Internal team competition
  11. Annual salary/merit reviews
  12. Management by Objectives
I can see where some of the above are causes of team breakdowns but others seem to be a lot smaller concerns comparatively.

Making Change Possible

The chapter opens with a quote by Steve McMenamin, of the Atlantic Systems Guild (the organization the authors are members of).

People hate change... and that's because people hate change... I want you to be sure that you get my point. People really hate change. They really, really do. 

People don't react to change logically and very little 'logical argument' will help them. People react to change emotionally and the most likely result is going to be failure and ridicule from ones peers and sometimes by the people who instituted the change. The book also brings up the idea of Jerry Johnson, Menninger Foundation, that there is a continuum of ways people react to the change:

  1. Blindingly Loyal (Ask no questions.)
  2. Questioners
    1. skeptics (show me)
    2. passive (what's in it for me?)
    3. opposed due to fear of change
    4. opposed due to fear of loss of knowledge/power
  3. Militantly Opposed (Will Undermine and Destroy).
While some people think that the second 2 sets are 'non-supporters', the Blindingly loyal are also the ones to most likely jump ship at the next newest thing. The only people who will stick it through are the various questioners if they are helped through the emotional journey through the initial Chaos and failures before the change can be truly evaluated. [The rest of the chapter goes into parts of that while referencing books like "The Prince" by Machiavelli and Virginia Satir's model of therapy change. I actually wish this chapter had been a bit more in depth but I think I will reread Managing Transitions (which was referenced in different parts of the book) instead.

 


2020-12-14

CentOS-8 End of Life 2021-12-31

What is Happening

Red Hat announced changes to CentOS Linux release structure last week, and I like everyone else around is working through what that means. I have worked with and on CentOS since 2005, and saw a lot of my work in EPEL as helping people in that community able to get things done. This has been a real kick to the guts for a lot of people, and I do not have the words to express myself on this. That said, as a System Administrator, you have to take the Tango Charlie Foxtrot's as they are handed to you, plan what to do next, and execute as efficiently as possible.

Current End of Life Dates for CentOS Releases

The following seem to be the current dates to plan against:

CentOS Linux ReleaseCurrent End of Life
CentOS 62020-11-30
CentOS 72024-06-30
CentOS 82021-12-31
CentOS Stream ReleaseCurrent End of Life
CentOS Stream 82024-06-??

What Do I Need to do Next

Deal with the Grief

At this point, System Administrators need to start making a lot of assessments of what they are using CentOS for, why they are using it, and what are the alternatives to move to. This is hard to do when you are angry, pissed, depressed, betrayed, enraged, apoplectic, or the other dozen emotions that have been going through my head lately. So the first thing one has to do is try to work on those emotions as best you can. Talk them out with someone who can take one more rant from you, write them in a blog you never publish, and go for long walks away from computers for several hours. [There are a lot of other methods too and each person has their own way to deal with this.. but the point is that until you get the emotions out, you can not operate.]

Do an analytical assessment of why you are currently using CentOS Linux

There are a plethora of reasons why people use CentOS in their environments.. and those need to be examined to figure out what you need to do next. Sadly like most environments, I think most System Administrators rarely have time to document these and so when having to redeploy you end up with what seems like an impossible project. Like all Charlie Tango Foxtrot situations, one needs to eat the elephant one bite at a time. I find that the following can help make this easier. For each set of servers/services you answer:

  1. Who are the consumers of this service or server? 
  2. What do they need from their services?
  3. What is the amount of change that they can deal with on this service?
  4. How many systems do I have in this service? How can I break them into smaller chunks?
  5. Where are my systems? [How are
  6. What automation can I use to roll out these changes?
  7. When are upcoming deadlines that my consumers need to deal with
  8. When are upcoming deadlines I have to deal with?
  9. What are my alternatives? (This is a larger question needing a breakout sheet of why this alternative, how close is it to meeting my needs, what are the costs involved, when can I switch to it, where can i run it, what do I need to change from what I do now?, etc)
  10. How long would it take to work these alternatives?

If you have hundreds of servers, then this still looks like a giant load of dung to crawl through. I start with my central Infrastructure server and work my way out.

  1. Who are the consumers? [I am as use this server to run config scripts to outlying systems.]
  2. What do they need from their services? [Good uptime and stability so that when an outage happens elsewhere I have a good homebase to work from.]
  3. What is the amount of change they can deal with on this service? [If I use this as a learning experience, a lot. If I do not have time to learn.. then not a lot.]
  4. How many systems do I have in this service? 1
  5. Where are my systems? [Home base is under my desk in my office.]
  6. What automaton can I use to roll out these changes? Ansible looks like a good choice
  7. When are upcoming deadlines that my consumers need to deal with? [check with home calendar and mark off dates when I can actually futz with this.]
  8. When are upcoming deadlines that I have to deal with? 2020-12-31 
  9. What are my alternatives? [Only listing some alternatives.. rationale needs to be filled out]
  10. How long would it take to work these alternatives? ... unknown at this time.

This list is a starting point for getting the conversation going with some questions not needed and others needing to be added. Once you have worked out the questions then it is time to work out with each of the stakeholders of your services to answer the questions for themselves and then come to a conclusion. For deadlines to work this out, I would use the rule of thumb I found when dealing with change over of Fedora services with several hundred systems and also previous Tango Foxtrot Charlie changes:

StepSoft DeadlineHard deadline
Inventory current systems2021-01-312021-02-15
Evaluate alternatives2021-02-282021-03-15
Write automation rules for minimal services2021-03-312021-04-15
Roll out staged minimal new infra using different choices if possible2021-04-302021-05-15
Evaluate rollout and go with best choice2021-05-152020-05-31
Determine convergence planning for additional services2021-05-152020-05-31
Begin Rollout of New Infra2021-06-302021-07-15
A series of miracles occur sorry I mean lots of planned deliverables are met.


Last systems are moved over.2021-12-312020-12-31
Last updates for CentOS-82020-12-312020-12-31

Note: None of the above is to say 'golly gee, its no problem, you just need to throw away all your other work plans for 2021.' or any similar flipant response. It is only meant as a possible way to deal with this in a sane way. Some ways of converting/change are easier than others, but they all require some work and knowing what the tradeoffs are for each one.

2020-10-29

RHEL-6/CentOS-6/SciLin-6/EPEL-6 End Of Life Notice 2020-11-30

CentOS-6 End of Life Notice 2020-11-30

This is a short reminder that Red Hat Enterprise Linux (RHEL) version 6 will enter 'Extended Lifetime Support' in about 30 days from when I am writing this. Extended Lifetime Support (ELS) is a specific contract with Red Hat for them to cover certain security fixes for some extended time to allow sites some time for last minute transitions. 

RHEL-6 was released in November of 2010, and was the first RHEL I got to work with/on after I returned to Red Hat in 2009. The release has seen 10 minor releases (1 less than RHEL-5), and has been in 'extended' mode since the last 6.10 release in June 2018. 

What does this mean to me?

When RHEL-6 enters its version of ELS, then it is considered 'end of life' by its 'downstream' rebuilders CentOS and Scientific Linux (and by extension EPEL). I am not sure what Scientific Linux plans are, but for CentOS, they will follow a similar plan as they did with EL-4/EL-5 distributions end. First they will copy all the code from the master servers to the vault servers. Then they will turn off the main mirrorlist systems any references to EL-6. This will cause the 'yum' command to fail and usually causes all kinds of yelling and screaming as people realize they needed to have moved their servers to something else before December 1, 2020. 

For EPEL users it will be slightly different. All EPEL builds for EL-6 will be 'stopped' on December 1st.  [Packages will no longer be allowed to branch to EL-6, builders will not be able to build code for EL-6. Packagers will not be able to move software from testing to prod.] At that point all RPMS on the download servers will be hardlinked to /pub/archive on the master servers. After a week, the mirrorlists will point to the archive zone, and Fedora Infrastructure will remove the code from /pub/epel/6/ trees. 

What can I do to deal with this?

Primarily, if you are going to be affected by the end of EL-6 services, you either need to get an ELS contract, move to another OS, or move to self-support. In order to self-support, you will need to mirror the source code from your distribution provider and learn the basics of RPM building. If you are on CentOS and find your servers not able to do yum installs anymore.. you will need to mirror the EL-6 from the CentOS vault somewhere locally and use that as your new 'mirror'. Depending on time and energy, I will try to outline some of these steps in future blog posts.