2020-12-18

Reading: Peopleware: Productive Projects and Team by Tom DeMarco and Timothy Lister

So last month I finished a re-read of "The Mythical Man-Month" by Fred Brooks.. and saw in the back of the book a reference to "Peopleware:..." by Tom DeMarco and Timothy Lister. I had heard of the book in the late 1990's (right before I joined Red Hat for the first time) from a fellow coder who was moving  into management and had wanted to do it right. Marshall had read the book, tried to put the ideas into place and was promptly let go by the company. [Within about 6 months, we were all let go also... for a reason given in the book..] In the years since I have seen the book brought  up several times, but I had never actually read it.

First off, the book is very ranty. Very very ranty in parts about open floor plans (and the Furniture police), open organizations, and a couple of other areas of middle management. It is not that I found their comments off the mark.. but you may have to soldier onto the next chapter through a couple of them. That said there are a lot of gems in the book which 'spoke to me'.

Parkinson's Law Doesn't Exist

Work expands to fill the time allocated to it. - C.  NorthCote Parkinson

I have had several managers quote this to teams I have been on in the last 30 years. People would bring up Parkinson's law about why we shouldn't add more time to a schedule but should cut it back. I, myself took it for certain that this was some sociological experiment which proved things about scheduling. I never realized until reading this book that a Mr. Parkinson was not a scientist, but a satirist and the things people have 'quoted' as being proven by it originally were all made up for a humorous article and books he wrote. 

In the extent that there may be places where people come up with busy work to meet a schedule... it usually is where people are not finding satisfaction in their current job. [chapter 5]

Open Floor Plans Don't Help Programmers Flow

This book was originally written in the 1980's when open floor plans were having 6 foot high cubicles versus offices with doors. Back then there was a lot of recorded data about how various 'knowledge' workers can't get into a flow state of thought because there are too many distractions. Today's open floor plans of no cubicle walls and brutalist concrete structures in order to get some sort of energy efficiency put those to shame. [This is where the rants on furniture police start and go on for quite some time.. ] The useful information I took from this is that one should measure their uninterrupted time and not their hours. Doing this can show where productivity is actually going and work out better planning on how efficient a project can meet its goals.

Hiring The Right People to Teamicide

This section of the book has a lot of useful parts, but one that stood out to me was the chapter "Happy to Be Here" with an excerpt from "Up the Organization" by Robert Townsend. It basically explains a method for dealing with an underperforming organization by moving it to a new location. The purported guaranteed results:
  1. the best people will move to the new location.
  2. the rest don't have to face the fact they have been fired. Instead they can say "The company moved town."
  3. your competition will move in and hire these less desirables thinking they are getting a bargain.
  4. the new people at your new site are infused with enthusiasm because all they know are your 'best' people. 
This was basically the strategy used by that company I worked for years before Red Hat. We were all told the company was moving town and we could reapply for our jobs at a different location. The problem they ran into is the one mentioned in the book.. most of the best people didn't go (they ended up being hired as contractors and paid 2.5x what they had been paid before so that deadlines could be met), and a lot of the people who did weren't the ones they wanted. Those of the best  people who did go left shortly after their moves. The psychological loyalty they had with the company had been broken so any problem at the new place was enough to send them packing. 

The next sections cover more about how to make a team of people "jell" which ends up being harder to describe than the ways to make a team break down. 
  1.  Defensive Management
  2. Bureaucracy
  3. Physical Seperation
  4. Fragmentation of people's time
  5. Quality of reduction of the product
  6. Phony deadlines
  7. Clique Control
  8. "Motivational" posters and awards
  9. Overtime causing undertime
  10. Internal team competition
  11. Annual salary/merit reviews
  12. Management by Objectives
I can see where some of the above are causes of team breakdowns but others seem to be a lot smaller concerns comparatively.

Making Change Possible

The chapter opens with a quote by Steve McMenamin, of the Atlantic Systems Guild (the organization the authors are members of).

People hate change... and that's because people hate change... I want you to be sure that you get my point. People really hate change. They really, really do. 

People don't react to change logically and very little 'logical argument' will help them. People react to change emotionally and the most likely result is going to be failure and ridicule from ones peers and sometimes by the people who instituted the change. The book also brings up the idea of Jerry Johnson, Menninger Foundation, that there is a continuum of ways people react to the change:

  1. Blindingly Loyal (Ask no questions.)
  2. Questioners
    1. skeptics (show me)
    2. passive (what's in it for me?)
    3. opposed due to fear of change
    4. opposed due to fear of loss of knowledge/power
  3. Militantly Opposed (Will Undermine and Destroy).
While some people think that the second 2 sets are 'non-supporters', the Blindingly loyal are also the ones to most likely jump ship at the next newest thing. The only people who will stick it through are the various questioners if they are helped through the emotional journey through the initial Chaos and failures before the change can be truly evaluated. [The rest of the chapter goes into parts of that while referencing books like "The Prince" by Machiavelli and Virginia Satir's model of therapy change. I actually wish this chapter had been a bit more in depth but I think I will reread Managing Transitions (which was referenced in different parts of the book) instead.

 


2020-12-14

CentOS-8 End of Life 2021-12-31

What is Happening

Red Hat announced changes to CentOS Linux release structure last week, and I like everyone else around is working through what that means. I have worked with and on CentOS since 2005, and saw a lot of my work in EPEL as helping people in that community able to get things done. This has been a real kick to the guts for a lot of people, and I do not have the words to express myself on this. That said, as a System Administrator, you have to take the Tango Charlie Foxtrot's as they are handed to you, plan what to do next, and execute as efficiently as possible.

Current End of Life Dates for CentOS Releases

The following seem to be the current dates to plan against:

CentOS Linux ReleaseCurrent End of Life
CentOS 62020-11-30
CentOS 72024-06-30
CentOS 82021-12-31
CentOS Stream ReleaseCurrent End of Life
CentOS Stream 82024-06-??

What Do I Need to do Next

Deal with the Grief

At this point, System Administrators need to start making a lot of assessments of what they are using CentOS for, why they are using it, and what are the alternatives to move to. This is hard to do when you are angry, pissed, depressed, betrayed, enraged, apoplectic, or the other dozen emotions that have been going through my head lately. So the first thing one has to do is try to work on those emotions as best you can. Talk them out with someone who can take one more rant from you, write them in a blog you never publish, and go for long walks away from computers for several hours. [There are a lot of other methods too and each person has their own way to deal with this.. but the point is that until you get the emotions out, you can not operate.]

Do an analytical assessment of why you are currently using CentOS Linux

There are a plethora of reasons why people use CentOS in their environments.. and those need to be examined to figure out what you need to do next. Sadly like most environments, I think most System Administrators rarely have time to document these and so when having to redeploy you end up with what seems like an impossible project. Like all Charlie Tango Foxtrot situations, one needs to eat the elephant one bite at a time. I find that the following can help make this easier. For each set of servers/services you answer:

  1. Who are the consumers of this service or server? 
  2. What do they need from their services?
  3. What is the amount of change that they can deal with on this service?
  4. How many systems do I have in this service? How can I break them into smaller chunks?
  5. Where are my systems? [How are
  6. What automation can I use to roll out these changes?
  7. When are upcoming deadlines that my consumers need to deal with
  8. When are upcoming deadlines I have to deal with?
  9. What are my alternatives? (This is a larger question needing a breakout sheet of why this alternative, how close is it to meeting my needs, what are the costs involved, when can I switch to it, where can i run it, what do I need to change from what I do now?, etc)
  10. How long would it take to work these alternatives?

If you have hundreds of servers, then this still looks like a giant load of dung to crawl through. I start with my central Infrastructure server and work my way out.

  1. Who are the consumers? [I am as use this server to run config scripts to outlying systems.]
  2. What do they need from their services? [Good uptime and stability so that when an outage happens elsewhere I have a good homebase to work from.]
  3. What is the amount of change they can deal with on this service? [If I use this as a learning experience, a lot. If I do not have time to learn.. then not a lot.]
  4. How many systems do I have in this service? 1
  5. Where are my systems? [Home base is under my desk in my office.]
  6. What automaton can I use to roll out these changes? Ansible looks like a good choice
  7. When are upcoming deadlines that my consumers need to deal with? [check with home calendar and mark off dates when I can actually futz with this.]
  8. When are upcoming deadlines that I have to deal with? 2020-12-31 
  9. What are my alternatives? [Only listing some alternatives.. rationale needs to be filled out]
  10. How long would it take to work these alternatives? ... unknown at this time.

This list is a starting point for getting the conversation going with some questions not needed and others needing to be added. Once you have worked out the questions then it is time to work out with each of the stakeholders of your services to answer the questions for themselves and then come to a conclusion. For deadlines to work this out, I would use the rule of thumb I found when dealing with change over of Fedora services with several hundred systems and also previous Tango Foxtrot Charlie changes:

StepSoft DeadlineHard deadline
Inventory current systems2021-01-312021-02-15
Evaluate alternatives2021-02-282021-03-15
Write automation rules for minimal services2021-03-312021-04-15
Roll out staged minimal new infra using different choices if possible2021-04-302021-05-15
Evaluate rollout and go with best choice2021-05-152020-05-31
Determine convergence planning for additional services2021-05-152020-05-31
Begin Rollout of New Infra2021-06-302021-07-15
A series of miracles occur sorry I mean lots of planned deliverables are met.


Last systems are moved over.2021-12-312020-12-31
Last updates for CentOS-82020-12-312020-12-31

Note: None of the above is to say 'golly gee, its no problem, you just need to throw away all your other work plans for 2021.' or any similar flipant response. It is only meant as a possible way to deal with this in a sane way. Some ways of converting/change are easier than others, but they all require some work and knowing what the tradeoffs are for each one.

2020-10-29

RHEL-6/CentOS-6/SciLin-6/EPEL-6 End Of Life Notice 2020-11-30

CentOS-6 End of Life Notice 2020-11-30

This is a short reminder that Red Hat Enterprise Linux (RHEL) version 6 will enter 'Extended Lifetime Support' in about 30 days from when I am writing this. Extended Lifetime Support (ELS) is a specific contract with Red Hat for them to cover certain security fixes for some extended time to allow sites some time for last minute transitions. 

RHEL-6 was released in November of 2010, and was the first RHEL I got to work with/on after I returned to Red Hat in 2009. The release has seen 10 minor releases (1 less than RHEL-5), and has been in 'extended' mode since the last 6.10 release in June 2018. 

What does this mean to me?

When RHEL-6 enters its version of ELS, then it is considered 'end of life' by its 'downstream' rebuilders CentOS and Scientific Linux (and by extension EPEL). I am not sure what Scientific Linux plans are, but for CentOS, they will follow a similar plan as they did with EL-4/EL-5 distributions end. First they will copy all the code from the master servers to the vault servers. Then they will turn off the main mirrorlist systems any references to EL-6. This will cause the 'yum' command to fail and usually causes all kinds of yelling and screaming as people realize they needed to have moved their servers to something else before December 1, 2020. 

For EPEL users it will be slightly different. All EPEL builds for EL-6 will be 'stopped' on December 1st.  [Packages will no longer be allowed to branch to EL-6, builders will not be able to build code for EL-6. Packagers will not be able to move software from testing to prod.] At that point all RPMS on the download servers will be hardlinked to /pub/archive on the master servers. After a week, the mirrorlists will point to the archive zone, and Fedora Infrastructure will remove the code from /pub/epel/6/ trees. 

What can I do to deal with this?

Primarily, if you are going to be affected by the end of EL-6 services, you either need to get an ELS contract, move to another OS, or move to self-support. In order to self-support, you will need to mirror the source code from your distribution provider and learn the basics of RPM building. If you are on CentOS and find your servers not able to do yum installs anymore.. you will need to mirror the EL-6 from the CentOS vault somewhere locally and use that as your new 'mirror'. Depending on time and energy, I will try to outline some of these steps in future blog posts. 

2019-10-07

Happy Halloween (Packages Not In EPEL-8 yet)

It is October, and in the US it means that all the decorations for Halloween are going up. This is a time of year I love because you get to dress up in a costume and give gifts to people. In the spirit of Halloween, I am going to make various packages available in a COPR to add onto the EPEL-8 repositories.

There are a lot of packages which are in EPEL-6 or EPEL-7 but are not in EPEL-8 yet. Some of these may not be possible due to missing -devel, others may just need someone interested in maintaining a branch for EPEL-8, etc etc. In order to try and get a push on this I wanted to see what packages could be built and made ready at some point. I also wanted to make it possible that if you really needed this package, that they could be available.

Important notes:

  1. These packages will not be getting updates
  2.  These packages will not be something you can file a bugzilla on if they don't work. 
  3. If they turn out to be filled with goblins, you have been warned. 
  4. If your system starts glowing green and moaning about Elder Gods.. I take no responsibility.

That said, this is how I am building these packages in case you want to do this also:


#Look up package src git repository
$ kinit
$ fedpkg clone {src name}
$ fedpkg srpm
$ mock --chain -r epel-8-x86_64 --localrepo=/home/smooge/not-yet-in-epel8/
# See what packages were needed to build or fix any spec file changes
# if packages needed start chain of packages and fixes
$ copr-cli build not-yet-in-epel8 {src.rpm 1} {src.rpm 2} 
# see what failures occurred.. see if they are fixable.

Currently packages which are not fixable are anything using perl-generators. There are 2 perl-generators in RHEL/CentOS-8 with one of them being a fully built module and one being a pseudo-module. The mock configs use best=1 which causes the fully built module perl to be pulled in.. which is the wrong package as it is built against perl-5.24 and the main perl is perl-5.26. Other packages which will not build are ones which need items which RHEL/CentOS-8 did not ship at all (various -devel and such).

In any case, the copr is at https://copr.fedorainfracloud.org/coprs/smooge/not-yet-in-epel8/ . I will maintain this repository until next March when it like all Halloween themed candy is not even sold at the dollar store anymore.

2019-10-01

Attention: Removal of python36 from EPEL-7 on 2019-10-03

With the release of RHEL-7.7, many of the packages for python36 in EPEL were replicated in the release as python3-3.6 packages. The normal pattern when this is seen is to remove the packages from EPEL so that they do not cause problems. However, this did cause problems for users of CentOS-7 who did not have access to the newer packages. Two weeks ago, CentOS-7.7.1908 was released and should have flowed out to users as needed.

So it is time to remove the following src.rpm packages from EPEL:

python36-3.6.8-1.el7.src.rpm
python3-setuptools-39.2.0-3.el7.src.rpm


As they are duplicated by:
python3-3.6.8-10.el7.src.rpm
python3-setuptools-39.2.0-10.el7.src.rpm

We will be removing the python packages on 2019-10-03 so that they should disappear during the repository compose on 2019-10-04. EPEL is a rolling release locked against the latest state of the Red Hat Enterprise Linux repositories. If you are using an older snapshot of RHEL or CentOS, you should sync down versions of the repository and lock particular versions for your use.

2019-09-19

Attention: Fedora Yahoo Email Users

Going from a blast of the past we are currently going through one of the Yahoo is not allowing many emails with either fedoraproject.org OR from our mail routers.  It would seem that the way to get yahoo to blacklist a domain is to get subscribed to mailing lists and then report the lists as SPAM. Enough accounts (or maybe if one person does it enough times).. yahoo will helpfully blacklist the domain completely. [It then is usually a multi-month process of people explaining that no Fedora is not a spam site, hasn't been taken over by a spam site, or a bunch of other things which do happen so any mail admin is going to be wary on.]

The funny thing is that their blockage doesn't work 100% so some people seem to still get emails delivered even when most of our logs show that yahoo is telling our servers various SMTP errors of GO AWAY.

At this point, if you are a packager with a yahoo.com email address, you probably have not gotten an email from any lists or possibly bugzilla for a bit. Trying to email you directly from our site to tell you this isn't going to work.. so we are going back to blogs on the hopes that someone still reads them.


2019-09-16

EPEL Bug: Bash errors on recent EL-8 systems.

Last week, I got asked about a problem with using EPEL-8 on Oracle Enterprise Linux 8 where trying to install packages failed due to bad license file. I duplicated the problem on RHEL-8 which had not happened before some recent updates.

[smooge@localhost ~]$ repoquery
bash: repoquery: command not found...
Failed to search for file: Failed to download gpg key for repo 'epel': Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8.0 [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8.0]
The problem seems to be that the EPEL release package uses the string $releasever for various short-cut strings. Take for example:

[epel-playground]
name=Extra Packages for Enterprise Linux $releasever - Playground - $basearch
#baseurl=https://download.fedoraproject.org/pub/epel/playground/$releasever/Everything/$basearch/os
metalink=https://mirrors.fedoraproject.org/metalink?repo=playground-epel$releasever&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-$releasever

The problem is that when I wrote new versions of the EPEL-8 repo file, I replaced the old key phrase gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 with gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-$releasever .  When I tested things with the dnf command it worked fine but I didn't check to see where things like bash completion would show up.

Moving back to the format that EPEL-6 and EPEL-7 used fixes the problem, so I will be pushing an updated release file out this week.  My apologies for people seeing the errors.