Dealing with network hackers in 1995

Going back to early 1995, I was working for Los Alamos National Labs as a contractor systems administrator. I didn't have a security clearance so could not work 'behind the fence' as they said. Instead, I worked with a large number of similarly uncleared post-docs, graduate students, and college interns in a strip mall converted into offices. The offices ran from nearly one end of the strip mall to the other with a large selection of Unix, PC, and Mac systems spread through the building connected together with 10base2 (or thin-wire). To make things even more fun, most of the systems were disk-less SunOS Sparc ELC/SLC and IPC systems booting off a Sparc 10 which had 64 MB of RAM and I think 2 2 GB disk drives.

The first problem I had to deal with was my most of the systems would crash at different times during the day. I got a Digital network book my Dad had given me, and learned about common problems with networking as this was not something I had dealt with before. I found that the local network was connected to a T1 which ran back to the main campus about 2 miles away. The T1 went to a hub which had 7 thin-wire lines running out of it. That seemed fine until I traced the thin-wire out. I was worried there were bad connectors (there were) or kinks in the line (there were) but the real problem was that out of the 7 thin-wire lines 3 were used.  Most of the systems were on one line. 2 (my desktop and the Sparc 10) were on another one, and the Next and SGI's were on the third. The other lines were just laying under the carpets not used. I met with my new boss Dale, and showed him what I had found. I learned a lot from Dale. He got me a copy of the Unix System Administrators Handbook and told me to start reading it on networks.

So I spent the next week or so learning how to properly crimp and connect thin-wire. I also learned testing signals and ohm resistance as I found connectors which didn't work very well. I moved all the Windows 3.11 and Macintosh 6? systems over to one set of cables and then spread the disk-less stations over to other lines in order to try and diagnose the crashes. At first, I didn't have a crash for 2-3 days and I thought everything was licked. And then the next Monday, things started crashing again.

I started investigating, but we started getting reports in a different building of a similar problem. I found a server with no disk space because of a file in /usr/tmp which seemed to be random data. I believe this was late January 1995, and I had been putting in 60-80 hour weeks trying to get caught up with things. [I was an hourly and this was allowed overtime so I was paying off my college debts pretty quickly.] I hadn't watched TV or read USENET in weeks and had no idea that there was a massive computer hunt at the time. Now when I had worked at university, I would have probably deleted the file and moved on, but because I was supporting scientists I didn't want to delete some research. I contacted Dale, and he helped me work out that the file was being written to by the rshd or telnetd command. He had me check the Ethernet port and sure enough it was in promiscuous mode. Now I had been slow up until this point but I realized this was not a good thing.

Now back in 1995, nothing on these networks was encrypted. You used telnet or rsh to login into systems. You had fingerd running on every box because you used it to see who was logged in. Most networks were on hubbed networks so you could just listen on one system and hear everything in your /24. Dale started contacting the people in security and I started looking at what other boxes were bad. It turned out several had this but one of the ones in my area which had crashed still had the source code for the sniffer daemon on it. I looked at it and found that it linked to the equivalent to tcpdump and filtered for for anything typed after Password: so it caught telnetd, rshd, ftpd and also the su command.

It then kept whatever came afterwords til a NULL and xor that data to the file it opened. I think the xor was against  libc random() stream of data using a starting string as the seed. [The string was then freed I guess to make sure gdb couldn't find it.] To de-crypt the data you just ran the same command with I think a -u and the same password. You could set it up to log even more data at compile time so I was trying that out to see if there was anything I could see about what had been captured on different systems.

At this point I had gotten it running on the system and was looking at what it was capturing when along came the attacker. They logged into the system as a normal user and ran a program ... I had missed which was setuid. They then checked what they had captured and logged out. In doing so I had captured the password they were using. I also had found that they had been running finger against a bunch of systems looking for a particular login. It turned out that a prominent researcher had collaborated with a professor at Los Alamos and the systems with their accounts were being targeted. I turned this all over to my boss who asked me if any of that meant anything to me. It didn't. He told me to go read some particular Usenet groups and that we would have to reinstall everything. [I believe he had been trying to get everything installed, updated and secured for years but it had been a low priority in his management chain.]

For a while it was thought that the attacker may have been on particular person, but these days I doubt it very much. There were a lot of people trying to be 'helpful' at that time by doing various things, and I expect it was one of them. It was an eye opening experience and I wanted to capture it here for the amount of naivete we had back then:

1. No firewall because it was meant to be an educational section where students and professors should just be able to telnet from their University to whatever computer they needed.
2. Most of the passwords were stored in a centralized space anyone could look at the hashes for. The Sun yellow page system was useful in mass deployments but had its limits. All of the systems stored their passwords in /etc/password so if you got onto the system at all you could see the hash.
3. Networks were generally hubbed so that anyone could listen to anyone else locally. This was considered less of a problem when only Unix systems were around because there was a 'separation' of root and user so you could 'control' who could look at the network. The growing number of Mac and PC's which allowed anyone to listen made this the next part hard.
4. Network communication was mostly in the clear. This was due in part because encryption was expensive on CPUs but it was also that encryption was export controlled so no one wanted to use it in case they got in trouble for it 'leaking'. 
5. Most systems came out of the box with many network services turned on which never needed to be. Just as these days, your Facebook or LinkedIN account starts off public to the universe, your computer would share if you were logged in, where you had logged in from, what your .plan might have in it and a dozen other things. 
6. That security problems tend to travel along lines of social trust. The attackers were following along various people who had worked with one researcher and using each set of systems to jump to the next ones. One of the computers hacked was done via a talk session where someone asked someone else if they could help them. I think the victim thought they were helping their adviser with a temporary .rlogin and it escalated from there. 
7. While Howard Tayler would put this better years later, I found that in security, failure is always going to happen. What is important is how to react to it. I made sure that I could always backup and restore systems en-mass if needed. I also made sure that I have a plan B and C for when the next security problem occurs. 

It would be great if that naivete was unique to that time frame, but we constantly reinvent ways to think we are completely safe.. only to find we covered ourselves in steak-sauce and laid down in the hungry lions pit. 

Addendum: I am recalling things from over 20 years ago and like a fish tale.. some things which make me look better will have grown in that time, and other things which would make me look like an idiot will have faded away..