NetWorker 25th Anniversary Celebration - Share Your Stories

Like many people I was dragged into the backup and storage world unintentionally.  I worked for a UK bank for 4 years during which I left for another job which fell through due to personal issues, then went back as a contractor.  A few months later their IT was outsourced and I went across with that until they decided they had too many staff (the outsourcer had just lost another contract and had brought all their existing staff on board who tended to take priority) and got rid of all contractors, except for me for whom they found a team leader role in their Remote Systems Support department, which I went into as a contractor with the intention to go back to being permanent.  This was to be a trial by fire.

This department monitored all servers to check they were working okay and all backups were completing daily. The previous team leader had spent his time reading Legato manuals before getting a job with the reseller providing Legato support which is surprising as the system was a total mess.  Since Nick, as we will call him, did not really do anything except read Legato manuals I inherited a team of 6 people all who were unenthusiastic and looking for different jobs and a system that rarely had above 60% success rate of backups.  I had no experience of Legato before, but for 6 months I had sat on the other side of the partition to the girl that initially supported it who spent literally all day talking, divided equally three ways between her wedding preparations, what was discussed at her "cross-stitch" evening (basically a group of ladies meet up and slag off their partners performance both in and out of the bedroom) and how much she hated Legato.  She hated Legato more than she hated her fiancee. So backups was not looking to be a fun experience.

Second day into the role the Legato 5.5 backup server died and with my limited knowledge it seemed as if it was trying to do too much with too little.  We called the reseller for phone support (it turned out to be Nick) who was fairly helpless but as evening approached we got them to find a senior consultant to help (for a price!)  A few hours later I had learned the mmrecover command, the nsrim command and the nsrck commands, all about WISS errors and we had the server up and running.  Their consultant pointed out that the system was severely overloaded and would probably not be stable.  They were right, it spent several more times down over the next few weeks.

Over the next two months we got the system sorted out thanks to some long hours - one week I hit 140 hours - all monitoring processes sorted out (mostly scripted, I thought I wrote good scripts but one of the kids working for me sorted it out during the night shift, when you looked at his scripts it was like reading Shakespeare), we had a new server going in, I'd mapped out training plans for the team and had some kind of agreement for an additional person so there was breathing space for holidays and time off for training, we were going to get the department moved in the building so the night shifts would no longer be 12 hours alone (with a TV in the room too) and backup success went up to 96%.

However, during this time there was an internal coup, my manager went on holiday and another manager had him kicked off the job to expand his little empire and in doing this he brought with him a complete psychopath who had been promised my role and I was standing in the way.  I eventually got annoyed by his interfering whenever I was away from my desk and quit, I'd already had an earlier experience of a bullying manager and wasn't going to put up with it again.  I was through with IT, I had a nice sideline selling records and I no longer needed a "real job".

Somehow my CV remained on Jobserve and 6 months down the line I got a call offering me a contract-to-permanent with a Legato reseller called Redstor who are a nice bunch of guys and I would still recommend them   I was quite surprised at this as I didn't think my two months experience would add up to much.  I'm glad they talked me out of retirement as it was a great learning opportunity supporting big and small companies.  One of the most amusing stories I have from this period is the police station that had to recover their backup server because their backup server had been stolen; it was also one of the most tragic stories as their bootstraps had been failing on and off for 3 months and they had only one recent backup so they put the tape on the shelf until the replacement server arrived, but someone needed a tape and thought it was a blank one and overwrote it.  An important lesson in ensuring you have a bootstrap backup daily!  Another story was one of our biggest customers who suddenly had backups failing for a few weeks with SCSI bus resets.  It was a total mystery why it was happening and we checked everything and even went as far as doing things like putting temperature probes in the cabinets.  We had a big grilling from their CEO (a little guy who reminded me of Mr Reindeer in Wild At Heart) and the problem remained a total mystery until it was discovered (I cannot remember how) that the security guard would go up and check the data centre and when he had covered everything he radioed to his partner that everything was okay, the radio signal interfered with the SAN causing a SCSI bus reset to be sent.  Sometimes its not your fault   They were suddenly happy again and continued giving us lots of lucrative business.

5 years on after a couple of bad managers following the Technical Director's departure (both of whom got fired) I decided it was time to move on and went to another reseller who will remain nameless because they were very incompetent.  The managers sat in offices and communicated with you through their secretaries and the salespeople were complete muppets, I sold more than the whole sales department in the 11 months I was there even though I wasn't the one driving around in Aston Martins and Porsches and feeling important.  The long term consultants picked and chose what they wanted to do which was not much unless it was near their home and easy which meant I did most of the consults as well as doing the phone support too (phone support being below the long term consultants remit).  The CEO would come out at the end of the day shouting and swearing that we should go home so he could lock up, usually while you were on the phone to some big company, very embarrassing really.  One time I was doing a week consultancy in Manchester and they expected me drive 400 miles daily rather than getting me a hotel (they did eventually relent).  They also had some training agreement with EMC which was great as I went into EMC's offices and taught the Legato course once a month and the long term consultants were able to take additional training courses in return.  I liked being away from their office and sort of not their employee for the day

I literally was dropped in the teaching, getting emailed a PDF at 4pm and being told I was teaching in Brentford the next day!  I had some dodgy ratings on my first course, sorry if anyone reading this was one of them, I was flying by the seat of my pants and learning from experience, but these improved over time.  EMC used to give everyone a set of UNIX and Windows notes each with a study guide and the same Command Reference Guide and each student often only took one away with them so each course I took back with me 5-10 Command Reference Guides which I gave to customers I went onsite to and it always put me in their good books

I also wrote my own courses and delivered them (and I sold them too though never got any commission!)  I was making a good pile of cash for the company while no one else was pulling their weight and I was not happy.  They knew I was disgruntled with them as they blocked me taking time off (even cancelling the time off I had for my CIMA exams which I had paid for and thus lost the fees on) which meant I ended up finding another role with an interview at a service station at 7am on the way to a consult.  They were sore losers, they did not let me take off the 3 weeks holiday that remained and made me stay with them for a month and paid me for the unused holiday.  That was my last permie job, no plans to go back there ever again after that experience   As for that company they went bankrupt not long after but have had a few buyouts and further bankruptcies since.

So resumes my days as a contractor.  My client put me into a well known mobile telco that was totally paranoid about making any changes - it was here I first worked with Networker 4, as well as Networker 7.3 with the new jobs database and GUI, which finally spelled the end for WISS errors (actually I think the resource database did this which was 7.2.2).  Most of this contract was spent battling the change management team who had a system that was half Franz Kafka and half Edgar Allen Poe.  We had a small office and I worked with a Scottish guy who was really clever but the slowest guy I have ever worked with.  Then there was the saleswoman who was a plump Indian girl with buck teeth who always started her professional phone conversations with "Do you remember that very sexy girl you met last week...", she had a habit of interfering with whatever we were doing like booking meetings for us with people we already had meetings booked with.  The storage department was ran by a guy straight out of some bad sitcom, on the day they found out the IT functions were going to be outsourced to Germany he went around making heil Hitler gestures, oblivious to the fact everyone found it extremely embarrassing.  My project manager was a timid straight-edged guy who was terrified when we had to get final approval from work we had done from a rocker guy with long hair and a few tattoos who he was convinced was a Satanist, it was kind of funny to watch him visibly quivering there.  I also did some presales for them (unfortunately the customer chose EMC over the NetApp we were selling) and some consultancy for DiskXtender which I learned on the train journey up to Sheffield as their "expert" had only been on the course and did not know much, the customer complained he had been on the course at the same time as their consultant, but was happy they had sent someone who knew what they were doing and I fixed all his problems and talked him round to liking it again, he never figured out I was winging it!

Next contract, big investment bank.  Shortest interview ever (about 2 minutes) and it was mostly me asking the questions!  Usual story, no investment in infrastructure, no investment in staff, staff leave, throw a lot of money at contractors to get it working again.  600 calls in the queue (mostly restore requests) which 4 of us got down to 50 in a month.  We had 3000 servers being backed up and a team of guys dedicated just to carrying tapes around,   A huge environment.  One project I had was a pile of legacy tapes for Africa and the Middle East from 10 years previous that they had decided to bring back from India as they felt the UK was geopolitically safer.  It was not helped by the project manager leaving and signing off the project as complete before testing so he could get the bonus before he left, it turned out he had ordered the wrong DLT drive and finding a good DLT2000 drive was near impossible.  You would think you would never need to recover 10 year old data but there was a plane impounded that could not be released until proof of ownership was produced - sorry if you were one of those people whose flight from Mauritius was cancelled that night, we did get the data back the next day and the plane went on its way immediately after

It was there I got my first taste of EMC Data Protection Advisor which at that time was called EMC Backup Advisor; I put in an amazing reporting system only to have it ripped out due to some hard sell by a competitor and the replacement product was a piece of junk that required a huge amount of scripting to get it to work (I really advanced my shell and PERL scripting and macro programming thanks to this competitor).

Bored after a couple of years I went and worked on a project for a big engineering company in the Midlands - thankfully getting out before I would probably have been pushed with the imminent financial crisis that was  due to come in a few months time.  This was a go-in-and-fix-a-bad-install type of consult.  The customer had thrown out Netbackup and replaced it with Networker and an EMC Data Library and were disenchanted by the fact that it didn't work very well.  The consultants that had put it all in had evidently not known what they were doing so it was planned and installed wrong, undocumented and staff had no training.  6 months later I had re-engineered the solution and it worked like clockwork, everyone knew how everything worked and the manager had worked on an amazing reporting script that meant there would be no need for EBA/DPA here.  Another happy customer!  It was a fairly sparse environment and we had to walk across the factory floor with 20 foot steel turbines everywhere to get the offsite tapes, it was always exciting to see all that engineering work going on.

Next off to Luxembourg where I worked for European Commission which again was a huge environment and another great bunch of guys to work with who were very tolerant of my inability to speak French (much better these days thanks to my time there).  I would have stayed there longer if someone would rent me an apartment but in France (where we wanted to stay) they won't rent an apartment to English people, and we eventually got tired of our basic one room aparthotel - summer was obscenely hot and myself and my wife would sit in the car in the evenings to take advantage of the air conditioning.  There was a Belgian guy who always used to greet me in German - I thought he must be German but it turned out he thought I was German English guys out there are rare, apparently they get homesick!  All meetings are meant to be in English but as I was the only English person everything was always in French  - I can follow one conversation but when it degenerates into 10 conversations I tend to get lost

One of my first jobs there was sorting out their cloning, they had a 1400 line script that wasn't working, 800 lines which turned out to be in subroutines that were never accessed at all!  For some reason it never checked whether anything had been already cloned so I found some savesets that had been cloned hundreds of times!  Despite having a management company that stole EU3000 from me and living hand-to-mouth due to high taxation, high living costs, taking 14 weeks to get paid, and a fall in Euro of 20% between signing contact and first payment I enjoyed it a lot, we saw a lot of France, Belgium, Luxembourg and Germany and I would be very happy to go back there again.

So we returned back to the UK to find that the financial crisis had pretty much decimated the employment landscape and I had two years pretty much being unwanted except for one two month contract that was very exploitative (I was almost losing money to do it!).   I was figuring there was no longer any IT roles left anymore and wondering what to do next (again I had fallen back on selling records but wasn't sure I wanted to do it as a career) when I got a call at 9am from a lady whom I pretty much grunted "yes" to and very little else when asked if I knew anything about Networker, then called me an hour later and asked if I can start the following week.  This was followed by regular calls knocking this back a week at a time for about 2 months until it really did happen and I found myself "Technical Lead" for a global engineering company and working on a massive modernization and decommissioning project.  Sometimes things happen just like that...

Another 1500 servers, some Networker 5 found, (Networker 8.1 had just come out), the only thing older than the servers (some of which had not been updated for 15 years or more) were the staff, it was the only place where we regularly had presentations for staff who were retiring after 40 years in the IT department, my exceptional colleague Mr Alan Sadler retired after 48 years in their IT department, he was still one of the sharpest minds there!  This was the first place I worked with DataDomains, they blew away the EDLs and tape libraries in terms of reliability   Pretty much business as usual on this contract: refresh hardware, upgrade software, fix cloning, do reporting, install EMC DPA and do reporting with that, get annoyed with DBAs.  An enjoyable time so much so I ended up moving here, finally moving in the week before the contract finished!  I could tell the tale of how the backup server just lost its resource files and died, but as my colleague whose SSH session from the production backup server to the one he was decommissioning closed while he was gossiping now is working for EMC it may prove embarrassing to him

Good timing, I now have a couple of months to move in to my new home (I threw out a few vintage media packs during the move, perhaps I should have put them on eBay), but not long enough it seems, next I am off to Newbury to sort out another modernization project.  A big refresh with VBlock architecture, VMWare, DataDomain and Networker using new Virtual Backup Appliance technology (kicking out several other backup solutions).  A real modernisation project, everything here is latest and greatest technology.  First thing I thought when we had it up and running was "that is going to put me out of a job", it was so straightforward and efficient.  However, my friend in EMC support says they get loads of calls on VBAs so perhaps we were lucky, or they had a good consultant put it in

Which brings me to the present.  16.5 years working as a Networker specialist and finally I end up working with EMC!  Another happy customer here as we refresh their systems and hopefully we can sell them DPA too.  I'm still not sure who or where I will be in a couple of weeks when we complete this but I am sure Networker will provide some kind of interesting work and exciting opportunities for new experiences.

So Networker has been an exciting product to work with all these years.  It still surprises me how the same old problems still crop up again and again - bad DNS / name resolutions, failure to invest except during a panic or after data loss, failure to keep software levels current, not checking what you think is happening is, not hiring a consultant when necessary or getting one that is unreliable, not checking restores - I always used to say at start of courses EMC don't make backup software they make restore software.