“In theory there is no difference between theory and practice. In practice there is.” Yogi Berra
Usually, no matter how much planning, testing, thinking about, or stalling you build in to an upgrade project, the bill comes due at the end and isn’t what you expected. Maybe someone ordered the lobster, or multiple drams of Johnny Walker Blue, but either way you have a situation to deal with… right now. How you deal with it determines many things about your skills and your character, but that’s for another post as I’m going to try my best to keep this one short and to the point.
I recently had the wonderful experience of upgrading an Active/Passive failover pair of ASA units to the newest of the new code, 9.0(2) from 8.4.2(8). After the 8.3 kerfuffle (NAT changes automagically, anyone?) I was particularly keen to not miss any possible gotchas in this upgrade. I also scheduled a larger than usual maintenance window–even though we didn’t expect any downtime–just in case.
I should interject here, for those of you aghast at the thought that we would possibly implement new, some would say, bleeding edge code on production systems, a couple of things:
- We always run bleeding edge code, usually because we have need of the bleeding edge features the code brings. In this case, IPv6 features sorely lacking in prior code versions
- We have adopted a very aggressive IPv6 stance, as I have written about before, and we tend to find our aspirations and designs are well out in front of significant portions of the code available for our equipment
- Noting the prior two items again, I’ll also add that Firewalls, in particular, seem to have code that is months or years behind the route/switch world. That holds true across all vendor platforms. Why? I don’t know, but that’s another post.
With our need-for-upgrade bona fides established, I dutifully read the entire Release notes for the 9.0(x) ASA code and while I was excited at many of the new features–mostly around IPv6–and disappointed at others (No OSPFv3 address families? Really?) something immediately jumped out at me: Page 19 and its disturbing title, “ACL Migration in Version 9.0”
Any time you see the word “migration” in any documentation referring to an upgrade of production code or configuration, you know two things:
- It probably happens auto-magically, which is basically a synonym for “we’re going to bork your code but we’re only going to loosely tell you where, how, and why.”
- You’d better have good backups and be prepared, because a roll-back is likely going to be as painful as just plowing ahead.
To summarize the boring details for you, prior to the new code you had two categories of Access Control Lists (ACLs): those for IPv4 and those for IPv6. Inside each of those macro-levels you had the normal standard and extended lists and whatever other features. You applied the IPv4 and the IPv6 access-lists to the interface in whatever direction and that was that. True to the dual-stack model, you really were running two parallel networks and never the ‘twain shall meet.
During and after the upgrade to 9.0(x) ASA code, a couple of things happen:
- IPv6 standard ACLs are no longer supported, and any you have are migrated to extended ACLs.
- If IPv4 and IPv6 ACLs are applied in the same direction, on the same interface they are merged.
- The new keywords any4 and any6 are added in place of the old any keyword.
- Supposedly, if certain conditions are met (and they were in my case) your IPv4 and IPv6 ACLs should be merged into one (they were not).
While it is a bit scary to have any vendor automagically migrating portions of your configuration to a new format, it happens and as long as they document well and you do your due diligence, things can work out just fine. Other times they completely go to hell because of an undocumented feature. This upgrade fell somewhere in the middle.
As it turns out, a critical fact was left out of the documentation. Namely, that all of your access-groups that had been applied in some direction or another would now, quite frankly, not be applied to anything. In other words, my firewalls were now letting anything out of the network and nothing in. I quickly applied my new access-lists to the interfaces a couple of times before I discovered that you can now only have one applied in any direction (par for most IOS devices).
Since these were production and I had some higher risk on the IPv4 side (we have a lot of rules, and a default-block outbound policy) than the IPv6 side, I did the following:
- I blocked IPv6 in and out, then applied the IPv4 lists to the interfaces in the correct directions.
- I hand migrated (notepad is your friend) the IPv6 access rules into the IPv4 lists and brought IPv6 access back online.
- I then deleted the redundant (old) ACLs.
Everything came back, life was good, mostly nobody noticed anything. What’s the lessons learned from this experience? Besides don’t upgrade ASAs? How about these:
- Always have a backup of your configuration, preferably taken a few minutes before you start the upgrade. In this case I didn’t use the backups for more than a reference, but they were available if I had wanted to roll back.
- Know your configuration and your devices. This seems intuitive, but a lot of people would have gotten part way through this migration, saw that their ACLs were borked, and been lost. If you’re going to live on the edge, at least have a helmet.
- Read the documentation. I did, and while it didn’t directly help, I at least knew ahead of time what was likely to break. I also knew once it broke what the likely problem area was. To tie this into the CCIE Lab (back to studying, so it’s on my mind) it’s a bit like being able to look at a network diagram and instinctively know where you’ll have problems (two routers doing redistribution between EIGRP and OSPF, check).
At the end of the day, it all worked out for a variety of reasons listed above. Would I suggest any readers out there try this sort of “no net” upgrade to bleeding edge code? Probably not. In my case, I’m a masochist it seems, and this is my therapy. Now on to my 6500 upgrade to 15.1(1)SY. I’m sure I’ll be writing about that not long from today.
In a previous life, I worked as a pre-sales engineer specializing in network restructuring. As such, I was attached to the sales organization, and always participated in sales conferences, meetings, etc. I was paid a base salary, plus had a comp plan for bonuses–though I had more of an 80/20 split in salary/bonus potential, whereas the sales folks had a more traditional 30/70 or so split.
Every year, right after the hustle and bustle of the holidays had settled down, the same drill would begin again for the new year: the forecasting and associated compensation plan roll-out. Luckily enough, this was usually somewhere nice, like a posh resort in Scottsdale, so I didn’t really mind much. Of course, I also had a little less on the line than the true sales folks since most of my bonus split was based on the entire region in total, and not on a particular vertical market or small territory.
Inevitably, after the glow of the open bars, sunshine, expensive dinners, and golf outings had died down, reality would sink in. No one would make as much as the legendary (and usually fictitious) rep from last year who had landed the big whale account; we’d all be lucky just to make the mortgage payment. Some people would talk about leaving and others would just worry.
Then there were the experienced reps; the ones who had been around for a few years or more and understood the game. Sure, they may have a complaint or two about the forecasting numbers or something, but they largely weren’t flipping out like the new guys. If you asked them why, or what their secret was, the answer was one you’ve heard before: work the comp plan.
Let me explain. Ostensibly, the company sales leadership has crystalized the strategy for the next year into goals, targets, etc., based on what’s been deemed to be important or critical for success, down into specific targets for each territory and sales rep. They have decided how much to pay in bonus money, sales multipliers, etc., based on this. The stuff that was less important would be paid less lucratively.
Hence, the experienced sales people would mostly ignore whatever speeches, memos, or strategy discussions came from the mouths of the leadership team, and instead focus like a laser on how they were being paid: the comp plan. I get a 10% for selling vendor X’s product, but only 5% for vendor Y? Then Vendor Y doesn’t exist for all intents and purposes. Work the plan.
This usually had the predicted outcome, of course, with goals and compensation wildly swinging from one extreme to the other as targets were hit, missed, or wildly exceeded. Often the company would have to perform some sort of course-correction mid-year to overcome the errors made manifest in a system like this. Through it all, the veterans would smile and keep “working the plan” as the younger, newer folks burned themselves out trying to keep up.
Why do I bring this up now? As the head of IT at a company in the enterprise space, I’m not selling anything, at least not in the traditional sense. I still have customers, however, and I still have a bonus structure. In fact, we just finished up this process internally and so this was all on my mind.
You would expect that my goals–and those of my team–would be based on critical company goals, distilled and crystalized over many planning sessions, with a lot of thought behind them, right? You would think that, yes, but you would be wrong–or at least partially wrong.
Sometimes, in both the VAR world and the enterprise or ISP spaces, the gap between the high-level company goals and the individual departments’ goals get run through a blender of sorts (will it blend?). High level company goals of “maximizing shareholder value through increased paradigm-shifting cloud strategy” sounds great if you’re a recently minted MBA getting paid by the word, but it translates into exactly as much gibberish as you’d expect by the time it gets to be time to make it an actionable metric. So, you get random goals and projects as your key metrics instead of a more measured and appropriate metric.
For instance, as the guy in charge of IT for the entire company–world-wide–you might expect that a major goal of my team and I would be… uptime. It’s a fairly standard measurement of how reliable the overall architecture of the network is, and it’s relatively fair and has the benefit of being easy to quantify. Do I have uptime as any kind of metric this year? No. In fact, most of the goals I have are special projects that pull in stakeholders from other parts of the functional side of the business, but very few are actual, real, quantifiable IT metrics.
I asked about this and was told that that “other stuff” just goes without saying. When I asked what things during the year, and at the end of next year, would contribute to any possible bonuses, raises, and good reviews I was told that the goals are the metrics by which my team and I will be assessed. All of that redundancy stuff? Nope. All of that failover, uptime, increased efficiencies, lower staff hours, quicker help-desk resolution? Nope.
So why am I still smiling? Because I’ll be working the comp plan.
“There is always a well-known solution to every human problem–neat, plausible, and wrong.” H.L. Mencken, Prejudices: Second Series, 1920
I had an interesting conversation today with a cohort of mine regarding network complexity. Specifically, the conversation started out with him mentioning that he thought my network was “bloated with complexity”. He didn’t mean it in a bad way–necessarily–but my first instinct was to take it that way.
Then I started thinking about it. I wondered what, specifically, he might be referring to. Was it the dual-stacked IPv6? The VLANs? The VRFs and multiple redistribution points between protocols? The MP-BGP? Multicast?
It was probably some of these and all of these since he had just been pouring over some network diagrams that I had sent. He did hone in a bit on the IPv6 portion, but more specifically he seemed concerned about the redundancy. Yes, we have multiple routes out of offices, with multiple routers–no single point of failure and all of that. In my mind, however, that is a bonus and evidence of seven years of effort to rebuild this world-wide network into a very, very reliable and robust network.
“We like to keep things simple,” was his response. I pointed out that I could probably rebuild the entire core infrastructure in 12–24 hours, at the command line, from memory, so I didn’t think it was really that complex. He wasn’t necessarily persuaded.
Now don’t get me wrong, the entire conversation was very cordial, and I respect this guy’s bona fides and his judgement. He has a great track record, and a great set of skills in both management and systems administration. But therein lies the rub. His skills mean that what I see as simple solutions to a problem, he sees as a massive level of complexity to be reverse-engineered. Honestly, I’d probably react the same way to some of what he can do on the Active Directory side of the house.
My specialties are data center, bridging/routing, and converged infrastructure, and I’ve been doing that for 19 years professionally, and a hell of a lot longer if you count the years before I figured out someone would pay me for what I know. I briefly wondered if that has put me into an insular world where I can’t see what could be made more simple?
I spent most of the rest of the day thinking about the various parts of my network, and where I had implemented solutions to particular problems. I thought long and hard about what I could do to simplify that particular area of the network and couldn’t come up with anything.
I then turned my attention to whether or not I could add complexity just for the sake of doing it. In most cases the answer was a resounding yes. In fact, I compared what I had done in many places with some of the similar layouts in some of the CCIE labs (admittedly these are at an almost asinine level of complexity, but still…) and found that I had “taken the easy way out” as some of my past teachers were wont to say.
So, then I turned to the more philosophical question of knowledge and the relative nature of it. One of the things that seems to happen to me more often than I’d care to admit is this: I’ll find something in my network that needs fixing, and as I’m looking at the code and the diagrams I’ll wonder what moron did this thing. You’ll probably not be surprised to learn that it was a previous version of me who was the moron.
At the time I was working with what I knew, and I solved the problem using the tools I had, but looking back on it now I realize that I didn’t have all the experience and knowledge I needed to fix the problem correctly. I was approaching the problem based on what I knew, and I probably would have, at the time, viewed my new solution of today as “overly complex and bloated”.
The idea of “keeping it simple” is a good one, and one we should embrace as network engineers and architects as often as we can. If the simplest solution meets all of our requirements without breaking anything else, we should use it. What we should not do, however, is use the concept of keeping it simple to be a catch-all of keeping it so simple that you can hire a CCNA to run a CCIE-level network.
If what you want is a CCNA-level network, and that’s all your business needs, then you’ll be fine. As soon as you start incorporating higher-end systems like top tier ERP systems based on Oracle, redundant SANs in order to provide a certain level of resiliency and performance, 10 and 40-gig Ethernet, blade servers, virtualization, redundant paths and automatic failover, you’ve pretty much let that ship sail, however, and you need to acknowledge that you have a greater need for skilled maintainers of your network than maybe you’d like to believe.
Maybe “keeping it simple” is management code for “we can pay a monkey to maintain this, and maybe make our bonus because we made the IT costs look good this cycle.” In the short term that works. But in the long term there’s a cost to be paid, and if you miscalculate when the note comes due, you’ll likely find yourself in a much worse position when you have to fly a team of high-cost consultants in to clean up the mess.
I just received this from one of my Linked-in groups and thought I’d share it:
|Early Bird pricing discount for Orlando ends this month. Register now to save up to $300 http://bit.ly/2013Rg
Scheduler opens 26 March. Search for the courses you want here http://bit.ly/WHC06q or go to the course listing http://bit.ly/15KpVhT and then visit the Event Catalog for course details. NetVets can begin to schedule their courses on 19 March.
View our rebroadcasts of two very popular IPv6 sessions from Cisco Live and connect with the course speakers during the event on twitter. Find the session and chat details for the 21 and 28 March events here http://bit.ly/Chat13
We look forward to seeing you in Orlando in June http://bit.ly/CLpage and connecting with you in Cisco Live 365 http://bit.ly/VxZ5Um for thousands of on demand sessions all year, for free! New sessions have just been added from the Melbourne event.
“You don’t burn out from going too fast. You burn out from going too slow and getting bored.” — Cliff Burton
I used to wonder how a person can “burn out,” get bored, or otherwise grow tired of doing something they absolutely love. I really never understood the premise, and I didn’t understand the people to whom it happened. Until it happened to me.
To be fair, after having been diagnosed with a pretty rare form of cancer midway through last year, it’s not retrospectively surprising that my focus would change; that I would want to spend more time with family and less studying to the nth degree the various technologies I love. That wasn’t it, though.
I felt tired. An almost depressed kind of tired, to the point that I was just going through the motions but with no particular excitement about any of it. The books from Cisco Press and INE, the stacks of RFC printouts, and the hundreds of pages of white-papers on this or that technology sat in my office only occasionally moving when one of my oddly overweigh cats clumsily knocked something over in a vein attempt to gain higher ground.
In short, I was done; done with the arguments about Apple vs. Microsoft, or Android vs. iPhone, BSD and Linux, Emacs and Vi. I started spending my free time watching more television with my family thinking that meant quality time, when it mostly just meant passing time. I began a couple of hobbies, started planning more vacations to warmer places, and generally found myself just being.
Just recently, however, I had a revelation. It wasn’t that I was burned out on the whole of IT, I was burned out on two things: stuff that doesn’t matter, and maintenance. To wit:
The little things that I was burned out on were all of the little, petty, fairly unimportant things that everyone in IT gets hung up on at some point: the “this vs. that” arguments that inevitably devolve into quasi holy wars. To be honest, I couldn’t care less which phone you have or which operating system you use; if it works for you, great.
That’s a minor point, though. The big point of burnout for me was the maintenance–the daily grind of resetting passwords, looking at endless streams of alerts, scheduling maintenance windows, negotiating with management, budgeting, re-budgeting, etc. This is what I’ll call the cat-herding part of my world. This is the part that was burning me out.
I sat down one night and started reading one of the classic self-help books–which one escapes me this second, but probably Dale Carnegie or some such–and I started thinking about what it is that I actually enjoy about IT. What is it that drew me in when I was much, much younger and sustained me for so long?
The short answer? Projects. Figuring stuff out, implementing new technologies, making something work that solves a particular problem. That’s what gets the proverbial juices flowing. That’s what I needed to get back to.
Throughout most of my career I’ve been a bit of a designated hitter. What do I mean by that? I mean that I’ve done a lot of post-sales implementation, and pre-sales engineering, and a lot of evangelizing of different technologies. In short, I’ve been the guy who helps a company fix a particular problem.
For the last few years, however, I’ve gotten away from that. I’ve become a successful Sr. Network Engineer and IT Manager of a multi-national company and I’ve spent almost 7 years rebuilding an entire world wide network from the ground up. It’s been thrilling, challenging, and frustrating all at the same and different times.
Now, however, the network is almost all done. It is built. The problems have all been solved, and now we’re in a maintenance mode–where the focus shifts from expanding and problem solving to maintenance and cost-cutting. And therein lies the crux of the issue.
It may be that I’m some sort of masochist, or extreme type-A personality, but I have an almost narcissistic, obsessive need to be fixing something. With me, it is not a case of “if it’s not broke, don’t touch it,” it is more like “if it’s not broke, I don’t care about it.”
To wrap it up for now, it seems as if what I’ve been dealing with is not a case of burnout at all; it is boredom. And that, my friends, is much more insidious. I’m much more aware now of all that boredom can do: It drags you down, it robs you of the joy of moving forward, and it takes the fun out of what used to sustain you.
The good news, however, is that with this awareness comes a better, move revived focus, and I’m starting to come alive again. I’m starting to look forward to the conventions, to the new technologies, and to some of the inevitable arguments and technology holy wars. In the meantime, I’m off to reset some passwords and work on some budgets… at least until the next big thing comes along.