packetqueue.net

I know I haven’t exactly been filling the blog-space with useful tidbits, random ramblings, or musings on clown psychology lately. To be fair, however, I have been pretty heads-down in two different areas: one on my CCIE Routing and Switching lab studies; the other on a large Flexpod (UCS, NetApp, VMWare, Nexus) implementation project at the office.

We are a long time VMware shop, and when I came on board back in 2006 I made a push for even more utilization. We started expanding heavily and made the choice at the time to use Equalogic iSCSI SANs for our backend storage solution. This worked well for a number of years, but now that Dell has purchased Equalogic we have seen our support quality slipping, resolution times stretching, and parts deliveries slowing significantly. As such we moved to NetApp.

One of the things with NetApp that makes it an intriguing solution is the software, particularly around cloning of virtual machines as well as snapshots and de-duplication. While these are powerful technologies to be sure, sometimes getting your head around the details can be tricky. Even trickier? Mating up what you thought you knew, or what you’ve grown used to, with the reality now.

What am I talking about in particular? Defragmentation and Microsoft.

Everyone who uses a SAN and VMware probably knows by now that defragmentation, especially where VMDK files are concerned, is a giant waste of time. After all, you’re trying to move data blocks on a hard drive around to make them more efficient, but those blocks are really just representations of blocks inside of a file, using blocks, on multiple hard drives. Simple, right?

Well, with NetApp snapshots the reasons for not defragmenting get even more ammunition: you’ll actually use more space. Why? Because the snapshots are tracking change blocks (deltas) and so don’t take up any space at all when first created (or very little) since you’re just duplicating the root inode. Every time you defragment, you’re potentially rearranging all of the “blocks” of the file system, which is going to then trigger a bigger delta come the next snapshot. It’s the same reason why you sometimes delete space on a volume and don’t see it: the snapshot has to grow by the same amount as the change, and since the snapshots are on the same volume as the data… well, there you go.

So, all of this is great. Don’t run defrag and life is good, right? Sure. But Microsoft is trying ever harder to be helpful and they’re actually crossing more and more into that territory occupied by Apple that I don’t care for: the one where they obscure all of the details to “just make it work” and you have to hunt to find even the most basic of features.

Turns out that in Windows 2008 and 2008 R2, defragmentation is set up as a scheduled task by default. Every Wednesday as a matter of fact. The good news is that the task is disabled out of the box. The bad news? Most server admins have an almost pavlovian need to defragment Windows boxes and probably have turned this (or some variation) on for many machines in your environment. Possibly even through a GPO that someone set and forgot many cycles ago.

At the end of the day, defragmenting a virtualized Windows box might make the OS think its happy, and it might even make the server admins happy. Storage folks, however? The shrieking from the unfashionable wing of the IT area will probably be all the indication you need that bad things are afoot. Hell, if you’re really looking for some of the old ultra-violence, run defrag on all your machines at night… at the same time. If you’re lucky, and everyone’s asleep, you might even manage to offline a LUN. And that’s just good fun for everyone.

Incoming search terms for the article:

• snapdrive for unix script
• ccie blog
• nm-air-wlc6-k9 software forums
• windows 2008 r2 defrag isci lun

After 17 years in the industry I have decided that it is time to pass down some advice to the newer entrants in to the field of Information Technology.  If you’re just out of school with a freshly minted degree, looking for that shiny new job that leads to fame and fortune, then this article is for you.  If you are looking to move up in the job you currently hold, then this article is for you.  Hell, if you’re breathing and have heard of a computer before, this might be for you as well.

(1)  The first thing you have to realize about working in Information Technology is this: it automatically leads to six-figure incomes, sports cars, and attractive women.  Most people won’t tell you this, and in fact go to great lengths to cover it up—much like we in the Seattle area tell people it rains all the time so you won’t move here—but it is true.  Anyone who denies this is either lying to you, or incompetent.  If you apply for a position in IT (cool-kids abbreviation warning) you should definitely expect this as a minimum package.  If you don’t get offered all of these things up front, or hear anything faintly resembling an insinuation that you might need something called experience, run the other way: this job is beneath you.

(2)  This brings us to another good point that we should discuss right away: this notion of experience.  Experience is something that people who’ve sat around at their job long enough claim you need in order to do what they do.  The reality, however, is far different.  Most of these so called “experienced” people have long ago given up on being useful, and are simply waiting around to retire.  They’re slow, ineffectual, and don’t know half of what you do.  They’re your parents age, aren’t cool, don’t dress right, stay home on weekends, don’t come in with hangovers, and sit around so much it’s painfully obvious they don’t do anything.  Experience is just a word they toss out there to keep fresh young people who know more than they do from exposing their weaknesses to the sober light of day.  Scoff openly when presented with the need for experience.  Tossing in an “old” joke or two wouldn’t hurt either… it helps let people know you’re on to them.

(3)  Everyone knows that IT types in general, and network engineers in particular, are opinionated people.  All day, every day, we’re called upon to give voice to others’ technology insecurities; to make them feel better by telling them what is good and bad in any given situation.  To truly be successful—to truly rise above the crowds of mediocrity in the field—you’ll need to take this natural predilection for opining and crank it up a few notches.  The best way to do this is to form as many opinions on technology as possible, and then never waver from them.  It works even better if your opinions aren’t based on anything useful like quantifiable data or experience, but rather on ego.  You’ll also want to pick technologies to evangelize that either few people know, you don’t currently have in place (this helps tremendously, because you can be the “expert” without having to get your hands dirty by proving it), or that make you seem “cool”.  To wit, let’s look at point number 4:

(4)  Technologies like Apple are ubiquitous in the network engineering world.  They are good products in many ways, but that’s not why you’ll want to use them.  You’ll want to use them because that’s what all the “cool” kids are using.  Old people with “experience” use PCs and you don’t want to be associated with that.  Having some off-handed platitudes about why you use Apple computers is going to be good here; things like “I only use the best tool for the job” is a great one.  If challenged, or heaven forbid proven wrong, above all else don’t acknowledge this.  Simply wave your hands in a dismissive way and insist that somehow the thing you like about Apple really hasn’t been disproven, and move on.  This doesn’t apply only to Apple, of course, you can use this technique to make yourself look smarter than those around you with just about anything.  As I stated, however, you’ll really want to pick as many technologies as possible that don’t have much market penetration in your company or circle of influence.  If you pick something well-known to evangelize, you run the risk of being labeled as difficult to work with.

(5)  On the topic of difficult to work with, this can be important as well.  Agreeable people get nowhere in corporate America, and you certainly aren’t working for any team.  The best thing to do with any new job is to immediately establish that you won’t play by everyone else’s rules.  The most effective way I’ve found to make this happen is to constantly complain about how much better things were done “at my old company.”  This applies even if, as is likely, your old company was just college.  What this does is establish the fact that you’ve seen better, you know better, and you won’t be held back by mediocrity.  It also lets everyone know up front that you aren’t a team player, and that you’ll drive the bus of success all on your own thank you very much.

There are many more tips I can share, and I’m certain that this is probably just the start of a multi-part article.  It is, after all, incumbent on those of us who have been in the industry for a while now to try to pass on all that we’ve learned to the next generation.  There is far too much disinformation out there on so-called “success,” and I believe that whatever I can do to debunk the common mythology, it is for the best.

Incoming search terms for the article:

• NUOVO CISCO ANYCONNECT MOBILE - ASA 5505 L
• 2960s-48lps-l
• anyconnect-essentials
• cisco 2054 wlc
• Labgear DVR LAB download
• screenrc bindkey -k k9

Just a brief recap of Cisco Live 2010 in photos:

Cisco Live 2010

I just wanted to drop a quick note here to let everyone know that I plan on blogging a little more frequently this week, as I’ll be in Las Vegas for the annual North American Cisco conference: Cisco Networkers/Live. I can’t promise I’ll be in my room slaving over long, elaborate breakdowns of technology–maybe an in depth review of whiskey selections by bar–but I will try to post some pictures and information about what I’m seeing and hearing during the show. In years where I haven’t been able to attend the show, I always liked seeing and hearing from folks who did. Now it’s my turn to give back something, so watch this space…

Oh, and keep up with real-time information on twitter where I hide behind the handle @someclown, or G+ where I can be found at: http://gplus.to/someclown.

So, nostalgia about computers and the “good ole’ days” gets strong with me whenever I turn on my old Apple II, or pull out any old magazines from the early 80′s or 90′s. My wife laughs at me because I still get goosebumps and a light in my eye describing how I felt when I first saw the Apple IIgs or the Amiga 1000, and how bad I wanted them. It takes me back to when computers were fun, magical, and represented a brave new world to my young mind (with apologies to Aldous Huxley).

So it is with great excitement that I now have in my possession a book on that early time, written about one of the pioneering companies subject to more historical revisionism than most people realize.

“Commodore, a company on the edge” by Brian Bagnall is an in-depth, interesting, and more historically accurate portrayal of the early history of microcomputers in general, but Commodore in particular, than many I’ve read. Most early histories are written by revisionist authors like Robert Cringely and tend to dramatically overstate Apple and IBM’s contributions at the expense of Commodore and Atari, among others.

I’ll post a complete review when I’m done with the book, but just what I’ve read so far has me pining for simpler times. Before I knew acronyms like CCIE, OSPF, NX-OS and had a global enterprise network to tame, I had my Apple II, the Commodore 64, the Amiga 1000 and the Atari ST. They say you can’t go back again, but I’m trying.

Everyone has been posting their schedules for Cisco Live to Twitter, Facebook and wherever else, so I thought I’d better jump in with the cool kids and publish mine as well.  I can’t guarantee this won’t change, but for now it stands as my best guess and current planned schedule.

Incoming search terms for the article:

• cisco live 2012 san diego
• cisco live 2012 san diego dates

Editor’s Note: If you haven’t already, check out the first installment in this–hopefully not ongoing–series at http://blog.packetqueue.net/asa-tu/

At approximately 1:58pm PST last Thursday the two edge ASA 5510 units at our corporate headquarters dropped off the network.  At the time I was in a different office up in Quebec, Canada and so delegated to one of the other engineers to work the problem with TAC and bring them back online.  That process took much longer than expected, and I won’t bore you with the details.  What I will bore you with, however, are a few observations I have now that we have more time and experience working with Cisco’s ASA product line:

• The ASA has some sort of systemic, though exceedingly rare, problem on 8.3(x) and newer code.
• Said problem causes the units to reboot and take out the system flash (disk0:) but not user flash (disk1:).
• The flash appears to be erased, but it is in fact the MBR that is gone, not the data (we used a hardware forensic disk analysis unit to verify this).
• Cisco doesn’t have enough data points yet to even acknowledge this is an issue.  I don’t believe they’re “hiding” a problem; I just don’t think enough people have experienced the particular set of circumstances that would cause this and subsequently reported back to Cisco.

My own suspicions about the root cause are below, though I’d welcome any additional thoughts from anyone with experiences in this area.  I should also point out that I have heard from at least two other people that they have experienced this exact problem.

• The behavior and crash lead me to believe that the ASA experiences, at the point of failure, the equivalent of a Windows “BSOD”.  This would point to either memory or motherboard itself as these are the primary hardware-based causes of this type of crash in any system.  Most other crashes can be recovered from and produce data.
• The ASA accesses the flash on initial load, but then runs from memory.  The flash cards in these units had trashed MBRs which leads me to believe that the ASA was touching the MBR at the time of the crash, which is inconsistent with what I know about how the ASA is supposed to operate.  It’s possible it was just accessing the flash to write a crash-dump and crashed partway through.  That makes some sense to me.
• All failures I have experienced and heard of from others have at least a couple of things in common:  They are all on 8.3(x) code.  They are all post user-upgraded to support 8.3(x).  This code required a memory and flash upgrade, and so you had to buy upgrades from Cisco and field-install them yourself.  These units were also all manufactured immediately following the Cisco manufacturing slowdown in 2008/2009 when lead times were running into the several months range.  This makes me a bit suspicious that quality control on either the memory or the units themselves could be to blame.  I’ve tried to verify with revision numbers, etc., but I haven’t been able to gather enough data from “out there” to settle on this as a cause.

I hope this helps someone out there, and I truly am interested in getting more information from anyone that has it.  Cisco is taking our units back, but pulling them aside before refurbishment so that their engineers can dissect the units.  If I find anything out from that I’ll post the findings here.

The configuration and build-out of the ASA 5510 units is as follows:

• 1 Gigabyte of memory, 512MB of system flash, 256MB of user flash.  IPS Module, Security-Plus, Botnet filter, AnyConnect Essentials, Mobile, etc. licenses.  Actually, just about every license is on board; these units are at this point maxed on everything.  Utilization is at a reasonable level still.
• Configuration includes use of multiple IPsec site-to-site VPNs, SSL VPN for all Mac, Linux, Windows, iPad and iPhone, sub-interfaces, stateful failover, both IPv4 and IPv6, OSPF with static redistribution, and full IPS functionality.

Incoming search terms for the article:

• cisco crash dump failover

Passing the CCIE R&S Written (350-001)

I am proud to say that I have completed the first step on my journey to the CCIE Routing and Switching certification: namely, I passed the written qualification exam.  I obviously have a lot more work to do before attempting the lab later this year, but it is a good solid first step, and considering how long I’ve contemplated taking said step it is just good to be moving forward.

I’m not going to go into any details, talk about my score (it wasn’t perfect by any means) or really discuss anything that even smells like an NDA violation.  If that’s why your here and how you found this short blog posting, you’re in the wrong place.  I’ve worked far too hard for this to diminish either the work I’ve put in to get here, or the work that so many other full CCIEs have put in to attain their certifications.  The only way you get the digits is to pay your dues like everybody else.

That said, my brief observation for what it’s worth, is that this test was not entirely what I was expecting.  After years of taking different certification tests, including a variety of other offerings from Cisco, this test seemed a bit, well, tame.  Not easy, just more straight-forward question and answer.  That wasn’t really a positive or negative in my mind since I don’t really consider myself a “test” person and would have preferred a few more hands-on scenarios than I got.  But I suppose I’ll get more than my fill come lab-day.

The other interesting thing I noticed was the questions.  Some were almost cloyingly easy, while others a bit harder than I would have thought.  Possibly that is just a side effect of my studying habits.  In other words, the questions I found easy might be the same ones that trip someone else up.  When you’ve been at the books long enough, you lose a little perspective on these things.  None of the questions, however, were surprising in any way.  I think that the subject matter described on the blueprint, as well as some base-level networking knowledge that is just assumed was all covered in a way that you should expect of this level of testing.

The last thing I found different than some of the other tests I’ve taken is the increased reliance on “stacking” technologies.  In other words, you could see a question ostensibly focused on a particular technology, but with one or two other technologies represented in the question as well.  In particular, you would be required to understand not only all three technologies in the question, but also the subtle interactions that can happen as they work together.  My sense is that this is probably intended to be more “real world” representative, and in general I think it worked well.

All in all I think it was like a lot of Cisco tests: fair but difficult.  If you know what you’re doing you should pass, and if you don’t, well… take your score breakdown and hit the areas where you were weak.  Oh, and Cisco: please make your example diagrams easier to read!  I’m not so old that I need reading glasses, but my god some of those diagrams were bordering on illegible.  On at least a couple of occasions I had to squint, look sideways, and try to see… like one of those damned “dot” pictures where if you stare long enough you see a dolphin or some other randomly insipid thing you feel cheated for having expended the effort to see.

And now?  Off into some hundreds of hours of rack time.  Doh!

Random (and not so deep) Thoughts by Some Clown

I haven’t been writing a lot lately, mostly due to a combination of my work and study schedule.  I thought, however, that it would be useful to just toss down a few random thoughts on the proverbial paper to wrap up 2010.  I’ll try to keep it somewhat cohesive, but I can’t really guarantee anything.

Studying

Having made the decision last year at Cisco Live to finally buckle down and pursue the CCIE Routing and Switching certification, I have been as busy as you might imagine with studying.  As I’ve gone down this road I’ve noticed a couple of things:

(1) In the office I’m used to studying large white papers, documents, manuals, command references, etc., quickly to get to the answers I need for either deployment or break-fix.  This is not the best way to study for the CCIE qualification exam, however, as I tend to just as quickly forget that information past the point of it being immediately useful.  I’ve had to change my habits now to include taking notes, reviewing portions over and over, and cross-referencing with multiple sources.  Nothing earth shattering to be sure, but a change for me.

(2) As alluded to above, I do a lot of cross-referencing on my study material.  I have material from CCBootCamp that I consider to be my primary source (by virtue of being enrolled in the Cisco 360 program through them).  I have also been reading the CCIE Routing and Switching Certification Guide, 4th Edition, as well as the CCIE Routing and Switching Exam Quick Reference Sheets–both by Cisco Press.  I think it helps me quite a bit to read different perspectives on the same material; to see it put a different way on the page.  I have a Cisco Live Virtual account as well, and so have been pulling some presentations–notably on QoS–from that site.

(3) I have over 16 years of professional experience in this industry, and while I am by no means an expert, I am confident in things that I know.  To that end I would say that at some point in your studies you will be almost guaranteed to come across information, answers to practice questions, etc., that you just know are wrong.  I’ve had to learn not to be afraid to challenge my study material.  I don’t do it blindly, but I do go out and research in other sources to verify what I think I know.  I have found many instances of incorrect information in several sources–more often than not in the Cisco IOS example configurations.  Sometimes using commands that won’t work on that platform, other times referencing non-existent class-maps or access-control-lists.  Less often have I found blatantly incorrect explanations of how a thing works, but even there I have found a couple of examples.  I take this as a good sign, actually; it’s a sign that I am becoming more aware of the details of what I am studying.

Interesting Design Decisions

It always fascinates and bewilders me to see some of the design decisions that other engineers make when putting together a network.  Much of what we do is subjective, and even the most experienced experts disagree on a good many things.  With that said, certain things just don’t strike me as particularly useful and it’s my prerogative to complain about them.  My top complaints from recent experience, in no particular order are:

(1) My predecessor who built our main datacenter using 4503 switches exclusively: access, distribution and core (mostly, but we do use a collapsed core model).  The 4500 series is great but my general argument is that they’re under-powered, or at least under-featured for the core (Sup II-plus) and just a bit overpowered for the access layer.  We use PoE 1-Gig to every port in the building, but the access layer is still barely running (less than 1 percent utilization ever, on any metric).  I think someone got a deal or something.  We’re now replacing the core with a pair of 6506, 720 supervisor, 10-gig, etc.

(2) A main distribution point had a single 3845 with a 100-meg Internet connection, and two full DS3 links.  Considering the 3845 maxes out at 45 Meg of throughput, this seems a particularly egregious violation in my mind.  We’ve now moved that to a 3945, which if under full load is probably still a tad over-subscribed, but much better and the price was right.

(3) Who was it at Cisco that decided that the ASA-5510 would only have two Gig links available, and only with the right license?  Why only two?  Why not three or all five?  This might be a backplane issue, I don’t know, but it just bothers me.

(4) My own stupidity in setting up the aforementioned ASA-5510 pair (failover) with the inside and outside interfaces on the gig links, when I should have had the two trunk links that handle much more traffic on those interfaces.  This will be changed soon, but I should have done it right the first time.

In Conclusion

2010 has been a good year overall, with a lot of interesting projects, experiences, and solid learning had by all–or at least me.  I’m looking forward to 2011 and all of the continued successes and experiences to come.  I’d also like to give a special shout-out to all of my Twitter colleagues, friends, followers, and various clingers-on and lurkers.  I have found the Twitter community to be an invaluable source of support, wisdom, and occasionally respite from the rigors of the daily grind.  If you’re not on Twitter, I’d highly encourage you to give it a look.

Happy New Year everyone!

Well, it’s over.

Vacations always seem way too short.  All of the months of planning, dreaming, working out the little myriad details: then, 10 days in Cozumel and Southern California and it’s done.

C’est la vie.

That does, however, mean that it is right about time for me to finally post something useful here.  To that end, look for my new posting on rebuilding access to your ESX Host Server after you bungle a Cisco Nexus 1000v migration.  Not that I would know anything about that.  *Ahem*

Sometimes the best lessons are the ones we teach ourselves.  Often inadvertent and unpleasant, yes, but the most long lasting.

Incoming search terms for the article:

• a good vacation is over now its time for work