I have been leading an IPv6 roll-out for some time now, among other things, and have found some interesting and widely differing levels of support for the next generation Internet Protocol.  With some vendors, many things work and work well, while with others it’s as if things haven’t changed in a decade or more.  Even with the vendors who do have relatively good support for IPv6, however, there remains often odd, even inexplicable, gaps in that support.  This has made our deployment a lot more challenging than it needed to be.

Much has been made of the chicken-and-egg nature of the problem: does demand drive the support, or does having support create the demand?  Self-named analysts, vendor representatives, media pundits, and even my dog seem to have an opinion on this, but I’ve heard little from the people in the trenches actually trying to implement this stuff.  Implement as in across the board full feature parity, not half-assed or “it worked in the lab” analyses.

Further exasperation comes as you figure out that you don’t know what you don’t know, and get 65% into the project before you figure out that some feature is missing.  A feature like, say, HSRP.  Whenever I complain that HSRP support is missing (or other FHRP) someone inevitably suggests RA tuning as a solution to the problem, which is a bit like handing someone asking for a Hamburger a popsicle; nice, but not the same.  Just how fast do you think you can achieve failover with RA tuning anyhow?  And don’t even get me started on what happens in a dual stacked failover scenario where RA tuning is handling IPv6 and some FHRP is handling IPv4.  At least BFDv3 is available for route failovers.

It’s not even that big, significant, oh-my-god features are always the ones missing, however.  Often times it’s the random, little features.  Cisco’s ASA, for instance, can’t do stateful failover using anything but an IPv4 address.  Why?  They’ve implemented IPv6 ACLs, objects, NAT (god help us all) and a lot of the bulk gotta-have-it features.  Why not failover?  Oh, and OSPFv3 support is missing too.   Why?  Dunno.

Our UCS is no exception to this rule, as almost nothing is IPv6 ready that I can find.  Ditto for the VMware installation we run on it.  Never mind that we’re at the newest patch levels, running VSphere 5, ESXi, etc.  View?  Nope, no support there either.  Our NetApp array on the back-end?  The big beast with multiple glorious 10-Gig connections?  Bubkiss for the IPv6 support there as well.  Although they do have a nice bit of marketing available online here.  See if you can tell when they’ll have IPv6 support from that document.

In all fairness here, I should point out that the Virtual Machines that you run in VMware, on the UCS do support IPv6 just fine, or at least as fine as the individual OS you’re installing (see previous rant on Windows 7 here).  SLES (Suse’s flagship server product) supports IPv6 from the command line, for instance, but not from within YaST.  Not a big deal if you’ve used Linux or any flavor of Unix for a while, but for a junior engineer?  That can mean more escalations and a more inefficient NOC.

In a lot of ways, actually, the Operating System purveyors seem to be way ahead of most infrastructure (network, storage, security) providers in supporting IPv6–even with their flaws–but that may be simply do to the amount of features they have to port vs. what a Cisco, Juniper or HP has to support.  The notable exception here being Apple, which for some inexcusable reason just dropped IPv6 support from their Airport Wireless product.

At the end of the day, I understand that rewriting absolutely everything to support an entirely new protocol is incredibly difficult.  I also understand that IPv6 has some behaviours that mean feature parity is not always going to be at 100% because it just doesn’t make sense.  I even understand that features will be rolled out in some sort of priority-ranked order, and that maybe management interfaces aren’t at the top of that list.  But what I don’t understand, or can’t get my head around, is why so many glaring inconsistencies exist when we’ve had so long to work at it.  Or why some vendors give little more than lipservice to IPv6 while not supporting any of it in their products.

Incoming search terms for the article:

• best blogs on ipv6 switching
• ipv6
• ipv6 feature parity with ipv4
• just say no to ipv6
• vista ipv6 default gateway

Incoming search terms for the article:

• ccie r&s lab diagram
• frame relay practise example
• ine ccie rack
• symbol of frame relay
• ltrccie cisco live 2011
• telco switch symbol
• cisco asa symbol
• nexus switch symbol cisco
• my cisco live 2012 schedule
• ipv6 cef vss svi

Earlier this week, I reviewed the appalling (lack of) progress so far and began moving forward.  I had already completed the acquisition of Provider Independent Address Space (PI), getting our upstream providers to advertise the routes, and the setting up of 90% of the infrastructure (SVI addressing, the different IPv6 settings like IPv6 CEF and unicast-routing, etc.)  Everything here worked as expected.

I decided to start with the infrastructure more related to the client side, including servers of interest (DNS, AD, mail, etc.) to the inside, and the end-user workstations.  It should be noted here that we had IPv6 turned off on all workstations and servers, despite Microsoft’s best practices which state that IPv6 should be left on because several services will break with it off.  Since these services (Homegroups, for one) don’t really apply in a corporate environment, I didn’t care too much what their best practices recommended.   The first step was to turn IPv6 on for a few test machines in the IT VLAN, and a couple of the Active Directory servers.

The good news is that the servers came up and were happy as soon as they got their addresses.  We’re going with a scheme of static assignment for servers, infrastructure management addresses, etc., so all I really had to do was click the checkbox for IPv6 (these are Windows 2008 R2 servers), assign address and gateway, and voila, we had reachability.  Next, I fixed up some DNS settings, added some AAAA records (the IPv6 equivalent of IPv4 A records), and we had DNS resolution on IPv6 working as well.

Windows 7 wouldn’t prove to be quite as easy, for a few reasons.  While the checkboxes and settings are ostensibly the same, some of the back-end code is obviously different and needs some non-obvious tweaking to get right.  I couldn’t for the life of me figure out why I had no reachability, when the configuration should actually be easy as pie (stateless auto configuration–just “turn on” IPv6 and walk away).  In fact, I ended up testing my Macbook Pro and a Linux machine, and both of them worked as expected right out of the gate, so I knew this was a Windows 7 problem.

The first thing that occurred to me after testing the other non-Windows machines was domain policies.  We have a lot of them, and you never know what might be causing issues.  I had one of my Server guys check that, and then check a non-domain Windows 7 machine with the same image (Windows 7 Enterprise), and he came up with no applicable policy problems and the same behavior.

Finally, after spending too much time with Google, I managed to piece together an idea of what was happening.  As it turns out, the mechanism by which Windows 7 puts together its interface identifiers is to blame.  I found the applicable information in a posting from February, 2010 on a site called itexpertvoice.com, linked to here.  It probably won’t surprise anyone to find out that Microsoft screwed up implementing something they helped create (RFC 4941).  The command you’ll want to run, from an elevated command prompt (or GPO, SCCM, etc.) is:

netsh interface ipv6 set global randomizeidentifiers=disabled

This takes you back to behavior supported by standards-compliant hardware, and allows your Windows 7 machine to actually use IPv6.

Another command you may want to use, and this is strictly a personal and policy choice, is:

netsh interface ipv6 set privacy disabled

This turns off privacy addressing, an idea also described in RFC 4941.  Given the globally unique, routable nature of IPv6, I can see where some people might get squirrelly, but I just don’t see the need.  Again, this is something you’ll want to review for your own environment.

Another interesting problem I found, and one more I had to solve before my Windows 7 machines were ready to be released back into the wild, revolves around voice.  Specifically, voice VLANs.  In a voice-enabled network using Cisco IP telephony, each access port on a switch typically belongs to two VLANs: a voice VLAN and a data VLAN.  You plug a phone in, your computer plugs in to a pass-through port on the phone, and each auto-magically gets an IP address in the correct VLAN.  Except with IPv6.

I should say, however, it didn’t work that way in my environment.  What happened here is that my workstation would actually get an IP address in both the data and the voice VLANs.  And, due to the way prefix policy works (more on that in a minute), would use the voice VLAN as the source address in traffic originating from the workstation.  The way I solved the problem for now is to turn off IPv6 and take away the IPv6 addressing from all of the voice VLANs (specifically, on the SVIs), since I haven’t gotten to the voice part of the IPv6 project anyway.

I don’t know what’s causing the workstation to get assigned a voice address, but I’m going to continue to look into it.  More than likely it has to do with Stateless Auto configuration and ND, but time will tell.  In the mean time, what’s this prefix policy thing I mentioned above?  Read about it in RFC 3484.

Put simply, IPv6 prefix policy exists because any given interface in IPv6 is going to have several addresses; everything from Unicast to link-local, multicast, and maybe more.  So, if I’m a workstation and I want to source traffic outbound on a given interface, which address do I use?  Prefix policy helps to solve that problem by “ranking” different addresses on an interface in a standards-specific way.

After reading the RFC, open up an elevated command prompt on your Windows 7 machine and enter the following command to see how your machine is set up:

netsh int ipv6 show prefix policies

and you’ll get something that looks like so:

You can manipulate the order of things here, and add if needed, using the following commands (note these are two commands, but word-wrap is in effect here):

netsh int ipv6 add prefixpolicy=d2a:d00c:b678:ecfb::1/128

precedence 2 label=22

netsh int ipv6 add prefixpolicy=d2a:d00c::b678::/48

precedence 2 label=22

This gives you a matching prefix pair, and says that for all traffic destined for d2a:d00c:b678::/48 network, use the d2a:d00c:b678:ecfb::1/128 address on the interface as the source.  In my case, my voice VLAN (applicable hextet: 107) was showing up in the interface prefix list before my data VLAN (applicable hextet: 1ff).  I could have used the example above to set the correct precedence, but this isn’t exactly scalable to thousands of phones and workstations, all of which may have a different mix of addresses.

So, we still have the voice VLAN issue to solve, but the Windows 7 machines can now talk comfortably on the network to other workstations and servers, including using SSH, DNS, RDP, and all other major services.  A lot of software still isn’t IPv6 compatible, but that’s why we’re in a dual-stack world for the foreseeable future.

Next challenge, if anyone has suggestions?  Working around some of the feature non-parity issues mostly in the area of security.  For instance, the ASA line as of last check still doesn’t support OSPFv3 or failover via IPv6, and no tools exist that I’m aware of for mapping complex security policy from IPv4 to IPv6, which is why our IPv6 world here still remains sequestered to just our network.

Ideally we’ll have outside traffic allowed and all of our public-facing services running in time for World IPv6 day this year, which happens on June 6th.  Mark your calendars!

Incoming search terms for the article:

• ie ccie
• cisco asa boot tasman drive
• CUCM-PAK
• cisco switch diagram
• clock cisco
• ccie rack diagram
• Cisco Catalyst 2960S-48TD-L stacked
• ine ccie lab
• cisco networkers 2960-s
• cisco tasman drive

We are a long time VMware shop, and when I came on board back in 2006 I made a push for even more utilization. We started expanding heavily and made the choice at the time to use Equalogic iSCSI SANs for our backend storage solution. This worked well for a number of years, but now that Dell has purchased Equalogic we have seen our support quality slipping, resolution times stretching, and parts deliveries slowing significantly. As such we moved to NetApp.

One of the things with NetApp that makes it an intriguing solution is the software, particularly around cloning of virtual machines as well as snapshots and de-duplication. While these are powerful technologies to be sure, sometimes getting your head around the details can be tricky. Even trickier? Mating up what you thought you knew, or what you’ve grown used to, with the reality now.

What am I talking about in particular? Defragmentation and Microsoft.

Everyone who uses a SAN and VMware probably knows by now that defragmentation, especially where VMDK files are concerned, is a giant waste of time. After all, you’re trying to move data blocks on a hard drive around to make them more efficient, but those blocks are really just representations of blocks inside of a file, using blocks, on multiple hard drives. Simple, right?

Well, with NetApp snapshots the reasons for not defragmenting get even more ammunition: you’ll actually use more space. Why? Because the snapshots are tracking change blocks (deltas) and so don’t take up any space at all when first created (or very little) since you’re just duplicating the root inode. Every time you defragment, you’re potentially rearranging all of the “blocks” of the file system, which is going to then trigger a bigger delta come the next snapshot. It’s the same reason why you sometimes delete space on a volume and don’t see it: the snapshot has to grow by the same amount as the change, and since the snapshots are on the same volume as the data… well, there you go.

So, all of this is great. Don’t run defrag and life is good, right? Sure. But Microsoft is trying ever harder to be helpful and they’re actually crossing more and more into that territory occupied by Apple that I don’t care for: the one where they obscure all of the details to “just make it work” and you have to hunt to find even the most basic of features.

Turns out that in Windows 2008 and 2008 R2, defragmentation is set up as a scheduled task by default. Every Wednesday as a matter of fact. The good news is that the task is disabled out of the box. The bad news? Most server admins have an almost pavlovian need to defragment Windows boxes and probably have turned this (or some variation) on for many machines in your environment. Possibly even through a GPO that someone set and forgot many cycles ago.

At the end of the day, defragmenting a virtualized Windows box might make the OS think its happy, and it might even make the server admins happy. Storage folks, however? The shrieking from the unfashionable wing of the IT area will probably be all the indication you need that bad things are afoot. Hell, if you’re really looking for some of the old ultra-violence, run defrag on all your machines at night… at the same time. If you’re lucky, and everyone’s asleep, you might even manage to offline a LUN. And that’s just good fun for everyone.

Incoming search terms for the article:

• cisco asa clone
• screenrc post examples
• screenrc mac cisco screen size
• schedule for kids dues
• nm-air-wlc6-k9 software forums
• netapp vmware 2008 r2 iscsi
• Netapp Engineer blog
• netapp defragmentation
• monitor size of cisco queue
• ucs netapp vmware san switch crash

(1)  The first thing you have to realize about working in Information Technology is this: it automatically leads to six-figure incomes, sports cars, and attractive women.  Most people won’t tell you this, and in fact go to great lengths to cover it up—much like we in the Seattle area tell people it rains all the time so you won’t move here—but it is true.  Anyone who denies this is either lying to you, or incompetent.  If you apply for a position in IT (cool-kids abbreviation warning) you should definitely expect this as a minimum package.  If you don’t get offered all of these things up front, or hear anything faintly resembling an insinuation that you might need something called experience, run the other way: this job is beneath you.

(2)  This brings us to another good point that we should discuss right away: this notion of experience.  Experience is something that people who’ve sat around at their job long enough claim you need in order to do what they do.  The reality, however, is far different.  Most of these so called “experienced” people have long ago given up on being useful, and are simply waiting around to retire.  They’re slow, ineffectual, and don’t know half of what you do.  They’re your parents age, aren’t cool, don’t dress right, stay home on weekends, don’t come in with hangovers, and sit around so much it’s painfully obvious they don’t do anything.  Experience is just a word they toss out there to keep fresh young people who know more than they do from exposing their weaknesses to the sober light of day.  Scoff openly when presented with the need for experience.  Tossing in an “old” joke or two wouldn’t hurt either… it helps let people know you’re on to them.

(3)  Everyone knows that IT types in general, and network engineers in particular, are opinionated people.  All day, every day, we’re called upon to give voice to others’ technology insecurities; to make them feel better by telling them what is good and bad in any given situation.  To truly be successful—to truly rise above the crowds of mediocrity in the field—you’ll need to take this natural predilection for opining and crank it up a few notches.  The best way to do this is to form as many opinions on technology as possible, and then never waver from them.  It works even better if your opinions aren’t based on anything useful like quantifiable data or experience, but rather on ego.  You’ll also want to pick technologies to evangelize that either few people know, you don’t currently have in place (this helps tremendously, because you can be the “expert” without having to get your hands dirty by proving it), or that make you seem “cool”.  To wit, let’s look at point number 4:

(4)  Technologies like Apple are ubiquitous in the network engineering world.  They are good products in many ways, but that’s not why you’ll want to use them.  You’ll want to use them because that’s what all the “cool” kids are using.  Old people with “experience” use PCs and you don’t want to be associated with that.  Having some off-handed platitudes about why you use Apple computers is going to be good here; things like “I only use the best tool for the job” is a great one.  If challenged, or heaven forbid proven wrong, above all else don’t acknowledge this.  Simply wave your hands in a dismissive way and insist that somehow the thing you like about Apple really hasn’t been disproven, and move on.  This doesn’t apply only to Apple, of course, you can use this technique to make yourself look smarter than those around you with just about anything.  As I stated, however, you’ll really want to pick as many technologies as possible that don’t have much market penetration in your company or circle of influence.  If you pick something well-known to evangelize, you run the risk of being labeled as difficult to work with.

(5)  On the topic of difficult to work with, this can be important as well.  Agreeable people get nowhere in corporate America, and you certainly aren’t working for any team.  The best thing to do with any new job is to immediately establish that you won’t play by everyone else’s rules.  The most effective way I’ve found to make this happen is to constantly complain about how much better things were done “at my old company.”  This applies even if, as is likely, your old company was just college.  What this does is establish the fact that you’ve seen better, you know better, and you won’t be held back by mediocrity.  It also lets everyone know up front that you aren’t a team player, and that you’ll drive the bus of success all on your own thank you very much.

There are many more tips I can share, and I’m certain that this is probably just the start of a multi-part article.  It is, after all, incumbent on those of us who have been in the industry for a while now to try to pass on all that we’ve learned to the next generation.  There is far too much disinformation out there on so-called “success,” and I believe that whatever I can do to debunk the common mythology, it is for the best.

I won’t bother you with my initial reactions to the Mac as they’ve been mostly positive and probably not entirely different than many other people’s experiences.  What I will say, though, is that as a long time user of various flavors of Unix, it’s nice to be operating on a platform that is Unix based again.  Specifically, it’s nice to not have to download bolt-on software in order to be able to use SSH, Finger, TFTP, etc.

One of the tools that I have used extensively in the past is GNU Screen.  For those unfamiliar with the program, you can check out a brief overview over at Wikipedia: http://en.wikipedia.org/wiki/GNU_Screen, or at the GNU Screen website here: http://www.gnu.org/software/screen/.

Despite using Screen since I was first introduced to it back around 1990 or so, if I recall correctly, I’ve only ever used it really for one thing: connecting from a Unix Machine to something using the serial port or a modem.  Even the modem bit I didn’t use all the time as I tended to favor Minicom.  So, when a couple of friends suggested that I use Screen for solving a problem I was having, it made sense.

As part of my Cisco lab I have some fourteen different devices that I need to access at any one time.  I have all of those devices set up to my main terminal server (3945 which is also my main a-lot-of-other-things as well) using reverse telnet.  In this way I can do a lot of things, but typically I would SSH to the terminal server, connect to a host by name, then use Ctrl-Shift-6 x to get back to the terminal server and do the same thing again for another host.

This is all made possible using an Async card:

NAME: "High Speed Wan Interface card with 16 RS232 async ports(HWIC-16A) on Slot 0 SubSlot 3", DESCR: "High Speed Wan Interface card with 16 RS232 async ports(HWIC-16A)"

With a pertinent configuration like so: 

interface Loopback0

ip address 172.16.0.1 255.255.255.0

ip host s2 2054 172.16.0.1

ip host s1 2053 172.16.0.1

ip host s4 2056 172.16.0.1

ip host s3 2055 172.16.0.1

ip host r5 2063 172.16.0.1

ip host r2 2060 172.16.0.1

ip host r6 2064 172.16.0.1

ip host r4 2062 172.16.0.1

ip host r3 2061 172.16.0.1

ip host r1 2059 172.16.0.1

ip host r8 2066 172.16.0.1

ip host r7 2065 172.16.0.1

ip host ASA 2051 172.16.0.1

ip host AP1 2052 172.16.0.1

ip host bb2 2057 172.16.0.1

ip host bb1 2058 172.16.0.1

line 0/3/0 0/3/15

session-timeout 120

no exec

transport input telnet

Still, opening fourteen hosts can get tedious, as you might imagine.

What this configuration also allows for, however, that I had only been using occasionally is logging in to any device by opening a telnet connection to the port.  So, I can either telnet to the parent device’s IP address (in this case it’s 10.7.68.221) or to the loopback address (if I have routing) and adding on the port number like so:

telnet 172.16.0.1 2056

which, as we see from above, would take me to the device labeled “s4”.   That’s all good, but still requires me to open each telnet session by hand, right?  Not necessarily: enter Screen and the magic of configuration files.

I have two configuration files for Screen: one is a .screenrc file which sits in my home directory as is standard on Unix systems.  Note that I could also put this in my /etc directory and change the name if I wanted a system-wide default for all users.  The other file is a custom one that I only use when I’m doing lab-work, and I call that one screen.ccie.

If I want to open just Screen by itself, from a terminal (I use iTerm2) I just type “screen –A” which opens up Screen with all possible emulations enabled (just in case).  If I want to open my ccie lab configuration I type “Screen –Ac screen.ccie” to reference the startup file.  I also use “screen –A /dev/tty.KeySpansomething” to use my USB/Serial adapter if I’m at the console somewhere.

Below you’ll find the screen.ccie file, with comments, as well as a screen shot of what it looks like in action.  I didn’t include the .screenrc file because it is identical, except for the last section that opens up all of the different telnet sessions.  I’m sure there are a lot of ways to do this, some better than others, and infinite combinations of colors, stats, etc. that can be configured, but this is what I have for now.

# SomeClown's .screenrc file.  Cribbed from a lot of places.
# Credit to both @IPv6Freely and @jay25f

# Next and Previous tabs shortcuts
bindkey "^e" next       # Next
bindkey "^w" prev       # Previous

# Various options
escape ^Zz                      # Control-A is needed elsewhere
nethack on                      # Just 'cuz
maptimeout 5            # Fixes Vi
vbell off                   # Pavlov
startup_message off     # Meh
defscrollback 30000     # Lots o' logging room
autodetach on           # Go away, come back.
shelltitle ""

# Status at bottom of screen; tabs, etc.  The code's a FusterCluck, but looks nice
hardstatus alwayslastline "%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%< %=%D %M %d %c"
hardstatus string '%{= kK}%-Lw%{= KW}%50>%n%f %t%{= kK}%+Lw%< %{=kG}%-= %d%M %c:%s%{-}'

# Specific for CCIE Lab.  Standard .screenrc file doesn't include text below

# Create named tabs for each device
screen -t Stuff 0                       # General
screen -t R1    1
screen -t R2    2
screen -t R3    3
screen -t R4    4
screen -t R5    5
screen -t R6    6
screen -t R7    7
screen -t R8    8
screen -t S1    9
screen -t S2    10
screen -t S3    11
screen -t S4    12

# For each tab above, do stuff
at R1 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2059 \015"
at R2 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2060 \015"
at R3 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2061 \015"
at R4 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2062 \015"
at R5 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2063 \015"
at R6 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2064 \015"
at R7 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2065 \015"
at R8 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2066 \015"
at S1 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2053 \015"
at S2 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2054 \015"
at S3 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2055 \015"
at S4 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2056 \015"

Incoming search terms for the article:

• reverse telnet
• screenrc mac
• screenrc
• bonjour wlc module
• backburner telnet
• telnet from mac book
• screen -t r1 1
• reverse telnet from mac
• screenrc ip
• screenrc for mac

Incoming search terms for the article:

• cisco live 2012
• potato to clock hook up diagram
• potato clock diagram
• cisco live crashed
• 6506 e firewall
• Cisco live 2011 - CAE
• Cisco live
• cisco lab with esx
• cisco 2960 best practices iscsi vmware
• symbol frame relay

Without going into the details of the entire Cisco Live experience, I’d just like to talk about the class I took on the first Sunday of the show–or the day before the show officially starts, depending on who you talk to.

On Sunday I attended a full-day mock CCIE R&S lab (Session LTRCCIE-3001). This was an instructor-led affair, with Bruce Pinsky (Distinguished Engineer) and Bruno van de Werve (Product Manager) acting as facilitators and proctors. Considering Bruno’s experience as both a proctor for the actual R&S lab, and now the head of the R&S program, this was an experience well-worth having if only for the ability to ask questions.

Unfortunately for all of us, and through no fault of either Bruce or Bruno, the in-class network was crashed from the moment we all got there. There were a number of failures, including some bad cables (how do you miss that in testing) which resulted in all of us essentially sitting around for over an hour.

To make up for the delay in getting started, someone from Cisco came in and apologized and handed out gift cards to Mandalay Bay. It was a nice gesture, but considering the gift cards had a face value of five dollars, it might have been better to not hand out anything. It had the affect of actually irritating several students, and giving the rest of us something to joke about for a while. The class cost $1000 (or 10 Cisco Learning Credits) so the value of even an hour should have been closer to $125 or so.

After that snafu, and a brief presentation by Bruno and Bruce on numbers of CCIE in the world, with breakdowns by region, we got started with the meat of the class: the labs themselves. We were all looking forward to this, since it was being run by Cisco and had the smell of real-world vs. some of the third-party labs (note that I use third party labs for training, and have no problems with them, but this was officially sanctioned and so had a little something extra, at least in “feel.”)

The troubleshooting section came first, and used the same system as the real lab so that was a nice touch. In our case we had only five trouble tickets to complete in one hour vs. the real lab which has ten in two hours. I believe this was done to facilitate the “instructor led” nature of the class, and allow us to ask plenty of questions. Bruce and Bruno were stellar in this regard, coming around to any student with a question and helping them to understand the problem or just passing out hints to those who still wanted to figure it out on their own.

I learned a lot about myself and my troubleshooting techniques during this portion of the day, as I got bogged down on the first ticket and blew the rest of my time. It was a relatively straightforward ticket where a particular address wasn’t answering an ICMP Echo to another device. It was a few routers together, with BGP. I spent the entire hour re-architecting the BGP–down to bare metal and rebuilding the config from scratch–and almost was done when time expired. As it turned out, it was a simple address statement that was missing.

Bruno got a chuckle out of this and pointed out that the lab is not intended as a “best practices” lab. He said that in most cases you won’t be removing configuration at all during the TS section; you’ll simply be adding something missing or correcting route statements, etc. It was helpful for me to hear this and to go through the experience, because it taught me that I really need to focus on finding the simple problem quickly and not rebuilding things the way I think they ought to be built. After 17 years in the industry, that’s a difficult habit to change, but one I’ll have to in order to be successful on the real lab.

After a brief recap and break, we moved on to the configuration section. For the most part there were no surprises here, and I had my Layer-2 (Frame, Spanning-tree, VTP, etc.) and IGP (RIP, OSPF, and EIGRP here) set up quickly enough. Redistribution was what you’d expect, with a lot of everything going every which way. Again, no one in their right mind would ever design that network, but it’s what you can expect to see in the lab.

The one thing I did miss and had to have Bruno point out to me, is in a redistribution task regarding OSPF. The task wanted a route from one area to show up in area 0. I got the route there, but Bruno said that I had it wrong. Reason? The area where the route originated was discontiguous, or detached from area 0. We all know that typically means you want a virtual link, but since the task didn’t specify this I simply brought the route into area 0 as an external. Bruno said that the task “implied” a virtual link, and while I disagree with the wording of the task and the nature of implied configurations, it was helpful to hear since this is likely the same kind of thing I’ll see in the real lab.

Where I slowed down–and I knew I would–is on the MPLS and BGP configuration sections. As a long-time enterprise engineer, I simply don’t touch either of these technologies in the real-world, and I haven’t spent enough time with them in the lab to feel comfortable. I still muddled my way through some of it, but with the amount of time it took I’d never make it through the real lab. The message for me here is that I really need to take some time with these technologies until I not only understand them well, but can configure them quickly.

Overall, this was a very valuable experience and one I would heartily recommend to anyone looking to take the R&S lab. It gave valuable insight into the time pressures you’ll face, as well as the number of tasks, the wording, and the level of difficulty you can expect to see. This is just one more reason that Cisco Live is where you want to be every year if you’re at all serious about your networking career.

Incoming search terms for the article:

• WS-C2960S-48TS-L
• cisco asa 5510
• ccie
• cisco 2960 stack
• requirements for CCIE lab
• ine ccie lab topology
• ine ccie topology
• CCIE home lab cabling
• bonjour cisco wlc 5505
• bazic ccie lab

Oh, and keep up with real-time information on twitter where I hide behind the handle @someclown, or G+ where I can be found at: http://gplus.to/someclown.