We are a long time VMware shop, and when I came on board back in 2006 I made a push for even more utilization. We started expanding heavily and made the choice at the time to use Equalogic iSCSI SANs for our backend storage solution. This worked well for a number of years, but now that Dell has purchased Equalogic we have seen our support quality slipping, resolution times stretching, and parts deliveries slowing significantly. As such we moved to NetApp.

One of the things with NetApp that makes it an intriguing solution is the software, particularly around cloning of virtual machines as well as snapshots and de-duplication. While these are powerful technologies to be sure, sometimes getting your head around the details can be tricky. Even trickier? Mating up what you thought you knew, or what you’ve grown used to, with the reality now.

What am I talking about in particular? Defragmentation and Microsoft.

Everyone who uses a SAN and VMware probably knows by now that defragmentation, especially where VMDK files are concerned, is a giant waste of time. After all, you’re trying to move data blocks on a hard drive around to make them more efficient, but those blocks are really just representations of blocks inside of a file, using blocks, on multiple hard drives. Simple, right?

Well, with NetApp snapshots the reasons for not defragmenting get even more ammunition: you’ll actually use more space. Why? Because the snapshots are tracking change blocks (deltas) and so don’t take up any space at all when first created (or very little) since you’re just duplicating the root inode. Every time you defragment, you’re potentially rearranging all of the “blocks” of the file system, which is going to then trigger a bigger delta come the next snapshot. It’s the same reason why you sometimes delete space on a volume and don’t see it: the snapshot has to grow by the same amount as the change, and since the snapshots are on the same volume as the data… well, there you go.

So, all of this is great. Don’t run defrag and life is good, right? Sure. But Microsoft is trying ever harder to be helpful and they’re actually crossing more and more into that territory occupied by Apple that I don’t care for: the one where they obscure all of the details to “just make it work” and you have to hunt to find even the most basic of features.

Turns out that in Windows 2008 and 2008 R2, defragmentation is set up as a scheduled task by default. Every Wednesday as a matter of fact. The good news is that the task is disabled out of the box. The bad news? Most server admins have an almost pavlovian need to defragment Windows boxes and probably have turned this (or some variation) on for many machines in your environment. Possibly even through a GPO that someone set and forgot many cycles ago.

At the end of the day, defragmenting a virtualized Windows box might make the OS think its happy, and it might even make the server admins happy. Storage folks, however? The shrieking from the unfashionable wing of the IT area will probably be all the indication you need that bad things are afoot. Hell, if you’re really looking for some of the old ultra-violence, run defrag on all your machines at night… at the same time. If you’re lucky, and everyone’s asleep, you might even manage to offline a LUN. And that’s just good fun for everyone.

Incoming search terms for the article:

• snapdrive for unix script
• ccie blog
• nm-air-wlc6-k9 software forums
• windows 2008 r2 defrag isci lun

(1)  The first thing you have to realize about working in Information Technology is this: it automatically leads to six-figure incomes, sports cars, and attractive women.  Most people won’t tell you this, and in fact go to great lengths to cover it up—much like we in the Seattle area tell people it rains all the time so you won’t move here—but it is true.  Anyone who denies this is either lying to you, or incompetent.  If you apply for a position in IT (cool-kids abbreviation warning) you should definitely expect this as a minimum package.  If you don’t get offered all of these things up front, or hear anything faintly resembling an insinuation that you might need something called experience, run the other way: this job is beneath you.

(2)  This brings us to another good point that we should discuss right away: this notion of experience.  Experience is something that people who’ve sat around at their job long enough claim you need in order to do what they do.  The reality, however, is far different.  Most of these so called “experienced” people have long ago given up on being useful, and are simply waiting around to retire.  They’re slow, ineffectual, and don’t know half of what you do.  They’re your parents age, aren’t cool, don’t dress right, stay home on weekends, don’t come in with hangovers, and sit around so much it’s painfully obvious they don’t do anything.  Experience is just a word they toss out there to keep fresh young people who know more than they do from exposing their weaknesses to the sober light of day.  Scoff openly when presented with the need for experience.  Tossing in an “old” joke or two wouldn’t hurt either… it helps let people know you’re on to them.

(3)  Everyone knows that IT types in general, and network engineers in particular, are opinionated people.  All day, every day, we’re called upon to give voice to others’ technology insecurities; to make them feel better by telling them what is good and bad in any given situation.  To truly be successful—to truly rise above the crowds of mediocrity in the field—you’ll need to take this natural predilection for opining and crank it up a few notches.  The best way to do this is to form as many opinions on technology as possible, and then never waver from them.  It works even better if your opinions aren’t based on anything useful like quantifiable data or experience, but rather on ego.  You’ll also want to pick technologies to evangelize that either few people know, you don’t currently have in place (this helps tremendously, because you can be the “expert” without having to get your hands dirty by proving it), or that make you seem “cool”.  To wit, let’s look at point number 4:

(4)  Technologies like Apple are ubiquitous in the network engineering world.  They are good products in many ways, but that’s not why you’ll want to use them.  You’ll want to use them because that’s what all the “cool” kids are using.  Old people with “experience” use PCs and you don’t want to be associated with that.  Having some off-handed platitudes about why you use Apple computers is going to be good here; things like “I only use the best tool for the job” is a great one.  If challenged, or heaven forbid proven wrong, above all else don’t acknowledge this.  Simply wave your hands in a dismissive way and insist that somehow the thing you like about Apple really hasn’t been disproven, and move on.  This doesn’t apply only to Apple, of course, you can use this technique to make yourself look smarter than those around you with just about anything.  As I stated, however, you’ll really want to pick as many technologies as possible that don’t have much market penetration in your company or circle of influence.  If you pick something well-known to evangelize, you run the risk of being labeled as difficult to work with.

(5)  On the topic of difficult to work with, this can be important as well.  Agreeable people get nowhere in corporate America, and you certainly aren’t working for any team.  The best thing to do with any new job is to immediately establish that you won’t play by everyone else’s rules.  The most effective way I’ve found to make this happen is to constantly complain about how much better things were done “at my old company.”  This applies even if, as is likely, your old company was just college.  What this does is establish the fact that you’ve seen better, you know better, and you won’t be held back by mediocrity.  It also lets everyone know up front that you aren’t a team player, and that you’ll drive the bus of success all on your own thank you very much.

There are many more tips I can share, and I’m certain that this is probably just the start of a multi-part article.  It is, after all, incumbent on those of us who have been in the industry for a while now to try to pass on all that we’ve learned to the next generation.  There is far too much disinformation out there on so-called “success,” and I believe that whatever I can do to debunk the common mythology, it is for the best.

Incoming search terms for the article:

• NUOVO CISCO ANYCONNECT MOBILE - ASA 5505 L
• 2960s-48lps-l
• anyconnect-essentials
• cisco 2054 wlc
• Labgear DVR LAB download
• screenrc bindkey -k k9

I won’t bother you with my initial reactions to the Mac as they’ve been mostly positive and probably not entirely different than many other people’s experiences.  What I will say, though, is that as a long time user of various flavors of Unix, it’s nice to be operating on a platform that is Unix based again.  Specifically, it’s nice to not have to download bolt-on software in order to be able to use SSH, Finger, TFTP, etc.

One of the tools that I have used extensively in the past is GNU Screen.  For those unfamiliar with the program, you can check out a brief overview over at Wikipedia: http://en.wikipedia.org/wiki/GNU_Screen, or at the GNU Screen website here: http://www.gnu.org/software/screen/.

Despite using Screen since I was first introduced to it back around 1990 or so, if I recall correctly, I’ve only ever used it really for one thing: connecting from a Unix Machine to something using the serial port or a modem.  Even the modem bit I didn’t use all the time as I tended to favor Minicom.  So, when a couple of friends suggested that I use Screen for solving a problem I was having, it made sense.

As part of my Cisco lab I have some fourteen different devices that I need to access at any one time.  I have all of those devices set up to my main terminal server (3945 which is also my main a-lot-of-other-things as well) using reverse telnet.  In this way I can do a lot of things, but typically I would SSH to the terminal server, connect to a host by name, then use Ctrl-Shift-6 x to get back to the terminal server and do the same thing again for another host.

This is all made possible using an Async card:

NAME: "High Speed Wan Interface card with 16 RS232 async ports(HWIC-16A) on Slot 0 SubSlot 3", DESCR: "High Speed Wan Interface card with 16 RS232 async ports(HWIC-16A)"

With a pertinent configuration like so: 

interface Loopback0

ip address 172.16.0.1 255.255.255.0

ip host s2 2054 172.16.0.1

ip host s1 2053 172.16.0.1

ip host s4 2056 172.16.0.1

ip host s3 2055 172.16.0.1

ip host r5 2063 172.16.0.1

ip host r2 2060 172.16.0.1

ip host r6 2064 172.16.0.1

ip host r4 2062 172.16.0.1

ip host r3 2061 172.16.0.1

ip host r1 2059 172.16.0.1

ip host r8 2066 172.16.0.1

ip host r7 2065 172.16.0.1

ip host ASA 2051 172.16.0.1

ip host AP1 2052 172.16.0.1

ip host bb2 2057 172.16.0.1

ip host bb1 2058 172.16.0.1

line 0/3/0 0/3/15

session-timeout 120

no exec

transport input telnet

Still, opening fourteen hosts can get tedious, as you might imagine.

What this configuration also allows for, however, that I had only been using occasionally is logging in to any device by opening a telnet connection to the port.  So, I can either telnet to the parent device’s IP address (in this case it’s 10.7.68.221) or to the loopback address (if I have routing) and adding on the port number like so:

telnet 172.16.0.1 2056

which, as we see from above, would take me to the device labeled “s4”.   That’s all good, but still requires me to open each telnet session by hand, right?  Not necessarily: enter Screen and the magic of configuration files.

I have two configuration files for Screen: one is a .screenrc file which sits in my home directory as is standard on Unix systems.  Note that I could also put this in my /etc directory and change the name if I wanted a system-wide default for all users.  The other file is a custom one that I only use when I’m doing lab-work, and I call that one screen.ccie.

If I want to open just Screen by itself, from a terminal (I use iTerm2) I just type “screen –A” which opens up Screen with all possible emulations enabled (just in case).  If I want to open my ccie lab configuration I type “Screen –Ac screen.ccie” to reference the startup file.  I also use “screen –A /dev/tty.KeySpansomething” to use my USB/Serial adapter if I’m at the console somewhere.

Below you’ll find the screen.ccie file, with comments, as well as a screen shot of what it looks like in action.  I didn’t include the .screenrc file because it is identical, except for the last section that opens up all of the different telnet sessions.  I’m sure there are a lot of ways to do this, some better than others, and infinite combinations of colors, stats, etc. that can be configured, but this is what I have for now.

# SomeClown's .screenrc file.  Cribbed from a lot of places.
# Credit to both @IPv6Freely and @jay25f

# Next and Previous tabs shortcuts
bindkey "^e" next       # Next
bindkey "^w" prev       # Previous

# Various options
escape ^Zz                      # Control-A is needed elsewhere
nethack on                      # Just 'cuz
maptimeout 5            # Fixes Vi
vbell off                   # Pavlov
startup_message off     # Meh
defscrollback 30000     # Lots o' logging room
autodetach on           # Go away, come back.
shelltitle ""

# Status at bottom of screen; tabs, etc.  The code's a FusterCluck, but looks nice
hardstatus alwayslastline "%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%< %=%D %M %d %c"
hardstatus string '%{= kK}%-Lw%{= KW}%50>%n%f %t%{= kK}%+Lw%< %{=kG}%-= %d%M %c:%s%{-}'

# Specific for CCIE Lab.  Standard .screenrc file doesn't include text below

# Create named tabs for each device
screen -t Stuff 0                       # General
screen -t R1    1
screen -t R2    2
screen -t R3    3
screen -t R4    4
screen -t R5    5
screen -t R6    6
screen -t R7    7
screen -t R8    8
screen -t S1    9
screen -t S2    10
screen -t S3    11
screen -t S4    12

# For each tab above, do stuff
at R1 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2059 \015"
at R2 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2060 \015"
at R3 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2061 \015"
at R4 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2062 \015"
at R5 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2063 \015"
at R6 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2064 \015"
at R7 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2065 \015"
at R8 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2066 \015"
at S1 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2053 \015"
at S2 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2054 \015"
at S3 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2055 \015"
at S4 stuff " echo 'Starting some shite...' && sleep 10 && telnet 10.7.68.221 2056 \015"

Incoming search terms for the article:

• reverse telnet
• blog packetqueue net
• cisco wlc module apple bonjour
• cisco wlc config for apple bonjours
• gnu screen blink on mac
• Cisco 3945
• mac where is screenrc
• ctrl-shift-6 nexus using mac terminal
• screenrc file for macbook pro
• screenrc defscrollback infinity

Incoming search terms for the article:

• cisco labs
• cisco home lab 2011
• cisco ethernet rack cable
• cisco live 2012
• cisco live 2011 las vegas ltrccie-3001
• cisco lab on esx call manager
• 2801 router diagram
• potato clock hook up diagram
• cisco iou ine
• ltrccie-3001 ccie routing and switching lab

Without going into the details of the entire Cisco Live experience, I’d just like to talk about the class I took on the first Sunday of the show–or the day before the show officially starts, depending on who you talk to.

On Sunday I attended a full-day mock CCIE R&S lab (Session LTRCCIE-3001). This was an instructor-led affair, with Bruce Pinsky (Distinguished Engineer) and Bruno van de Werve (Product Manager) acting as facilitators and proctors. Considering Bruno’s experience as both a proctor for the actual R&S lab, and now the head of the R&S program, this was an experience well-worth having if only for the ability to ask questions.

Unfortunately for all of us, and through no fault of either Bruce or Bruno, the in-class network was crashed from the moment we all got there. There were a number of failures, including some bad cables (how do you miss that in testing) which resulted in all of us essentially sitting around for over an hour.

To make up for the delay in getting started, someone from Cisco came in and apologized and handed out gift cards to Mandalay Bay. It was a nice gesture, but considering the gift cards had a face value of five dollars, it might have been better to not hand out anything. It had the affect of actually irritating several students, and giving the rest of us something to joke about for a while. The class cost $1000 (or 10 Cisco Learning Credits) so the value of even an hour should have been closer to $125 or so.

After that snafu, and a brief presentation by Bruno and Bruce on numbers of CCIE in the world, with breakdowns by region, we got started with the meat of the class: the labs themselves. We were all looking forward to this, since it was being run by Cisco and had the smell of real-world vs. some of the third-party labs (note that I use third party labs for training, and have no problems with them, but this was officially sanctioned and so had a little something extra, at least in “feel.”)

The troubleshooting section came first, and used the same system as the real lab so that was a nice touch. In our case we had only five trouble tickets to complete in one hour vs. the real lab which has ten in two hours. I believe this was done to facilitate the “instructor led” nature of the class, and allow us to ask plenty of questions. Bruce and Bruno were stellar in this regard, coming around to any student with a question and helping them to understand the problem or just passing out hints to those who still wanted to figure it out on their own.

I learned a lot about myself and my troubleshooting techniques during this portion of the day, as I got bogged down on the first ticket and blew the rest of my time. It was a relatively straightforward ticket where a particular address wasn’t answering an ICMP Echo to another device. It was a few routers together, with BGP. I spent the entire hour re-architecting the BGP–down to bare metal and rebuilding the config from scratch–and almost was done when time expired. As it turned out, it was a simple address statement that was missing.

Bruno got a chuckle out of this and pointed out that the lab is not intended as a “best practices” lab. He said that in most cases you won’t be removing configuration at all during the TS section; you’ll simply be adding something missing or correcting route statements, etc. It was helpful for me to hear this and to go through the experience, because it taught me that I really need to focus on finding the simple problem quickly and not rebuilding things the way I think they ought to be built. After 17 years in the industry, that’s a difficult habit to change, but one I’ll have to in order to be successful on the real lab.

After a brief recap and break, we moved on to the configuration section. For the most part there were no surprises here, and I had my Layer-2 (Frame, Spanning-tree, VTP, etc.) and IGP (RIP, OSPF, and EIGRP here) set up quickly enough. Redistribution was what you’d expect, with a lot of everything going every which way. Again, no one in their right mind would ever design that network, but it’s what you can expect to see in the lab.

The one thing I did miss and had to have Bruno point out to me, is in a redistribution task regarding OSPF. The task wanted a route from one area to show up in area 0. I got the route there, but Bruno said that I had it wrong. Reason? The area where the route originated was discontiguous, or detached from area 0. We all know that typically means you want a virtual link, but since the task didn’t specify this I simply brought the route into area 0 as an external. Bruno said that the task “implied” a virtual link, and while I disagree with the wording of the task and the nature of implied configurations, it was helpful to hear since this is likely the same kind of thing I’ll see in the real lab.

Where I slowed down–and I knew I would–is on the MPLS and BGP configuration sections. As a long-time enterprise engineer, I simply don’t touch either of these technologies in the real-world, and I haven’t spent enough time with them in the lab to feel comfortable. I still muddled my way through some of it, but with the amount of time it took I’d never make it through the real lab. The message for me here is that I really need to take some time with these technologies until I not only understand them well, but can configure them quickly.

Overall, this was a very valuable experience and one I would heartily recommend to anyone looking to take the R&S lab. It gave valuable insight into the time pressures you’ll face, as well as the number of tasks, the wording, and the level of difficulty you can expect to see. This is just one more reason that Cisco Live is where you want to be every year if you’re at all serious about your networking career.

Incoming search terms for the article:

• cisco 2960 stack
• WS-C2960S-48TS-L
• home cisco lab
• ccie
• cisco asa 5510
• 2960s stacked
• ASA 5505 de Cisco
• ASA 5510 fan
• ccie topology
• Apple IIc Plus

Oh, and keep up with real-time information on twitter where I hide behind the handle @someclown, or G+ where I can be found at: http://gplus.to/someclown.

So it is with great excitement that I now have in my possession a book on that early time, written about one of the pioneering companies subject to more historical revisionism than most people realize.

“Commodore, a company on the edge” by Brian Bagnall is an in-depth, interesting, and more historically accurate portrayal of the early history of microcomputers in general, but Commodore in particular, than many I’ve read. Most early histories are written by revisionist authors like Robert Cringely and tend to dramatically overstate Apple and IBM’s contributions at the expense of Commodore and Atari, among others.

I’ll post a complete review when I’m done with the book, but just what I’ve read so far has me pining for simpler times. Before I knew acronyms like CCIE, OSPF, NX-OS and had a global enterprise network to tame, I had my Apple II, the Commodore 64, the Amiga 1000 and the Atari ST. They say you can’t go back again, but I’m trying.

You must turn on IPv6 by using the IPv6 unicast-routing command.

Not true.  This is one of the more persistent, yet wildly incorrect, pieces of information regarding IPv6.  I have even seen many training centers and instructors at the CCIE level get this one wrong and it falls into the category of attention to detail.  What this command actually does is enable unicast routing for IPv6, just as it says.  To actually enable IPv6 you simply need to go to any interface and use the ipv6 enable command.  And yes, you can enable IPv6 on the interface without enabling unicast routing.  Of course, it would be helpful to have an address on the interface as well.

Yes, but if you don’t turn on unicast routing you can’t route IPv6 traffic.

Not strictly speaking true.  You can still set up a default route for IPv6 traffic and get it off of your system.  To the extent that you want to argue whether or not this is actually routing is fine, but you can move IPv6 traffic off of your local device using a default route, and never have enabled routing for IPv6.

Using a /127 address on point-to-point links is wrong, wrong, wrong.

This is an interesting one, and usually sparks a fair amount of debate.  Up until very recently, the recommendation across the board (RFC 4291) was to use /64 addresses even on point-to-point links, ostensibly because the IPv6 space is so big anyhow, and because several protocols will break (notably subnet-router anycast, specified in RFC 3627).  While I’m not disputing that this is what the current best-practices reflect, I will say that RFC 6164 which has a status of Proposed Standard makes a fairly compelling case for using /127 on point-to-point links.  I’m sure this won’t be resolved anytime soon, standards or no, but I would say that if you have a compelling reason for using /127 and know what you’re doing it for, go for it.  Just be aware that standards can change, and you don’t want to leave a steaming pile for the poor person who has to follow you.

Incoming search terms for the article:

• rfc6164 cisco
• rfc 6164
• rfc 6164 cisco
• ipv6 /127 point to point is bad
• ipv6 /127 point to point
• how disable subnet router anycast cisco
• disable Subnet-Router anycast ipv6 cisco
• cisco support rfc 6164
• cisco rfc 6164
• cisco disable subnet-router anycast

(1) My study hours don’t always match well with what slots the online rack vendors have available.

(2) I just like physical equipment and the flexibility it provides in both studying and in research.

Also, I just wanna.

With that said, one of the next things people want to know is what gear it is that I have, and how do I have it configured.  Therefore, with the recent posting frequency here severely lacking, writing about my lab is a nice way to get something fresh on the blog and hopefully it provides something useful to someone out there.  I’m going to break this down into two general categories: equipment that I have purely for my Cisco CCIE lab, and other equipment that I have either for my home network or for random reasons.

Random Non-Lab Specific Equipment List:

1. WS-2950T-24 switch
2. Two 1142N access points
3. Wireless Controller Module (NME-AIR-WLC6-K9) which is pretty fun, but breaks bonjour and so is the bane of my existence (see previous post here.)
4. ASA 5505 with IPS module, running bot-net filter and some other things.  This is also the main gateway for the home network, connecting up to the cable modem.  It’s also the IPsec endpoint for my always-on connection to the office and segments my home network, lab network, work network, etc.
5. Comcast Cable Modem, made by Motorolla
6. Random doohickey for my “Whole-Home DVR” with DirecTV
7. Sun SunFire v240 Server with StorEdge 3300 storage array

Non-plugged in Equipment

1. Sun Enterprise 3500
2. Two Cisco 3550 switches
3. Two PIX 501

Computer storage:

1. Seagate 750GB external drive (USB)
2. Iomega 1TB external (eSATA)
3. Drobo S with 5 2TB drives for 10TB raw (eSATA)

Cisco Lab Equipment:

1. Four 3560-X switches, with four-gig uplink modules (might still get the 10 at some point), fully licensed with IPServices and running 12.2(53) SE2.
2. Eight 2801 routers, all running 12.4(22)T5 Advanced Enterprise, and all with at least one Wic-2T smart serial card which provides two smart serial connections.  Four of the 2801 have two Wic-2T cards, and a couple others have a mixture of 1-Wic-DSU-T1 cards, FXO cards, and FXS cards (mostly leftovers and hand-me downs, but there are some interesting possibilities.)
3. One 2811 running the same IOS as the 2801 routers, used as a backbone router for injecting routes and some other misc. stuff.
4. One 2621 running something-or-other and acting as another backbone router.
5. One 3845 running the same Advanced Enterprise as the others.  This has five Wic-2T cards and acts as the frame switch. It also has an HWIC-16A card and does reverse-telnet to everything else (terminal server).  It also houses some random stuff including the wireless controller mentioned above.

All of this is cabled and wired almost identically to the CCBootCamp lab topology.  This is because I have all of their workbooks and wanted to be able to study with my own equipment.  A couple of the details are different, mostly around interface numbers and the specifics of the backbone routers and such.  Also, the switches I have are way overkill but satisfy the lab requirements.  Given the actual topology from just about any mainstream training provider, I can copy it with the equipment I have, and that’s exactly what I wanted to be able to do.  As always, contact me here or on twitter with questions and comments.

Pictures of the lab and sundries are included below.

Incoming search terms for the article:

• ccie lab
• ccie home lab
• asa 5505
• ccie rack
• ccie lab equipment
• cisco 3550 lab
• ccie lab equipment 2011
• ccie lab home lab
• How to study for the ccie lab
• installing labgear to a data cabinet