packetqueue.net

So, nostalgia about computers and the “good ole’ days” gets strong with me whenever I turn on my old Apple II, or pull out any old magazines from the early 80′s or 90′s. My wife laughs at me because I still get goosebumps and a light in my eye describing how I felt when I first saw the Apple IIgs or the Amiga 1000, and how bad I wanted them. It takes me back to when computers were fun, magical, and represented a brave new world to my young mind (with apologies to Aldous Huxley).

So it is with great excitement that I now have in my possession a book on that early time, written about one of the pioneering companies subject to more historical revisionism than most people realize.

“Commodore, a company on the edge” by Brian Bagnall is an in-depth, interesting, and more historically accurate portrayal of the early history of microcomputers in general, but Commodore in particular, than many I’ve read. Most early histories are written by revisionist authors like Robert Cringely and tend to dramatically overstate Apple and IBM’s contributions at the expense of Commodore and Atari, among others.

I’ll post a complete review when I’m done with the book, but just what I’ve read so far has me pining for simpler times. Before I knew acronyms like CCIE, OSPF, NX-OS and had a global enterprise network to tame, I had my Apple II, the Commodore 64, the Amiga 1000 and the Atari ST. They say you can’t go back again, but I’m trying.

Incoming search terms for the article:

• VSphere Enterprise Diagram
• cisco 1142 bonjour protocol arp

This post will be a short one, and mostly just comes from a discussion I had the other day with another engineer.  It turns out that even among people who are comfortable with IPv6, and maybe even have experience deploying it, a lot of misinformation still persists.  Hopefully I can correct a couple of those today.  I also tossed in a hot-potato at the end just to see how many folks get hopped up.  Discussion is welcome, and in addition to comments here I can be found on twitter hiding behind the handle: @someclown.

You must turn on IPv6 by using the IPv6 unicast-routing command.

Not true.  This is one of the more persistent, yet wildly incorrect, pieces of information regarding IPv6.  I have even seen many training centers and instructors at the CCIE level get this one wrong and it falls into the category of attention to detail.  What this command actually does is enable unicast routing for IPv6, just as it says.  To actually enable IPv6 you simply need to go to any interface and use the ipv6 enable command.  And yes, you can enable IPv6 on the interface without enabling unicast routing.  Of course, it would be helpful to have an address on the interface as well.

Yes, but if you don’t turn on unicast routing you can’t route IPv6 traffic.

Not strictly speaking true.  You can still set up a default route for IPv6 traffic and get it off of your system.  To the extent that you want to argue whether or not this is actually routing is fine, but you can move IPv6 traffic off of your local device using a default route, and never have enabled routing for IPv6.

Using a /127 address on point-to-point links is wrong, wrong, wrong.

This is an interesting one, and usually sparks a fair amount of debate.  Up until very recently, the recommendation across the board (RFC 4291) was to use /64 addresses even on point-to-point links, ostensibly because the IPv6 space is so big anyhow, and because several protocols will break (notably subnet-router anycast, specified in RFC 3627).  While I’m not disputing that this is what the current best-practices reflect, I will say that RFC 6164 which has a status of Proposed Standard makes a fairly compelling case for using /127 on point-to-point links.  I’m sure this won’t be resolved anytime soon, standards or no, but I would say that if you have a compelling reason for using /127 and know what you’re doing it for, go for it.  Just be aware that standards can change, and you don’t want to leave a steaming pile for the poor person who has to follow you.

Incoming search terms for the article:

• cisco disable subnet-router anycast
• disable cisco subnet router anycast
• CCIE Routing and Switching Certification Guide 4th Edition for sale
• ipv6 /127 address point to point
• rfc 6164 best practise
• rfc6164 cisco
• WLC failover scenario

(1) My study hours don’t always match well with what slots the online rack vendors have available.

(2) I just like physical equipment and the flexibility it provides in both studying and in research.

Also, I just wanna.

With that said, one of the next things people want to know is what gear it is that I have, and how do I have it configured.  Therefore, with the recent posting frequency here severely lacking, writing about my lab is a nice way to get something fresh on the blog and hopefully it provides something useful to someone out there.  I’m going to break this down into two general categories: equipment that I have purely for my Cisco CCIE lab, and other equipment that I have either for my home network or for random reasons.

Random Non-Lab Specific Equipment List:

1. WS-2950T-24 switch
2. Two 1142N access points
3. Wireless Controller Module (NME-AIR-WLC6-K9) which is pretty fun, but breaks bonjour and so is the bane of my existence (see previous post here.)
4. ASA 5505 with IPS module, running bot-net filter and some other things.  This is also the main gateway for the home network, connecting up to the cable modem.  It’s also the IPsec endpoint for my always-on connection to the office and segments my home network, lab network, work network, etc.
5. Comcast Cable Modem, made by Motorolla
6. Random doohickey for my “Whole-Home DVR” with DirecTV
7. Sun SunFire v240 Server with StorEdge 3300 storage array

Non-plugged in Equipment

1. Sun Enterprise 3500
2. Two Cisco 3550 switches
3. Two PIX 501

Computer storage:

1. Seagate 750GB external drive (USB)
2. Iomega 1TB external (eSATA)
3. Drobo S with 5 2TB drives for 10TB raw (eSATA)

Cisco Lab Equipment:

1. Four 3560-X switches, with four-gig uplink modules (might still get the 10 at some point), fully licensed with IPServices and running 12.2(53) SE2.
2. Eight 2801 routers, all running 12.4(22)T5 Advanced Enterprise, and all with at least one Wic-2T smart serial card which provides two smart serial connections.  Four of the 2801 have two Wic-2T cards, and a couple others have a mixture of 1-Wic-DSU-T1 cards, FXO cards, and FXS cards (mostly leftovers and hand-me downs, but there are some interesting possibilities.)
3. One 2811 running the same IOS as the 2801 routers, used as a backbone router for injecting routes and some other misc. stuff.
4. One 2621 running something-or-other and acting as another backbone router.
5. One 3845 running the same Advanced Enterprise as the others.  This has five Wic-2T cards and acts as the frame switch. It also has an HWIC-16A card and does reverse-telnet to everything else (terminal server).  It also houses some random stuff including the wireless controller mentioned above.

All of this is cabled and wired almost identically to the CCBootCamp lab topology.  This is because I have all of their workbooks and wanted to be able to study with my own equipment.  A couple of the details are different, mostly around interface numbers and the specifics of the backbone routers and such.  Also, the switches I have are way overkill but satisfy the lab requirements.  Given the actual topology from just about any mainstream training provider, I can copy it with the equipment I have, and that’s exactly what I wanted to be able to do.  As always, contact me here or on twitter with questions and comments.

Pictures of the lab and sundries are included below.

Incoming search terms for the article:

• ccie lab
• ccie home lab
• ccie rack
• ccie lab equipment
• asa 5505
• ccie lab 2811
• storedge 3300en w
• ccie study schedule
• router lab ccie
• study of telnet lab

Everyone has been posting their schedules for Cisco Live to Twitter, Facebook and wherever else, so I thought I’d better jump in with the cool kids and publish mine as well.  I can’t guarantee this won’t change, but for now it stands as my best guess and current planned schedule.

Incoming search terms for the article:

• cisco live 2012 san diego
• cisco live 2012 san diego dates
• cisco live 2012 san diego schedule

Editor’s Note: Google Chrome seems to dislike my site theme and is hyphenating absolutely everything.  Apologies for that, and I’ll look into it just as soon as I get done with a few items on the “honey-do” list.

If you are studying for the CCIE Routing and Switching exam, one of the technologies that is still heavily prevalent is Frame Relay.  It is expected that you know both the technology itself, and how to configure it, but also how it interacts with and affects other key technologies like OSPF and EIGRP.  Having the ability to study Frame Relay, then, and get plenty of hands-on configuration time becomes as important as with anything on the R&S 4.0 Blueprint.

While many network engineers are already familiar with Frame Relay from a consumer side–in other words, from the perspective of an entity which buys Frame Relay services from a provider–not many of us are familiar with the service provider portion of the equation.  This makes setting up practice labs difficult if you are trying to study using your own equipment.  Fortunately, you can set up your own Frame Relay switch fairly easily, and that is what we’re going to walk through today.

A Frame Relay switch is the DCE device that sits inside a service provider’s network and moves the frames along from point A to point B.  There are many of these devices all working together inside of your provider’s network to move your information along, but fortunately for lab candidates studying at home, you can easily get by with just one.  Even more fortunate is that you can use a fairly low powered router to act as a Frame Relay switch, and not miss anything that you’ll need for purposes of the lab.

A quick note on the lab is in order here.  It used to be a part of the lab blueprint (dont’ ask me which one, or how far back in time) that you had to know how to set up a Frame Relay switch.  Cisco has since taken that requirement away, at least from the R&S lab, and so a lot of that knowledge isn’t communicated in teaching texts any longer.  What you’ll find in the lab itself is an already configured Frame Relay switch that you’ll have no direct access to, but all of the information you need to make your equipment talk to it.

It may seem counterintuitive, but for a home lab the best device to use for a Frame Switch is actually a router.  For instance, I’m using an older Cisco 2621 model for my Frame Switch, and it does everything I need it to do.  Service providers will typically use more specialized gear, but all we’re going for in our studies is a reasonable facsimile.  If you want to spend a lot of money, follow the advice of so many others and spend it on your layer-3 switches.

Another thing we want to briefly discuss is interfaces.  Generally speaking, you can either follow the “run what’cha brung” philosophy of just using what you have access to, or you can buy the interfaces you want.  In my case I had a couple of WIC-T1 cards that I’ve used, and then I bought a handful of WIC-2T serial interface cards.  The key is to have a serial interface for each router you want to connect via the Frame switch.  So I have one T1 interface, and six serial interfaces for a total of seven devices I can connect into the Frame “cloud”.  I find this to me more than adequate, though if you’re trying to duplicate a specific topology you may need more or less.

The configuration of a Frame switch is actually very simple, as you’ll see, though attention to detail does matter.  I’m assuming here, by the way, that you already know how to set up your router for basic access, clock, etc., so I won’t cover that here.  So, the first step in configuring your router to be a Frame switch is to put it into Frame switching mode using the commands:

ip cef

frame-relay switching

These commands turn on Cisco Express Forwarding, put the router into a Frame switching mode, and change quite a bit of the default behavior, so don’t expect to use this device as a router in any lab topology you’re working on.  This device will be just a Frame switch and nothing more.

The next step is to configure the individual interfaces you’ll connect your other routers to, and you have a lot of choices here.  I don’t know exactly how the R&S lab devices are set up, so I’m just going to give you the configuration I use.  I’ll post the configuration below, and then go over the key comands:

interface Serial0/1

no ip address

encapsulation frame-relay

logging event subif-link-status

logging event dlci-status-change

clock rate 8000000

no frame-relay inverse-arp

frame-relay intf-type dce

frame-relay route 220 interface Serial0/2 120

frame-relay route 221 interface Serial0/0 320

The first few lines of the configuration should be familiar to you already.  We’re setting our interface encapsulation to frame-relay, and then logging on a couple of events.  The logging is completely up to you, and not necessary one way or another.  I just find them helpful.  Next we set the clock rate, and we tell the interface that we are the DCE end of the connection.  Remember, in a Frame Relay network the clocking (DCE end) comes from the line or provider side, so this is what you’ll want.  If I am working with a T1 serial interface, I’ll also need a line for that:

service-module t1 clock source internal

This can change depending on the type of card and how you have it configured.

Now, the other options we have here require a little more explanation.  The “no frame-relay inverse-arp” command does just what it says, and you can argue for the Frame switch having this turned on, or off.  In most cases in the lab, you’ll be instructed to not use inverse arp on the DTE devices, so I’ve just turned that functionality off on my Frame switch from the outset.  It’s really your call.

The next two lines beginning with frame-relay route are the ones that always seem to cause confusion.  You can read the first line as “If some traffic comes in from DLCI 220, with a destination of DLCI 120, send it out interface Serial 0/2″.  Substitute DLCI 221 and 320 on the next line, but otherwise read it the same.  So if I now plug in a router to interface Serial 0/1, and assign DLCI 220 and 221 to two different sub-interfaces (for instance, different options are possible) the Frame switch will know what to do with that traffic.

So, if we have a diagram that looks like the following:

Then we have a configuration for interfaces that looks like so:

interface Serial0/0

no ip address

encapsulation frame-relay

logging event subif-link-status

logging event dlci-status-change

service-module t1 clock source internal

no frame-relay inverse-arp

frame-relay intf-type dce

frame-relay route 320 interface Serial0/1 221

frame-relay route 321 interface Serial0/2 121

!

interface Serial0/1

no ip address

encapsulation frame-relay

logging event subif-link-status

logging event dlci-status-change

clock rate 8000000

no frame-relay inverse-arp

frame-relay intf-type dce

frame-relay route 220 interface Serial0/2 120

frame-relay route 221 interface Serial0/0 320

!

interface Serial0/2

no ip address

encapsulation frame-relay

logging event subif-link-status

logging event dlci-status-change

clock rate 8000000

no frame-relay inverse-arp

frame-relay intf-type dce

frame-relay route 120 interface Serial0/1 220

frame-relay route 121 interface Serial0/0 321

I hope that helps out, and as always if you have any questions or clarifications please drop me a line here or on twitter where I’m known as @SomeClown.

Incoming search terms for the article:

• frame relay
• switch frame relay
• dlci cloud
• ine frame relay switch config
• 02 router frame relay lab
• dlci
• frame relay t1 make dce
• how does frame relay know whats on the other end of circuit
• how to setup a frame relay lab
• inside frame relay cloud ip address

As is typical in the world of IT, problems have a way of sneaking up on you when you least expect it, then viciously attacking you with a Billy-club.  Often this happens when you are asleep, on vacation, severely inebriated, or have already worked 40-hours straight with no sleep.  In my case, Super-Bowl Sunday at around 8:30pm was my time to get the stick.  And get it I did.

For reasons too sad to warrant comment, and far too irritating to explain in a family forum like this, our ESX host servers all became disconnected from our SAN array.  The root problem was something else on layer-2, and got resolved quickly, but the virtual world was not so quick to recover.  In retrospect, the problem was not a bad one, but when you’ve been drinking and can’t see the obvious answer you tend to dig the hole you’ve fallen into deeper rather than climb promptly out.

By way of background, we are currently running VSphere 4.0, with a few servers having 32GB or memory and 8-cores, and a few having 512GB of memory and 24-cores. All ESX Hosts are SAN booting using iSCSI initiators on a dedicated layer-2 network.  We use Nexus 1000v soft switches and have our ESX Hosts trunked using 802.1q to our Core (6506-E switches running VS-S720-10G supervisors).  Everything is redundant (duplicate trunks to each Core switch) and using ether-channel with mac-pinning).  So there you have that, for what it’s worth.  Now back to the crashed servers.

We rebooted all of the ESX host servers, and with the exception of some FSCK-complaining they all came up quite nicely.  The problem was that none of the virtual machines came up.  Let me add that we have the domain controllers, DHCP, DNS, etc. on these hosts.  Crap.

So the first thing I did in my addled state was to add DHCP scopes to the DHCP servers at another office across the country, and point the VLANs off “that-a-way” by changing the ip helper-address on each VLAN on the Core.  That got DHCP and DNS back online.  As you can probably guess by now, I was MacGyver-ing the situation nicely, but really didn’t need to.  That’s one of the problems when you’re in the trenches: you tend to think in terms of right-now instead of root cause.

The next thing I did was to start bringing up the virtual machines one-by-one using the command line on the ESX hosts.  Why?  Because I had no domain authentication and the VSphere Client uses domain authentication.  Here is where someone in a live talk would be interrupting me to point out that the VSphere Client can always be logged into using the root user of the hosts, even when domain authentication is set up for all users.  Yes, that is true and it would have been handy to know at the time.

In order to bring up the virtual machines, I had to first find the proper name by issuing:

vmware-cmd –l

from the command line.  This command can take a while to run, especially if you have a lot of VMs sitting around, so go get a cup of coffee.

Once I had that list I prioritized the machines I wanted up first, and issued the:

vmware-cmd //server-name.vmx start

command on each one.  That should have been the end of the boot-up drama, but it wasn’t.  As it turns out, a message popped up (and I don’t remember the exact phrasing) to the effect of “you need to interact with the virtual machine” before it would finish booting.  So, now I issued the:

vmware-cmd //servername.vmx answer

command and got something that looked about like this:

Virtual machine message 0:

msg.uuid.altered:This virtual machine may have been moved
or copied.

In order to configure certain management and networking
features VMware ESX needs to know which.

Did you move this virtual machine, or did you copy it?

If you don't know, answer "I copied it".

0. Cancel (Cancel)

1. I _moved it (I _moved it)

2. I _copied it (I _copied it) [default]

Well, I didn’t know so I selected the default option (I copied it) and went on my way.  That is fine in almost every circumstance and got all of my servers booted up.  It did not, however, entirely fix the problem.  In fact, even though all of my servers were booted, none could talk or be reached on the network.

This is where a little familiarity with the Nexus 1000v soft switches comes in handy.  Very briefly, the architecture is made up of two parts: the VSM or Virtual Supervisor Module and the VEM or Virtual Ethernet Module.  The VSM corresponds roughly to the supervisor module in a physical chassis switch, and the VEMs are the line cards.  The interesting bit to remember for our discussion is that the VSMs (at least two for redundancy) are also Virtual Machines.

Some of you may have guessed already what the problem turned out to be, and are probably chortling self-righteously to yourself right about now.  For the rest of us, here’s what happened:

I figured out the log-in-using-root thing and got the VSphere client back up and running (oh, not before having to restart a few services on the Virtual Center Server, which is not a virtual machine, by the way.  I’m not totally crazy!).  Once I got that far I could log in to the Nexus VSM, and look at the DVS to see what was going on.  All of my uplink ports (except for ones having to do with control, packet, vmkernel, etc.) were in an “UP Blocked” state.

The short-term fix (again, the MacGyver job) was to create a standard switch on all hosts and migrate all critical VMs to that switch.  That didn’t, however, fix the problem permanently and besides, we like the Nexus switches and wanted to use them.  With that in mind, and a day or two to normalize the old sleep patterns, I set up call with VMware support.  This actually took longer than I expected since I had to wait for a call-back from a Nexus Engineer, and they are apparently as rare as honest sales-people or Unicorns.  That said, I did get a call back and we proceeded to troubleshoot the problem.

One thing that surprised me was that it took the Nexus Engineer a bit longer than I would have thought to find the problem, but even once he did it took longer to get resolution because we had to get Cisco involved.  The problem, as it turns out was licensing.

When you license the Nexus, you receive a PAK and you use that to install the VSM.  Once you do that, you have to request your license using the Host UID of the now installed VSM.  Cisco then sends you a license key that you install from the command-line of the VSM.  This is all somewhat standard and not surprising.  What was surprising was that we would have to do this at all considering we had been licensed at the highest level (Enterprise, superdy-duperty cool or something) for years.

What happened was that the copy VSphere made in order to get each Virtual Machine back up after our crash changed the Host UID of the VSM virtual machine(s).  Thus, the license keys were no longer valid and all host uplink ports went into a blocked state.  (I’ll save you the obvious gripe I have with the Nexus not offering any kind of command-line message about our licensing being hosed.)  This is where we had to get Cisco Licensing involved, as we had to send them the old license key files and the new Host-UID information so that they could generate new keys.  Considering I was only on the phone with them for only 15 minutes, it was as pleasant an experience as I’ve ever had dealing with Cisco’s Licensing department.  At least that’s something.

After fixing the licensing, the ports unblocked and I went through the tedium of adding back adapters to the Nexus, moving servers, etc.  At the end of the day, however, it is all back to normal and working.  There are a lot of lessons learned here, and you’ll no doubt pull your own, but the one overriding thing to be on the lookout for is that, under certain circumstances, if your Nexus VSMs are part of a crash and come back up, look to licensing first before troubleshooting anything else.  Oh, and try to schedule your major system crashes for a more convenient time… when you’re sober.  Just saying.

Incoming search terms for the article:

• Reset triggered due to HA policy of Reset
• Reason: Reset triggered due to HA policy of Reset
• nexus Reset triggered due to HA policy of Reset
• is 1000v worth it?
• 1000v cisco crash
• cdp hap reset
• reset triggered due to ha policy of reset n7k
• service: cdp hap reset
• reason reset triggered due to ha policy of reset
• virtual machine packet altering

Editor’s Note: If you haven’t already, check out the first installment in this–hopefully not ongoing–series at http://blog.packetqueue.net/asa-tu/

At approximately 1:58pm PST last Thursday the two edge ASA 5510 units at our corporate headquarters dropped off the network.  At the time I was in a different office up in Quebec, Canada and so delegated to one of the other engineers to work the problem with TAC and bring them back online.  That process took much longer than expected, and I won’t bore you with the details.  What I will bore you with, however, are a few observations I have now that we have more time and experience working with Cisco’s ASA product line:

• The ASA has some sort of systemic, though exceedingly rare, problem on 8.3(x) and newer code.
• Said problem causes the units to reboot and take out the system flash (disk0:) but not user flash (disk1:).
• The flash appears to be erased, but it is in fact the MBR that is gone, not the data (we used a hardware forensic disk analysis unit to verify this).
• Cisco doesn’t have enough data points yet to even acknowledge this is an issue.  I don’t believe they’re “hiding” a problem; I just don’t think enough people have experienced the particular set of circumstances that would cause this and subsequently reported back to Cisco.

My own suspicions about the root cause are below, though I’d welcome any additional thoughts from anyone with experiences in this area.  I should also point out that I have heard from at least two other people that they have experienced this exact problem.

• The behavior and crash lead me to believe that the ASA experiences, at the point of failure, the equivalent of a Windows “BSOD”.  This would point to either memory or motherboard itself as these are the primary hardware-based causes of this type of crash in any system.  Most other crashes can be recovered from and produce data.
• The ASA accesses the flash on initial load, but then runs from memory.  The flash cards in these units had trashed MBRs which leads me to believe that the ASA was touching the MBR at the time of the crash, which is inconsistent with what I know about how the ASA is supposed to operate.  It’s possible it was just accessing the flash to write a crash-dump and crashed partway through.  That makes some sense to me.
• All failures I have experienced and heard of from others have at least a couple of things in common:  They are all on 8.3(x) code.  They are all post user-upgraded to support 8.3(x).  This code required a memory and flash upgrade, and so you had to buy upgrades from Cisco and field-install them yourself.  These units were also all manufactured immediately following the Cisco manufacturing slowdown in 2008/2009 when lead times were running into the several months range.  This makes me a bit suspicious that quality control on either the memory or the units themselves could be to blame.  I’ve tried to verify with revision numbers, etc., but I haven’t been able to gather enough data from “out there” to settle on this as a cause.

I hope this helps someone out there, and I truly am interested in getting more information from anyone that has it.  Cisco is taking our units back, but pulling them aside before refurbishment so that their engineers can dissect the units.  If I find anything out from that I’ll post the findings here.

The configuration and build-out of the ASA 5510 units is as follows:

• 1 Gigabyte of memory, 512MB of system flash, 256MB of user flash.  IPS Module, Security-Plus, Botnet filter, AnyConnect Essentials, Mobile, etc. licenses.  Actually, just about every license is on board; these units are at this point maxed on everything.  Utilization is at a reasonable level still.
• Configuration includes use of multiple IPsec site-to-site VPNs, SSL VPN for all Mac, Linux, Windows, iPad and iPhone, sub-interfaces, stateful failover, both IPv4 and IPv6, OSPF with static redistribution, and full IPS functionality.

Incoming search terms for the article:

• configure disk1 asa5510

Why Bonjour Hates my Wireless Network

Many of you know my struggle as of late to integrate all of my recently acquired Apple devices into my existing network.  Many of you also know the frustration I’ve had with this process and have been innocent passers-by to my incessant twitter updates, rants and spontaneous bursts of misplaced anger.  Here then, is my brief explanation of what the problem is, and why I now—after too many rabbit-hole adventures to list—believe that I will not solve my problem without different equipment or a radical re-design of my network.

I should point out, by the way, that it’s not that I’m a masochist—really I’m not—but rather that in my studies I find it useful to have a lot of equipment lying around.  That equipment inevitably works its way into my home network, and after some time I have a large and sometimes convoluted structure in place.  In this case, however, the wireless is pretty far outside the scope of anything I’ll deal with on the CCIE Routing and Switching Lab Exam, and was brought in specifically to support some future upgrades to my home: wireless security, roaming VoIP phones, etc.  The irony, as my wife so perfectly pointed out the other evening, is that if we just had a “regular” little wireless router “like all the normal, non-computer geek people” our Apple devices would all work.

If you haven’t read my previous posting on Bonjour, that might provide some more background but isn’t, strictly speaking, necessary.  Some understanding of Bonjour might be helpful, however, so very quickly, here it is: Bonjour is Apple’s implementation of a service discovery protocol similar to Microsoft’s zero-conf.  It uses a couple of addresses to make things work, and it is the protocol behind Apple’s “everything just works” magic.  If you want more than that, Google can offer you much deeper explanations.

Bonjour uses two addresses, really, to do its work: 224.0.0.251 and 224.0.0.252, the latter of which is the “discovery” part of the protocol and the former where the action happens.  The astute among you will notice that these are both link-local addresses and so won’t be forwarded by layer-3 devices (even really, really broken ones) at all.  I had already been around the block with this once before, and so figured that because my wireless network was one broadcast domain (thought I smugly) everything would be all good.  I was wrong.

Now would be a good time to toss in a quick network diagram so that you can visualize what we’re talking about here.  The drawing below is just the wireless portion of my network as it applies to what we’re discussing in this article.  Rest assured, there is a lot more out there, but none of it is applicable to this situation.

As you can hopefully see, we have a 2811 ISR connected to a 2950 switch via 802.1q, and two 1142 APs connected at layer-2 to the switch.  What might not be as obvious at first is that the Wireless Lan Controller you see at the upper right of the diagram is a module sitting in the 2811 router.  This is where the heart of evil apparently lies, but more on that in a minute.  The access points are on VLAN 16, and get DHCP assignment from the 2811 along with option 43 and option 60 which are both necessary (despite what you may hear) to get the radios registered to the controller, at least in this configuration.  All VLANs are allowed everywhere (for testing) and no ACL/VACLs or any other security outside of standard wireless is applied.

Before anyone points out the obvious, by the way, I did reconfigure this arrangement to put the APs on the same VLAN as the WLC management interface, make that the native VLAN all the way through, and bridge the switch and router at layer 2 with BVI, just as a test to eliminate layer-3 boundaries.  While interesting to do, that didn’t solve the problem we’re having here.  In fact, I didn’t even notice the real problem location until I made this diagram (who would have thought?).

The WLC modules that plug into a router, while running the same software and otherwise operating almost identically, are different in at least one key respect from their stand-alone counterparts: they can’t communicate at layer-2 with the router.  A standard controller (say a 4400 series) can communicate at layer-2 with radios plugged in to access switches, thereby becoming the first layer-3 hop from the radios—even when different VLANs are assigned than management.  The integrated module, however, communicates with the host router across the backplane at layer-3.  Looking back at the diagram, you can clearly see that drawn out.  So no matter what I do with bridging from the radios, switch, router, etc., inevitably I’ll have layer three separation between the radios and the controller.

This is all well and good for most protocols, but not for link-local multicast.

I think I found every rabbit-hole possible to get lost down, and proceeded to do just that.  When I finally ran out of said holes to explore, kind folks on twitter that I respect and look up to sent me off in still more directions.  I tried, in no particular order:

(1)    Using Destination NAT to change the 224.0.0.251 and 252 addresses to multicast in the 239.x.x.x range

(2)    Using Destination NAT to change the 224.0.0.251 and 252 addresses to unicast

(3)    Using helper maps

(4)    Bridging everything under the sun to everything under the moon.  No love because the backplane can’t be bridged.

I was going to even try GRE tunnels, DCI, or any other type of tunnel to move Layer-2 over Layer-3.  At the end of the day, however, besides getting tired of the project, I decided that nothing was likely to work.  Why?  Because one of the first things a layer-3 device does when it receives a packet is to decrement the TTL.  So no matter what I do with NAT, or tunnels, or any other damned thing, the router will always decrement the TTL before it decides to pass the packet to some other service (like DNAT, GRE, whatever), thereby discarding the packet before it ever reaches those processes.

As far as I can tell today, this is unsolvable.  Apple hates me, and others like me.  Using a TTL of 1 as your method of locking down communications is pretty rock-solid from a DRM viewpoint, but also very inflexible and heavy-handed.  I’m going to put a portable 3560 in my entertainment center to support my DirecTV box, Apple TV and other entertainment devices so that they can share the iTunes library on my main computer, but I’m not happy about it.  I lose my shiny N-connected coolness, and my iPad won’t be able to control those devices.  In addition, I’ve had to hard-set my wife’s printer, since her Mac can’t find it any more.

The bottom line is that all of the auto-configuration magic that Apple devices can have has gone away in my current set up.  I could fix it by running a parallel wireless network using autonomous access points, or buy a cheap-o wireless router, but then I have the other problem where I lose visibility and control, just to make a quirky system work.  The only viable option, really, is to change out my WLC module for a stand-alone controller—which I may do at some point—but at this point I’m tired and may just move on, defeated.

Incoming search terms for the article:

• cisco wlc bonjour
• cisco wlc apple bonjour
• bonjour cisco
• bonjour cisco wireless controller
• bonjour wireless
• apple bonjour cisco wlc multicats
• apple tv cisco wlc
• bonjour wlc
• cisco wireless controller bonjour
• multicast bridging bonjour cisco

Passing the CCIE R&S Written (350-001)

I am proud to say that I have completed the first step on my journey to the CCIE Routing and Switching certification: namely, I passed the written qualification exam.  I obviously have a lot more work to do before attempting the lab later this year, but it is a good solid first step, and considering how long I’ve contemplated taking said step it is just good to be moving forward.

I’m not going to go into any details, talk about my score (it wasn’t perfect by any means) or really discuss anything that even smells like an NDA violation.  If that’s why your here and how you found this short blog posting, you’re in the wrong place.  I’ve worked far too hard for this to diminish either the work I’ve put in to get here, or the work that so many other full CCIEs have put in to attain their certifications.  The only way you get the digits is to pay your dues like everybody else.

That said, my brief observation for what it’s worth, is that this test was not entirely what I was expecting.  After years of taking different certification tests, including a variety of other offerings from Cisco, this test seemed a bit, well, tame.  Not easy, just more straight-forward question and answer.  That wasn’t really a positive or negative in my mind since I don’t really consider myself a “test” person and would have preferred a few more hands-on scenarios than I got.  But I suppose I’ll get more than my fill come lab-day.

The other interesting thing I noticed was the questions.  Some were almost cloyingly easy, while others a bit harder than I would have thought.  Possibly that is just a side effect of my studying habits.  In other words, the questions I found easy might be the same ones that trip someone else up.  When you’ve been at the books long enough, you lose a little perspective on these things.  None of the questions, however, were surprising in any way.  I think that the subject matter described on the blueprint, as well as some base-level networking knowledge that is just assumed was all covered in a way that you should expect of this level of testing.

The last thing I found different than some of the other tests I’ve taken is the increased reliance on “stacking” technologies.  In other words, you could see a question ostensibly focused on a particular technology, but with one or two other technologies represented in the question as well.  In particular, you would be required to understand not only all three technologies in the question, but also the subtle interactions that can happen as they work together.  My sense is that this is probably intended to be more “real world” representative, and in general I think it worked well.

All in all I think it was like a lot of Cisco tests: fair but difficult.  If you know what you’re doing you should pass, and if you don’t, well… take your score breakdown and hit the areas where you were weak.  Oh, and Cisco: please make your example diagrams easier to read!  I’m not so old that I need reading glasses, but my god some of those diagrams were bordering on illegible.  On at least a couple of occasions I had to squint, look sideways, and try to see… like one of those damned “dot” pictures where if you stare long enough you see a dolphin or some other randomly insipid thing you feel cheated for having expended the effort to see.

And now?  Off into some hundreds of hours of rack time.  Doh!

Random (and not so deep) Thoughts by Some Clown

I haven’t been writing a lot lately, mostly due to a combination of my work and study schedule.  I thought, however, that it would be useful to just toss down a few random thoughts on the proverbial paper to wrap up 2010.  I’ll try to keep it somewhat cohesive, but I can’t really guarantee anything.

Studying

Having made the decision last year at Cisco Live to finally buckle down and pursue the CCIE Routing and Switching certification, I have been as busy as you might imagine with studying.  As I’ve gone down this road I’ve noticed a couple of things:

(1) In the office I’m used to studying large white papers, documents, manuals, command references, etc., quickly to get to the answers I need for either deployment or break-fix.  This is not the best way to study for the CCIE qualification exam, however, as I tend to just as quickly forget that information past the point of it being immediately useful.  I’ve had to change my habits now to include taking notes, reviewing portions over and over, and cross-referencing with multiple sources.  Nothing earth shattering to be sure, but a change for me.

(2) As alluded to above, I do a lot of cross-referencing on my study material.  I have material from CCBootCamp that I consider to be my primary source (by virtue of being enrolled in the Cisco 360 program through them).  I have also been reading the CCIE Routing and Switching Certification Guide, 4th Edition, as well as the CCIE Routing and Switching Exam Quick Reference Sheets–both by Cisco Press.  I think it helps me quite a bit to read different perspectives on the same material; to see it put a different way on the page.  I have a Cisco Live Virtual account as well, and so have been pulling some presentations–notably on QoS–from that site.

(3) I have over 16 years of professional experience in this industry, and while I am by no means an expert, I am confident in things that I know.  To that end I would say that at some point in your studies you will be almost guaranteed to come across information, answers to practice questions, etc., that you just know are wrong.  I’ve had to learn not to be afraid to challenge my study material.  I don’t do it blindly, but I do go out and research in other sources to verify what I think I know.  I have found many instances of incorrect information in several sources–more often than not in the Cisco IOS example configurations.  Sometimes using commands that won’t work on that platform, other times referencing non-existent class-maps or access-control-lists.  Less often have I found blatantly incorrect explanations of how a thing works, but even there I have found a couple of examples.  I take this as a good sign, actually; it’s a sign that I am becoming more aware of the details of what I am studying.

Interesting Design Decisions

It always fascinates and bewilders me to see some of the design decisions that other engineers make when putting together a network.  Much of what we do is subjective, and even the most experienced experts disagree on a good many things.  With that said, certain things just don’t strike me as particularly useful and it’s my prerogative to complain about them.  My top complaints from recent experience, in no particular order are:

(1) My predecessor who built our main datacenter using 4503 switches exclusively: access, distribution and core (mostly, but we do use a collapsed core model).  The 4500 series is great but my general argument is that they’re under-powered, or at least under-featured for the core (Sup II-plus) and just a bit overpowered for the access layer.  We use PoE 1-Gig to every port in the building, but the access layer is still barely running (less than 1 percent utilization ever, on any metric).  I think someone got a deal or something.  We’re now replacing the core with a pair of 6506, 720 supervisor, 10-gig, etc.

(2) A main distribution point had a single 3845 with a 100-meg Internet connection, and two full DS3 links.  Considering the 3845 maxes out at 45 Meg of throughput, this seems a particularly egregious violation in my mind.  We’ve now moved that to a 3945, which if under full load is probably still a tad over-subscribed, but much better and the price was right.

(3) Who was it at Cisco that decided that the ASA-5510 would only have two Gig links available, and only with the right license?  Why only two?  Why not three or all five?  This might be a backplane issue, I don’t know, but it just bothers me.

(4) My own stupidity in setting up the aforementioned ASA-5510 pair (failover) with the inside and outside interfaces on the gig links, when I should have had the two trunk links that handle much more traffic on those interfaces.  This will be changed soon, but I should have done it right the first time.

In Conclusion

2010 has been a good year overall, with a lot of interesting projects, experiences, and solid learning had by all–or at least me.  I’m looking forward to 2011 and all of the continued successes and experiences to come.  I’d also like to give a special shout-out to all of my Twitter colleagues, friends, followers, and various clingers-on and lurkers.  I have found the Twitter community to be an invaluable source of support, wisdom, and occasionally respite from the rigors of the daily grind.  If you’re not on Twitter, I’d highly encourage you to give it a look.

Happy New Year everyone!

Incoming search terms for the article:

• ascolta cisco 360 complaints