For all of you paying any attention at all, I owe you an apology on the complete lack of writing the last week or so. Last week, however, a large pile of backordered Cisco gear showed up at the office and needed to be staged *immediately* or as close to that as could reasonably be expected. We’re in the throes of a complete infrastructure upgrade and everything sort of backed up unexpectedly with the recent delivery problems from Cisco. I would question how that becomes my problem, but after 16+ years doing this professionally, I already know the answer to that question. At any rate, look for a new post in my series on 802.1x in the next day or so.
802.1x for Wired Networks (Part 1)
Most of you are probably familiar with the IEEE 802.1x specification, at least in general terms, but are more than likely used to seeing it applied in the context of wireless networks and services. In this multi-part series, I’m going to explore the use of 802.1x as it applies to wired networks. In part I we’ll begin with what 802.1x is and what it isn’t, followed by some basic configuration adhering to Cisco’s current best practices recommendations. Then, in part II we’ll follow-up with a more complex configuration that you really, really don’t want to use but I put out as an academic example of what can be done if you like the kind of complexity that will guarantee you or your support staff never sleep again. We’ll also talk about what happens when you mix other security features with 802.1x.
Dot1x, as I’ll be referring to it from now on, is at its most fundamental level simply a method for verifying that a device connected to your network is who it appears to be. If a company-owned computer connects to the network, we let them have access to certain resources. If someone connects an unauthorized computer (personal laptop, etc.) to the network, they have no access to anything. This is the basic idea behind dot1x when we’re talking about wired networks.
Dot1x is not encryption, and doesn’t provide any kind of privacy or anti-replay protection like IPsec. It isn’t intended for that. Think of dot1x as login security for your switch ports. If your switch ports are connected through to wall jacks, with no authentication, then rogue machines have at least limited (layer 1 and 2) access to, at minimum, the VLAN or network that switch port is a part of. Is that enough to cause damage to your network? You’ll have to decide for yourself based on your own appetite for risk and internal security policies.
So, what do we do to make it all work? I’m glad you asked. You’ll need to do four things, of which I’ll cover the two that happen on the Cisco side:
(1) Configure your switch to act as an authenticator
(2) Configure individual switch ports to use dot1x
(3) Configure your Radius Server to provide authentication and authorization to clients
(4) Turn on dot1x authentication on your clients (Windows, Mac, Linux, etc.)
The first step is fairly straight forward, and that is to tell your switch you want to do dot1x authentication. In order for this to work you have to have AAA enabled which is going to change your entire login method, so definitely play around with in a lab if you’re not familiar with it already. So, the commands we want look like this, entered from Global Config mode:
! Turn on aaa authentication
aaa new-model
! Define our Radius server, port, and shared secret (must match
! with our server setup)
radius-server host 10.0.0.1 auth-port 1812 key abc123
! Here we tell the switch that our default authentication method is to
! use the Radius host we defined above
aaa authentication dot1x default group radius
! Enable dot1x on the switch
dot1x system-auth-control
Now that we have the switch set to authenticate clients to our Radius (Remote Authentication Dial-in User Server) Server, we have to make sure the Radius server has some policies defined. This is beyond the scope of what I’m going to cover in this post, though if I get enough comments requesting it I can cover that portion in another post. Basically you’re just telling your Radius server (typically IAS or NPS on some version of Windows Server) that the switch will be contacting it for client authentication, and what that client can and can’t do once authenticated (the authorization piece.)
The next step is to configure your clients to use dot1x authentication, and this is different for each client OS you use. Windows clients need to have a service turned on which is not on by default, whereas Macintosh has the service turned on but not configured. For Linux and all manner of mobile devices, there are as many options as there are devices, so I’ll leave that as an exercise for the reader. As above with the Server section, if I get enough comments I can cover the Windows and Macintosh configuration piece in another post. Mobile devices and Linux I have limited experience with as pertains to dot1x setup.
The last step in this whole business is to turn on dot1x authentication on a per port or port-range basis. Keep in mind that you’ll want to test this before turning on carte-blanche as I’ve seen a lot of misconfiguration in the initial setup and the last thing you want is a horde of screaming users looking for your head on a stick. Therefore, to turn on one port, presumably for testing, we do like so from interface configuration mode:
dot1x port-control auto
! kind of anti-climactic isn’t it
That is the basic configuration of dot1x on a Cisco switch. In part II of this series, I’ll explore some more complex configurations, as well as some less-than-ideal ramifications of combining other security features with dot1x. As with anything you configure in the realm of security in particular, it is always a trade-off between security and usability and dot1x is no exception. Test, test, and test some more.
TELCO FAIL
Last Thursday afternoon, at approximately 2:25pm, there was a loud sucking sound that can only be heard by network engineers conditioned to expect bad, ugly things to happen at inopportune times, and all upstream connectivity to our corporate office died.
*Ka-phoot*
Predictably, IT was immediately assisted by many, many helpful people wandering by our area, sending emails, making phone calls, or stopping us in the hall to ask if we knew that the network was down. Usually in these situations the first couple of people get a good explanation of what we think the problem is, and what an ETA might be. After the 10th person, however, my responses tend to devolve a bit and I either end up giving a curt one-word answer, or feigning shock and amazement.
I should explain here that the way the architecture of our network works, we have our IP provider, SIP Trunks, Point-to-Point circuits, VPN end-points, and all of our external-facing servers in a very robust telecom hotel–The Westin Building, for those keeping score–in downtown Seattle. From there, we move everything over our DS3 to our corporate headquarters not far from Seattle. We also have many other dedicated circuits, IPsec tunnels, and assorted ballyhoo to other locations around the world, but for discussion here just keep in mind the three locations I’ve described.
So the DS3 that is our lifeline was down. It was after hours in our Canadian location so with any luck nobody would notice all night–they use a lot of critical services across another DS3, but that also routes through Seattle first. Additionally, it was a particularly nice day in Seattle (rare) and a lot of people were already out of the office when this link went down. Hopefully we could file a trouble ticket and get this resolved fairly quickly.
Within just a few minutes of filing said trouble ticket, I had a representative of the provisioning telecom on the line who said that, yes, they saw a problem and would be dispatching technicians. There were some other calls following that, but the short version is that by 5:30pm “everything was fixed” according to the telecom and would we please verify so they could close the ticket. Unfortunately, the problem was not fixed.
Now the fun began. To appease the telecom representative, I accepted the possibility that my DS3 controller card had coincidentally died or locked the circuit or some other bunch of weird pseudo-engineer guessing from the telecom representative. This meant I had to drive to our data center in Seattle, in rush hour traffic, to personally kick the offending router in the teeth.
After an hour or so of typically nasty Seattle rush-hour traffic I arrived at the datacenter and began testing. Our DS3 controller was showing AIP on the line, so more technicians were dispatched to find the offending problem. Meanwhile, I wandered over to the Icon Grill to get some dinner and an après-ski beverage or two.
Fast forward a few hours and the AIP condition on the DS3 controller was gone, but I now had an interface status of “up up (looped)” which is less than ideal, shall we say. I decided at this point to cut my losses and head home and possibly get some sleep while the telecom engineers and their cohort tried to figure out how this might be my fault.
With some three hours of sleep or so, I woke up at 5am and started looking at all of my emails, listening to all of my voicemails, and generally cursing anyone within earshot–mostly consisting of the cats–as my wife was still asleep. At this point I got on a conference bridge with the President of the telecom broker we use and together we managed to drag a rep in from the provisioning company who could then drag in as many engineers as needed to get the problem solved. Not, however, before I was rather pointedly told by said provisioning woman that I would have to pay for all of this cost since the problem was “obviously with my equipment, since her software showed no loops in the circuit.”
Once the engineers started hooking up testers to the circuit–physically this time–they could see a loop, but at the Seattle side (the side reporting the loop.) Another engineer saw a loop on the headquarters side, and still a third saw no loop at all. As it turns out, the circuit was provisioned by company “A” who then handed off to company “B” and finally to company “C” who terminated the circuit at the demarcation point at our headquarters. All for less than 20 miles, but I digress. Finally we all agreed to have Company “C” come onsite, interrupt the circuit physically at the demarcation equipment and look back down the link to see what he could see. As a precaution at this point, and tired of being blamed for ridiculous things, I and my staff physically powered down our routers on either side of the link. Since the loop stayed, that was the last time I had anyone point the finger my way. Small miracles and all of that.
Once the rep from Company “C” got onsite and interrupted the circuit for tests, he was still seeing “all green” down the line. Since the other engineers monitoring were still seeing a loop, they asked him to shut down the circuit. He did, and they still saw a loop. This was one of those “Aha” moments for all of us except the engineer from Company “C” who just couldn’t figure out what the problem might be. All of us suspected that the loop was between the Fujitsu OC‑3 box at our Demarc, and the upstream OC-48 Fujitsu Mux a couple of miles away and we finally convinced this guy to go check out the OC-48. Sure enough, a few minutes after he left our circuit came back on again. And we all rejoiced, and ate Robin’s Minstrels.
At the end of the day, we ended up with just short of 24 hours of downtime, for a DS‑3 from a major telecom provider that everyone here would recognize; 23 hours and 5 minutes, to be exact. So what was the problem, and the solution? Any telecom guys want to stop reading here and take a guess?
As it turns out, the original cause of our link going down was this same engineer pulling the circuit by mistake. When the trouble ticket was originally filed, he rushed out and “fixed” his mistake. But, what he hadn’t noticed the first time is two critical things:
(1) The circuit had failed over to the protect pair. DS3 circuits use one pair of fiber for the normally used (or working) circuit, and a separate fiber pair for the fail-over (or protect) circuit.
(2) The protect pair at the OC‑3 box at the demarcation point hadn’t ever been installed.
For lessons learned here, the main thing that comes to me is that we absolutely have to find a way to get true redundancy on this link, even if it means connecting our own strings between tin-cans. I should explain, by the way, that redundancy to this headquarters building is very difficult due to location: the last mile provider is the same no matter who we use to provision the circuit. In addition, with one major fiber loop in the area, even if we could get redundancy on the last mile we would still be at the mercy of that loop. We are at this point, after this incident, looking at a fixed LoS wireless option that has recently become available. Apparently we can get at least 20Mb/s although I haven’t heard any claims on the latency, so we’ll see.
I’m also shocked and appalled that three major telecoms, all working in concert, took almost a full day to run this problem to ground. I’m probably naive, but I expect more. The only saving grace in all of this is the level of professionalism and support I received from the telecom brokers we use. They were absolutely on top of this from the beginning, shepherded the whole process along, even facilitating communications between the players with their own conference bridge for the better part of a day. If anyone needs any telecom services brokered, anywhere in the world I’m told, contact Rick Crabbe at Threshold Communications.
With this summation done, my venting complete, and everything right with the world, I’m off for a beverage.
2960‑S 1st Impressions
Just noticing some differences in this 2960‑S model switch; things I haven’t seen before in this product line. One thing is the console port is now two ports: traditional and a small mini-USB. Also, there is a classic USB plug onboard these now too, presumably for IOS updates and whatnot. These are also stackable, which makes sense given Cisco’s new direction in their SWITCH course training material eschewing Spanning-Tree and trying to push Layer‑3 everywhere.
1st boot-up below, more impressions later.
Using driver version 1 for media type 1
Base ethernet MAC Address: a8:b1:d4:64:8f:00
Xmodem file system is available.
The password-recovery mechanism is enabled.
Initializing Flash…
mifs[2]: 0 files, 1 directories
mifs[2]: Total bytes : 1806336
mifs[2]: Bytes used : 1024
mifs[2]: Bytes available : 1805312
mifs[2]: mifs fsck took 1 seconds.
mifs[3]: 0 files, 1 directories
mifs[3]: Total bytes : 3870720
mifs[3]: Bytes used : 1024
mifs[3]: Bytes available : 3869696
mifs[3]: mifs fsck took 0 seconds.
mifs[4]: 5 files, 1 directories
mifs[4]: Total bytes : 258048
mifs[4]: Bytes used : 9216
mifs[4]: Bytes available : 248832
mifs[4]: mifs fsck took 0 seconds.
mifs[5]: 5 files, 1 directories
mifs[5]: Total bytes : 258048
mifs[5]: Bytes used : 9216
mifs[5]: Bytes available : 248832
mifs[5]: mifs fsck took 1 seconds.
mifs[6]: 559 files, 19 directories
mifs[6]: Total bytes : 57931776
mifs[6]: Bytes used : 14971392
mifs[6]: Bytes available : 42960384
mifs[6]: mifs fsck took 14 seconds.
…done Initializing Flash.
done.
Loading “flash:/c2960s-universalk9-mz.122–53.SE2/c2960s-universalk9-mz.122–53.SE2.bin”…@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File “flash:/c2960s-universalk9-mz.122–53.SE2/c2960s-universalk9-mz.122–53.SE2.bin” uncompressed and installed, entry point: 0x3000
executing…
Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
© of the Commercial Computer Software — Restricted
Rights clause at FAR sec. 52.227–19 and subparagraph
© (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227‑7013.
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134–1706
Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9‑M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright © 1986–2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 06:08 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x01B00000
Initializing flashfs…
Using driver version 1 for media type 1
mifs[3]: 0 files, 1 directories
mifs[3]: Total bytes : 1806336
mifs[3]: Bytes used : 1024
mifs[3]: Bytes available : 1805312
mifs[3]: mifs fsck took 0 seconds.
mifs[3]: Initialization complete.
mifs[4]: 0 files, 1 directories
mifs[4]: Total bytes : 3870720
mifs[4]: Bytes used : 1024
mifs[4]: Bytes available : 3869696
mifs[4]: mifs fsck took 1 seconds.
mifs[4]: Initialization complete.
mifs[5]: 5 files, 1 directories
mifs[5]: Total bytes : 258048
mifs[5]: Bytes used : 9216
mifs[5]: Bytes available : 248832
mifs[5]: mifs fsck took 0 seconds.
mifs[5]: Initialization complete.
mifs[6]: 5 files, 1 directories
mifs[6]: Total bytes : 258048
mifs[6]: Bytes used : 9216
mifs[6]: Bytes available : 248832
mifs[6]: mifs fsck took 0 seconds.
mifs[6]: Initialization complete.
mifs[7]: 559 files, 19 directories
mifs[7]: Total bytes : 57931776
mifs[7]: Bytes used : 14971392
mifs[7]: Bytes available : 42960384
mifs[7]: mifs fsck took 1 seconds.
mifs[7]: Initialization complete.
…done Initializing flashfs.
POST: MA BIST : Begin
FC 1 MBIST Test Passed.
DP Sg1 MBIST Test Passed.
DP Xg1 MBIST Test Passed.
NI 1 MBIST Test Passed.
FC 0 MBIST Test Passed.
DP Sg0 MBIST Test Passed.
DP Xg0 MBIST Test Passed.
NI 0 MBIST Test Passed.
UPB MBIST Test Passed.
POST: MA BIST : End, Status Passed
POST: TCAM BIST : Begin
POST: TCAM BIST : End, Status Passed
front_end/ (directory)
extracting front_end/fe_type_4 (78476 bytes)
extracting front_end/front_end_ucode_info (43 bytes)
extracting ucode_info (77 bytes)
Waiting for Stack Master Election…
POST: Thermal, Fan Tests : Begin
POST: Thermal, Fan Tests : End, Status Passed
POST: PortASIC Stack Port Loopback Tests : Begin
POST: PortASIC Stack Port Loopback Tests : End, Status Passed
POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed
POST: EMAC Loopback Tests : Begin
POST: EMAC Loopback Tests : End, Status Passed
Election Complete
Switch 1 booting as Master
Waiting for Port download…Complete
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
[email protected]
cisco WS-C2960S-48TS‑L (PowerPC) processor (revision A0) with 131072K bytes of memory.
Processor board ID FOC1425W1J7
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
52 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : A8:B1:D4:64:8F:00
Motherboard assembly number : 73–11909-05
Power supply part number : 341‑0327-01
Motherboard serial number : FOC14251HQ7
Power supply serial number : DCA1413P1BF
Model revision number : A0
Motherboard revision number : A0
Model number : WS-C2960S-48TS‑L
Daughterboard assembly number : 73–11933-04
Daughterboard serial number : FOC14232S4V
System serial number : FOC1425W1J7
Top Assembly Part Number : 800–30950-01
Top Assembly Revision Number : A0
Version ID : V01
CLEI Code Number : COMGF00ARA
Daughterboard revision number : A0
Hardware Board Revision Number : 0x01
Switch Ports Model SW Version SW Image
—— —– —– ———- ———-
* 1 52 WS-C2960S-48TS‑L 12.2(53)SE2 C2960S-UNIVERSALK9‑M
Putty
When I think of all of the tools I use on a regular basis to do my job more effectively, there are a handful that stand out as being the most useful of all. As I get the chance, I plan on writing a least a short homage to each one in turn. In my line of work, it is much more common to notice the things that break, or go wrong, or just don’t work as advertised; it is much less common to appreciate the things that just work.
With that in mind, it is only fitting that I start with one of the most useful and reliable programs I have ever used: Putty. Most of you are probably familiar with this little gem, but for those of you who aren’t: a little information is in order.
Putty is a combination SSH, Telnet, and Rlogin client, as well as terminal emulation software. I first discovered this program at some point back when I started hating–with a blind rage–Microsoft’s built-in Hyperterminal application. I soon had switched to Putty full-time, and haven’t ever had the urge to even investigate other options. I don’t know if the writers of Putty are rich, but if they aren’t they damn well deserve to be.
To wit:
I use Putty daily for doing all of my Cisco console work, as well as for all SSH connections to both Cisco and Unix devices. On the Cisco side, it just works nicely allowing for a variety of customizations as well as easy cutting-and-pasting of code to and from Notepad or whatever you use (Complex ACL editing, etc.) On the *nix side, one of the nicest features is the X‑11 forwarding which allows you to tunnel back X11 applications, via SSH, to your local client (assuming you are running *nix locally, or have an X11 window manager of some sort running. We happen to use Exceed, but there are free versions available.)
The amount of customization you can do to Putty, from auto-magical window labeling, to key-strokes and shortcuts, to saved session information, etc. is a beautiful thing. At home, for lab work, I have settings saved for all of my console-server connections so that I just have to “point-and-click” to open any of my myriad devices.
I could probably go on and on, but suffice it to say that this little gem of a program should be in your arsenal if it isn’t already. It is one of the first programs I install on any machine from which I plan on doing serious work.
Putty can be found at: Putty Downloads Page and is well worth a look.
ASA TU
You ever have one of those weeks where everything that can go wrong, does? And even things that can’t go wrong, still do? Last week was that week for me. But more on that later.
I’m finally relaxing with a beer in my home office, Friday afternoon, after said hell-week, when suddenly I notice that my desk phone has mysteriously powered off. Beyond the visual cue of no screen display, and a nagging suspicion that something was still not right in the world, I also heard it when it shut down (The Cisco 7940 models in particular seem to make some noise when turning on and off.) Since this phone is using PoE from a Cisco ASA 5505, I glanced over at the 5505 to see what might be causing all of the unhappiness. Immediately I noticed that something wasn’t right, as the status light was orange for a second, then the whole unit rebooted. At that point the display lights go all green, then amber, then another reboot… ad naseum.
What the @#$!?
Trying to log in via either SSH or ASDM yields no love at all, so I hook up a console cable. There it is: the ASA apparently doesn’t have any OS to boot.
Again with the @#$!?
So, I take the cover off and pull out the *cough* expensive-as-hell *cough* flash card to double-check things on desktop computer. Sure enough, the computer reports that the flash card is not formatted. So I formatted it, reinstalled the OS and license information from–wait for it–the backup I had made recently. At this point I could have used this as another learning experience and re-configured the unit from scratch, just to test myself, but at the time I was trying to get some movie tickets purchased and had just gotten done with a very, very tiring week. Not surprisingly then, I took the easy way out and restored the configuration as well.
After a quick reinstall, the unit came back up and everything is fine.
My main lesson learned here–and I’m wondering if this is something that has happened to other people, or happens frequently with the ASA units–is that the Cisco ASAs seem to occasionally wipe out the installed flash card (cards plural if you have 5510s or bigger.) Either that, or the ridiculously expensive (like $300 or something) Cisco-branded 512mb flash cards are flakey as hell. I don’t tend to go with that last option, however, simply because when flash cards go bad they’re not usually amenable to being re-formatted and working properly again: it’s usually game-over, buy another one.
So, another random problem at least solved in the short-term. I’ll keep everyone posted on whether or not this happens again. I haven’t seen anything indicating that this is a known issue, but admittedly I haven’t really been looking. Auto-magically wiping out the entire boot OS, configuration, licenses, etc. would seem to be a fairly non-useful type of bug to have–even by, say, Microsoft standards… let alone Cisco.
So, has anyone seen this before?