packetqueue.net

Musings on computer stuff, and things... and other stuff.

Cisco

ASA

ASA Upgrade to 9.0(2)

Read­ing Time: 4 min­utes

“In the­o­ry there is no dif­fer­ence between the­o­ry and prac­tice. In prac­tice there is.” Yogi Berra

Usu­al­ly, no mat­ter how much plan­ning, test­ing, think­ing about, or stalling you build in to an upgrade project, the bill comes due at the end and isn’t what you expect­ed. Maybe some­one ordered the lob­ster, or mul­ti­ple drams of John­ny Walk­er Blue, but either way you have a sit­u­a­tion to deal with… right now. How you deal with it deter­mines many things about your skills and your char­ac­ter, but that’s for anoth­er post as I’m going to try my best to keep this one short and to the point.

I recent­ly had the won­der­ful expe­ri­ence of upgrad­ing an Active/Passive failover pair of ASA units to the newest of the new code, 9.0(2) from 8.4.2(8). After the 8.3 ker­fuf­fle (NAT changes automag­i­cal­ly, any­one?) I was par­tic­u­lar­ly keen to not miss any pos­si­ble gotchas in this upgrade. I also sched­uled a larg­er than usu­al main­te­nance window–even though we did­n’t expect any downtime–just in case.

I should inter­ject here, for those of you aghast at the thought that we would pos­si­bly imple­ment new, some would say, bleed­ing edge code on pro­duc­tion sys­tems, a cou­ple of things:

  1. We always run bleed­ing edge code, usu­al­ly because we have need of the bleed­ing edge fea­tures the code brings. In this case, IPv6 fea­tures sore­ly lack­ing in pri­or code ver­sions
  2. We have adopt­ed a very aggres­sive IPv6 stance, as I have writ­ten about before, and we tend to find our aspi­ra­tions and designs are well out in front of sig­nif­i­cant por­tions of the code avail­able for our equip­ment
  3. Not­ing the pri­or two items again, I’ll also add that Fire­walls, in par­tic­u­lar, seem to have code that is months or years behind the route/switch world. That holds true across all ven­dor plat­forms. Why? I don’t know, but that’s anoth­er post.

With our need-for-upgrade bona fides estab­lished, I duti­ful­ly read the entire Release notes for the 9.0(x) ASA code and while I was excit­ed at many of the new features–mostly around IPv6–and dis­ap­point­ed at oth­ers (No OSPFv3 address fam­i­lies? Real­ly?) some­thing imme­di­ate­ly jumped out at me: Page 19 and its dis­turb­ing title, “ACL Migra­tion in Ver­sion 9.0”

Any time you see the word “migra­tion” in any doc­u­men­ta­tion refer­ring to an upgrade of pro­duc­tion code or con­fig­u­ra­tion, you know two things:

  1. It prob­a­bly hap­pens auto-mag­i­cal­ly, which is basi­cal­ly a syn­onym for “we’re going to bork your code but we’re only going to loose­ly tell you where, how, and why.”
  2. You’d bet­ter have good back­ups and be pre­pared, because a roll-back is like­ly going to be as painful as just plow­ing ahead.

To sum­ma­rize the bor­ing details for you, pri­or to the new code you had two cat­e­gories of Access Con­trol Lists (ACLs): those for IPv4 and those for IPv6. Inside each of those macro-lev­els you had the nor­mal stan­dard and extend­ed lists and what­ev­er oth­er fea­tures. You applied the IPv4 and the IPv6 access-lists to the inter­face in what­ev­er direc­tion and that was that. True to the dual-stack mod­el, you real­ly were run­ning two par­al­lel net­works and nev­er the ‘twain shall meet.

Dur­ing and after the upgrade to 9.0(x) ASA code, a cou­ple of things hap­pen:

  1. IPv6 stan­dard ACLs are no longer sup­port­ed, and any you have are migrat­ed to extend­ed ACLs.
  2. If IPv4 and IPv6 ACLs are applied in the same direc­tion, on the same inter­face they are merged.
  3. The new key­words any4 and any6 are added in place of the old any key­word.
  4. Sup­pos­ed­ly, if cer­tain con­di­tions are met (and they were in my case) your IPv4 and IPv6 ACLs should be merged into one (they were not).

While it is a bit scary to have any ven­dor automag­i­cal­ly migrat­ing por­tions of your con­fig­u­ra­tion to a new for­mat, it hap­pens and as long as they doc­u­ment well and you do your due dili­gence, things can work out just fine. Oth­er times they com­plete­ly go to hell because of an undoc­u­ment­ed fea­ture. This upgrade fell some­where in the mid­dle.

As it turns out, a crit­i­cal fact was left out of the doc­u­men­ta­tion. Name­ly, that all of your access-groups that had been applied in some direc­tion or anoth­er would now, quite frankly, not be applied to any­thing. In oth­er words, my fire­walls were now let­ting any­thing out of the net­work and noth­ing in. I quick­ly applied my new access-lists to the inter­faces a cou­ple of times before I dis­cov­ered that you can now only have one applied in any direc­tion (par for most IOS devices).

Since these were pro­duc­tion and I had some high­er risk on the IPv4 side (we have a lot of rules, and a default-block out­bound pol­i­cy) than the IPv6 side, I did the fol­low­ing:

  1. I blocked IPv6 in and out, then applied the IPv4 lists to the inter­faces in the cor­rect direc­tions.
  2. I hand migrat­ed (notepad is your friend) the IPv6 access rules into the IPv4 lists and brought IPv6 access back online.
  3. I then delet­ed the redun­dant (old) ACLs.

Every­thing came back, life was good, most­ly nobody noticed any­thing. What’s the lessons learned from this expe­ri­ence? Besides don’t upgrade ASAs? How about these:

  1. Always have a back­up of your con­fig­u­ra­tion, prefer­ably tak­en a few min­utes before you start the upgrade. In this case I did­n’t use the back­ups for more than a ref­er­ence, but they were avail­able if I had want­ed to roll back.
  2. Know your con­fig­u­ra­tion and your devices. This seems intu­itive, but a lot of peo­ple would have got­ten part way through this migra­tion, saw that their ACLs were borked, and been lost. If you’re going to live on the edge, at least have a hel­met.
  3. Read the doc­u­men­ta­tion. I did, and while it did­n’t direct­ly help, I at least knew ahead of time what was like­ly to break. I also knew once it broke what the like­ly prob­lem area was. To tie this into the CCIE Lab (back to study­ing, so it’s on my mind) it’s a bit like being able to look at a net­work dia­gram and instinc­tive­ly know where you’ll have prob­lems (two routers doing redis­tri­b­u­tion between EIGRP and OSPF, check).

At the end of the day, it all worked out for a vari­ety of rea­sons list­ed above.  Would I sug­gest any read­ers out there try this sort of “no net” upgrade to bleed­ing edge code?  Prob­a­bly not.  In my case, I’m a masochist it seems, and this is my ther­a­py.  Now on to my 6500 upgrade to 15.1(1)SY.  I’m sure I’ll be writ­ing about that not long from today.

Share

Cisco

Cisco Live 2013 Announcement

Read­ing Time: 1 minute

I just received this from one of my Linked-in groups and thought I’d share it:

Ear­ly Bird pric­ing dis­count for Orlan­do ends this month. Reg­is­ter now to save up to $300 http://bit.ly/2013Rg

Sched­uler opens 26 March. Search for the cours­es you want here http://bit.ly/WHC06q or go to the course list­ing http://bit.ly/15KpVhT and then vis­it the Event Cat­a­log for course details. NetVets can begin to sched­ule their cours­es on 19 March.

View our rebroad­casts of two very pop­u­lar IPv6 ses­sions from Cis­co Live and con­nect with the course speak­ers dur­ing the event on twit­ter. Find the ses­sion and chat details for the 21 and 28 March events here http://bit.ly/Chat13

We look for­ward to see­ing you in Orlan­do in June http://bit.ly/CLpage and con­nect­ing with you in Cis­co Live 365 http://bit.ly/VxZ5Um for thou­sands of on demand ses­sions all year, for free! New ses­sions have just been added from the Mel­bourne event.
Share

ASA

Policy-based VPN between Juniper SRX and Cisco ASA

Read­ing Time: 4 min­utes

One of the things that I am called upon to do fair­ly often in my cur­rent role is to con­fig­ure remote access VPN devices for some site or anoth­er.  Often these sites are tran­sient in nature, only stay­ing active for weeks or months at the longest before dis­ap­pear­ing forever–or at least until I don’t care any more.

Because I spend a fair amount of time set­ting these VPN tun­nels up, I have got­ten fair­ly good at the ins and outs of IPsec VPN tun­nel con­fig­u­ra­tion and trou­bleshoot­ing.  I was even begin­ning to think of myself as a bit of a VPN whis­per­er.  That was about to change, how­ev­er.

We use Cis­co’s ASA prod­uct line every­where for this type of thing, and as many of you no doubt know, if you use one ven­dor’s prod­uct for VPN tun­nels you are gen­er­al­ly in a good posi­tion.  You’ll like­ly still have prob­lems from time to time–just enough to keep you honest–but it works itself out.  This is the mod­el we’d been fol­low­ing for sev­er­al years.  Until now.

For rea­sons pass­ing inter­est here, we made the deci­sion to start explor­ing the use of Juniper’s SRX prod­uct line for our remote, tran­sient tun­nel des­ti­na­tions and small­er offices.  We were not–and are not–prepared to rip-and-replace our entire ASA installed base, though, so these Juniper devices would have to inte­grate with the exist­ing infra­struc­ture.  This, as they say, is when the prover­bial excre­ment hit the fan.

As any­one who deals in these things knows, mix­ing ven­dors on either side of a VPN tun­nel is gen­er­al­ly a recipe for trou­ble.  Depend­ing on ven­dors, your trou­bles range from the “few drinks after work” type to the “move to Mex­i­co and open a beach bar” kind.  I had heard from many peo­ple that I trust that the SRX to ASA con­fig­u­ra­tion was this lat­ter type.

Juniper SRX devices pre­fer a type of VPN tun­nel known as a route-based VPN.  I’m not going to go into specifics here, but suf­fice it to say it’s a tech­nique that makes sense and a lot of ven­dors work this way.  Cis­co’s ASA, on the oth­er hand, prefers a type of VPN tun­nel known as pol­i­cy-based.  Pol­i­cy-based VPNs have some lim­i­ta­tions and seem to be favored most­ly by Cis­co and any­one who wants to inte­grate with Cis­co.  I had to make things work with­out chang­ing much on the HQ side (where the Cor­po­rate ASA units sit) out­side of what we nor­mal­ly do, so that meant that the SRX at the remote site need­ed to be con­fig­ured in a pol­i­cy-based way.

Bring on the pain.

The process I fol­lowed went some­thing like this:

  1. Spend a cou­ple of days learn­ing the Juniper SRX syn­tax.  This part was actu­al­ly kind of fun.
  2. Spend 5 min­utes con­fig­ur­ing new tun­nel on cor­po­rate ASA.
  3. Spend 3+ days try­ing to get Juniper to talk to ASA.  Spend only slight­ly more time con­fig­ur­ing as bang­ing head on desk.
  4. Spend 5.5 hours spread over two more days with jTAC.  Bang head more while they can’t fig­ure it out either.
  5. Go home, relax, have eure­ka moment, race back to office, make one-line change, FIX EVERYTHING!
  6. Go back home and drink.

 

As it turns out, the prob­lem can best be described as some­thing that every Cis­co Engi­neer learns in infan­cy: Cis­co is con­sis­tent­ly incon­sis­tent.  For exam­ple, when the ASA refers to SHA, do you sup­pose it’s refer­ing to the old SHA‑0 or the SHA‑1 that cor­rect­ed a flaw in the old SHA‑0?  Dun­no.  Since they, in some places, only say “SHA” and in oth­ers “SHA‑1” it’s any­body’s guess, real­ly.

The main thing I re-learned from this expe­ri­ence is that the defaults–the lit­tle timers, iden­ti­ties, and var­i­ous oth­er lit­tle bits–that make up a suc­cess­ful nego­ti­a­tion for an IPsec tun­nel are dif­fer­ent across plat­forms.  Some­times you real­ly have to search to find what they’re called.  Some­times you have to bang your head on the desk for a few days.

One more thing: this post was typed up quick­ly, with no edit­ing, and way too much cof­fee.  If I’ve over­looked things, got­ten things wrong, or just con­fused you more, I apol­o­gize and I’ll try to come back and clean it up lat­er.  In the mean time, hope­ful­ly the dia­gram and con­fig­u­ra­tions below will help you in your own quest to get a pol­i­cy-based VPN con­fig­ured between the SRX and ASA.

Oh, and as soon as I recov­er from this expe­ri­ence I’ll build out a con­fig­u­ra­tion to do a route-based exam­ple, which looks to only require a cou­ple of changes on the SRX side and a tad more tweak­ing per­haps on the ASA side.

 

ASA CONFIGURATION (Sanitized):

crypto map outside_map 1 match address outside_cryptomap_3
crypto map outside_map 1 set connection-type bi-directional
crypto map outside_map 1 set peer 5.5.5.5
crypto map outside_map 1 set ikev1 phase1-mode main
crypto map outside_map 1 set ikev1 transform-set ESP-AES-256-SHA
crypto map outside_map 1 set reverse-route
crypto map outside_map interface outside
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto ipsec security-association replay window-size 64
crypto ipsec fragmentation before-encryption outside
crypto ipsec fragmentation before-encryption inside
crypto ipsec df-bit copy-df outside
crypto ipsec df-bit copy-df inside
crypto ikev1 enable outside
crypto ikev1 policy 4
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 28800

nat (inside,outside) source static CISCO_NETWORK CISCO_NETWORK destination static JUNIPER_NETWORK JUNIPER_NETWORK no-proxy-arp
object network CISCO_NETWORK
 subnet 10.0.0.0 255.255.0.0
 description Cisco Network
object network JUNIPER_NETWORK
 subnet 10.7.24.0 255.255.255.0
 description Juniper Network

JUNIPER CONFIGURATION (Sanitized):

proposal ike-policy1 {
 authentication-method pre-shared-keys;
 dh-group group2;
 authentication-algorithm sha1;
 encryption-algorithm aes-256-cbc;
 lifetime-seconds 28800;
}
policy ike-policy1 {
 mode main;
 proposals ike-policy1;
 pre-shared-key ascii-text "$%#@!!!"; ## SECRET-DATA
}
gateway ike-gate {
 ike-policy ike-policy1;
 address 5.5.5.5;
 local-identity inet 6.6.6.6;
 external-interface fe-0/0/0;
}
proposal IPSEC_PROPOSAL {
 protocol esp;
 authentication-algorithm hmac-sha1-96;
 encryption-algorithm aes-256-cbc;
 lifetime-seconds 28800;
}
policy IPSEC_POLICY {
 proposals IPSEC_PROPOSAL;
}
vpn ike-vpn {
 ike {
 gateway ike-gate;
 inactive: proxy-identity {
 local 10.7.24.0/24;
 remote 10.0.0.0/16;
 service any;
 }
 ipsec-policy IPSEC_POLICY;
 }
 establish-tunnels immediately;
}
source {
 rule-set trust-to-untrust {
 from zone trust;
 to zone untrust;
 rule nonat {
 match {
 source-address 10.7.24.0/24;
 destination-address 10.0.0.0/16;
 }
 then {
 source-nat {
 off;
 }
 }
 }
 rule ZZZ {
 match {
 source-address 0.0.0.0/0;
 destination-address 0.0.0.0/0;
 }
 then {
 source-nat {
 interface;
 }
 }
 }
 }
}
Share

Cisco

Cisco Live 2012 Schedule

Read­ing Time: 1 minute

Everyone–My sched­ule (as it stands cur­rent­ly) for Cis­co Live in San Diego this year is below.

Share

ASA

Windows 7 and IPv6

Read­ing Time: 5 min­utes

One of the projects that I began ear­ly last year is an IPv6 roll­out across our entire world­wide net­work.  Dur­ing the ini­tial heady days of the project, I man­aged to get the infra­struc­ture large­ly con­fig­ured in a dual-stack arrange­ment.  Then some­thing came up.  I can’t remem­ber exact­ly what, at this point, but some­thing did, and so the project sat.  And sat.  And sat some more.

Ear­li­er this week, I reviewed the appalling (lack of) progress so far and began mov­ing for­ward.  I had already com­plet­ed the acqui­si­tion of Provider Inde­pen­dent Address Space (PI), get­ting our upstream providers to adver­tise the routes, and the set­ting up of 90% of the infra­struc­ture (SVI address­ing, the dif­fer­ent IPv6 set­tings like IPv6 CEF and uni­cast-rout­ing, etc.)  Every­thing here worked as expect­ed.

I decid­ed to start with the infra­struc­ture more relat­ed to the client side, includ­ing servers of inter­est (DNS, AD, mail, etc.) to the inside, and the end-user work­sta­tions.  It should be not­ed here that we had IPv6 turned off on all work­sta­tions and servers, despite Microsoft­’s best prac­tices which state that IPv6 should be left on because sev­er­al ser­vices will break with it off.  Since these ser­vices (Home­groups, for one) don’t real­ly apply in a cor­po­rate envi­ron­ment, I did­n’t care too much what their best prac­tices rec­om­mend­ed.   The first step was to turn IPv6 on for a few test machines in the IT VLAN, and a cou­ple of the Active Direc­to­ry servers.

The good news is that the servers came up and were hap­py as soon as they got their address­es.  We’re going with a scheme of sta­t­ic assign­ment for servers, infra­struc­ture man­age­ment address­es, etc., so all I real­ly had to do was click the check­box for IPv6 (these are Win­dows 2008 R2 servers), assign address and gate­way, and voila, we had reach­a­bil­i­ty.  Next, I fixed up some DNS set­tings, added some AAAA records (the IPv6 equiv­a­lent of IPv4 A records), and we had DNS res­o­lu­tion on IPv6 work­ing as well.

Win­dows 7 would­n’t prove to be quite as easy, for a few rea­sons.  While the check­box­es and set­tings are osten­si­bly the same, some of the back-end code is obvi­ous­ly dif­fer­ent and needs some non-obvi­ous tweak­ing to get right.  I could­n’t for the life of me fig­ure out why I had no reach­a­bil­i­ty, when the con­fig­u­ra­tion should actu­al­ly be easy as pie (state­less auto configuration–just “turn on” IPv6 and walk away).  In fact, I end­ed up test­ing my Mac­book Pro and a Lin­ux machine, and both of them worked as expect­ed right out of the gate, so I knew this was a Win­dows 7 prob­lem.

The first thing that occurred to me after test­ing the oth­er non-Win­dows machines was domain poli­cies.  We have a lot of them, and you nev­er know what might be caus­ing issues.  I had one of my Serv­er guys check that, and then check a non-domain Win­dows 7 machine with the same image (Win­dows 7 Enter­prise), and he came up with no applic­a­ble pol­i­cy prob­lems and the same behav­ior.

Final­ly, after spend­ing too much time with Google, I man­aged to piece togeth­er an idea of what was hap­pen­ing.  As it turns out, the mech­a­nism by which Win­dows 7 puts togeth­er its inter­face iden­ti­fiers is to blame.  I found the applic­a­ble infor­ma­tion in a post­ing from Feb­ru­ary, 2010 on a site called itexpertvoice.com, linked to here.  It prob­a­bly won’t sur­prise any­one to find out that Microsoft screwed up imple­ment­ing some­thing they helped cre­ate (RFC 4941).  The com­mand you’ll want to run, from an ele­vat­ed com­mand prompt (or GPO, SCCM, etc.) is:

netsh interface ipv6 set global randomizeidentifiers=disabled

 

This takes you back to behav­ior sup­port­ed by stan­dards-com­pli­ant hard­ware, and allows your Win­dows 7 machine to actu­al­ly use IPv6.

Anoth­er com­mand you may want to use, and this is strict­ly a per­son­al and pol­i­cy choice, is:

netsh interface ipv6 set privacy disabled

 

This turns off pri­va­cy address­ing, an idea also described in RFC 4941.  Giv­en the glob­al­ly unique, routable nature of IPv6, I can see where some peo­ple might get squir­rel­ly, but I just don’t see the need.  Again, this is some­thing you’ll want to review for your own envi­ron­ment.

Anoth­er inter­est­ing prob­lem I found, and one more I had to solve before my Win­dows 7 machines were ready to be released back into the wild, revolves around voice.  Specif­i­cal­ly, voice VLANs.  In a voice-enabled net­work using Cis­co IP tele­pho­ny, each access port on a switch typ­i­cal­ly belongs to two VLANs: a voice VLAN and a data VLAN.  You plug a phone in, your com­put­er plugs in to a pass-through port on the phone, and each auto-mag­i­cal­ly gets an IP address in the cor­rect VLAN.  Except with IPv6.

I should say, how­ev­er, it did­n’t work that way in my envi­ron­ment.  What hap­pened here is that my work­sta­tion would actu­al­ly get an IP address in both the data and the voice VLANs.  And, due to the way pre­fix pol­i­cy works (more on that in a minute), would use the voice VLAN as the source address in traf­fic orig­i­nat­ing from the work­sta­tion.  The way I solved the prob­lem for now is to turn off IPv6 and take away the IPv6 address­ing from all of the voice VLANs (specif­i­cal­ly, on the SVIs), since I haven’t got­ten to the voice part of the IPv6 project any­way.

I don’t know what’s caus­ing the work­sta­tion to get assigned a voice address, but I’m going to con­tin­ue to look into it.  More than like­ly it has to do with State­less Auto con­fig­u­ra­tion and ND, but time will tell.  In the mean time, what’s this pre­fix pol­i­cy thing I men­tioned above?  Read about it in RFC 3484.

Put sim­ply, IPv6 pre­fix pol­i­cy exists because any giv­en inter­face in IPv6 is going to have sev­er­al address­es; every­thing from Uni­cast to link-local, mul­ti­cast, and maybe more.  So, if I’m a work­sta­tion and I want to source traf­fic out­bound on a giv­en inter­face, which address do I use?  Pre­fix pol­i­cy helps to solve that prob­lem by “rank­ing” dif­fer­ent address­es on an inter­face in a stan­dards-spe­cif­ic way.

After read­ing the RFC, open up an ele­vat­ed com­mand prompt on your Win­dows 7 machine and enter the fol­low­ing com­mand to see how your machine is set up:

netsh int ipv6 show prefix policies

 



and you’ll get some­thing that looks like so:




 


 


 


 


 


 


You can manip­u­late the order of things here, and add if need­ed, using the fol­low­ing com­mands (note these are two com­mands, but word-wrap is in effect here):


netsh int ipv6 add prefixpolicy=d2a:d00c:b678:ecfb::1/128


precedence 2 label=22


netsh int ipv6 add prefixpolicy=d2a:d00c::b678::/48


precedence 2 label=22


 


This gives you a match­ing pre­fix pair, and says that for all traf­fic des­tined for d2a:d00c:b678::/48 net­work, use the d2a:d00c:b678:ecfb::1/128 address on the inter­face as the source.  In my case, my voice VLAN (applic­a­ble hex­tet: 107) was show­ing up in the inter­face pre­fix list before my data VLAN (applic­a­ble hex­tet: 1ff).  I could have used the exam­ple above to set the cor­rect prece­dence, but this isn’t exact­ly scal­able to thou­sands of phones and work­sta­tions, all of which may have a dif­fer­ent mix of address­es.


So, we still have the voice VLAN issue to solve, but the Win­dows 7 machines can now talk com­fort­ably on the net­work to oth­er work­sta­tions and servers, includ­ing using SSH, DNS, RDP, and all oth­er major ser­vices.  A lot of soft­ware still isn’t IPv6 com­pat­i­ble, but that’s why we’re in a dual-stack world for the fore­see­able future.


Next chal­lenge, if any­one has sug­ges­tions?  Work­ing around some of the fea­ture non-par­i­ty issues most­ly in the area of secu­ri­ty.  For instance, the ASA line as of last check still does­n’t sup­port OSPFv3 or failover via IPv6, and no tools exist that I’m aware of for map­ping com­plex secu­ri­ty pol­i­cy from IPv4 to IPv6, which is why our IPv6 world here still remains sequestered to just our net­work.


Ide­al­ly we’ll have out­side traf­fic allowed and all of our pub­lic-fac­ing ser­vices run­ning in time for World IPv6 day this year, which hap­pens on June 6th.  Mark your cal­en­dars!


Share
 Cisco 
Cisco Live 2011 — CAE


Read­ing Time:  1 minute
		
		


Share