Back from China
After roughly one week in Shanghai, China to set up a new site on our corporate network, it is painfully apparent that I need to get back into this writing business. The dates on my posts belie my weak attempts at covering up my laziness. That said, there were at least a couple of things of note worth using up a few words on.
Note One (where it really is someone else’s fault)
Due to an unfortunate series of poor decisions, poor project management, and a quite sudden and unreasonable expectation of delivery dates, we [IT] were forced to poach some bandwidth from a sister company of ours who had a slight excess in the manufacturing facility where our new office was to be set up. By slight, I mean exactly 256kb. For those of you not accustomed to seeing the abbreviation for kilo, well, you’re too young. More on that in a moment.
All of that aside, we engineered the circuit all the way from the provider’s edge in Shanghai, back to our facilities on the West Coast and verified traffic was flowing. Once we got to Shanghai and hooked up our router and started building out the network behind it, we noticed that we couldn’t move any traffic at all. With a quick extended ping using the inside network interface as the source, as well as some trace-routes from other places, we verified that the provider had neglected to put a route in the BGP tables for the new network. Thank God for 24-hour NOC support, and within 30 minutes that problem was resolved.
Note Two (where the author tries to check the turn-signal fluid)
As we moved on to creating and joining up a shiny new R2 Read-only Domain Controller (RODC) everything went off the rails. Timeouts galore. DNS wouldn’t resolve quickly enough to allow the new DC to join the forest. Off I go on a jolly search for default timeout settings, registry tweaks, offline methods to install a DC (ugly at best) and generally going further down the rabbit hole of complexity, even going so far as to direct another engineer working for me to prep for a call to Microsoft (never fun.)
Having already racked a lot of gear, we decided to call it a night and come back fresh in the morning. I always find it helpful to contemplate problems like this over a good single-malt Scotch. So I did, a few times, and that led to morning and a face-palm moment. To wit:
In the cab on the way to the office the next day it occurred to me that I should check the security on the firewalls back home. I knew I didn’t put any ACLs on that new link, knowing I’d be testing and I prefer to test in the absence of artificial problems, then crank down the screws once I’m confident of the design. I thought, however, that I had overlooked a NAT exemption or something else and decided to spend some quality time checking that portion of the infrastructure.
So, I got my coffee at the office (thankfully the office manager is a Kiwi who favors Starbucks) and started to look over the configuration of the firewalls. Right there was my face-palm moment: an ACL which, in some security-conscious delusion I had put on the link in question, allowing ICMP traffic and denying everything else. *GAAAAAH* Long story short, I changed the ACL and everything “magically” started working.
All I can say here is that it doesn’t matter who you are, or how much experience with troubleshooting you have, always start with the simple stuff first. Some of the best advice I ever got was from an instructor of mine who was fond of saying “be the packet.” By that he just meant that you have to start from the packet’s point of view and slowly work through everything that happens to said packet from beginning to end. Wiring, arping, routing, etc. whatever. Be the packet.
Also, don’t be afraid to admit mistakes. They will happen and hopefully other people around you can learn from them. At least after they’re done laughing. Which sometimes takes a while.
Note Three (hey you kids, get off my @#$ lawn)
I just wanted to take a quick moment to address the inevitable questions about our link at 256kb, and the musings that I obviously must have meant mb instead. Bandwidth is always seen as one of those more is better kind of things, and ignoring the temptation to toss out the standard bandwidth/delay screed here, let me just tell you that you can get by with less than you think. By the way, those of us who can remember when a 2400 baud modem was blow-your-hair-back fast (multiple lines of text at once!) tend to be more pragmatic about these things.
On that link we currently have a domain controller running, several workstations with email and our main corporate ERP software. We also, and this is what I really like, have two 7900-series IP phones running. These are both homed to our main CUCM, Unity and IPCC servers back in the U.S. and have excellent call quality. In fact, we tested a phone call (no local calling for these phones in China) from those phones in Shanghai, through the CUCM in the U.S., and back to a U.S. homed cell phone in Shanghai and got no discernable jitter.
Moral of the story? You can get by with less bandwidth than you think. Would I choose to? Hell no!