From: Somewhere Behind A Blinking Uplink

Dear Friend & Engineer,

A new engineer once took down an entire operation with one switchport.

And the worst part?

He was absolutely certain the outage had nothing to do with him.

This story doesn’t involve BGP.

There are no route reflectors, no ugly redistribution problem involving three routing protocols, and no mysterious firewall nobody remembers installing.

No.

This one started with a single switchport, one simple change, and an engineer who’d been on the job for about two weeks.

At the time, we were looking after the networks inside several commercial buildings.

Those buildings were part of a much larger operation where goods arrived, were registered, and then moved through different facilities.

Each building had its own network.

Computers, servers, monitoring systems, IP phones, switches, and uplinks connecting everything back to the main site.

Usually, either I went there myself or one of the more experienced IT engineers handled the work.

But they’d recently hired someone new.

He’d already passed his CCNA and, as he proudly told me, had started studying for his CCNP.

He was excited.

Confident.

And eager to prove he could handle real network work on his own.

Nothing wrong with that.

Every one of us has been that engineer.

Anyway, someone from the site called me and said they needed to connect a few computers and a server inside the equipment room.

They didn’t have another access switch available, but there were several unused ports on the main switch.

“Can we connect them directly to this switch?” they asked.

I said yes.

Then I gave the new engineer one warning.

“Don’t touch port 20 or anything above it.”

Those ports were being used as uplinks.

Now, should I have said, “Don’t touch ports 20 through 24”?

Probably.

Should I have written down the exact interface numbers?

Probably.

But at the time, I thought “port 20 and above” was clear enough.

So I asked him whether he knew what he needed to do.

“Yes,” he said. “There are two VLANs here. I just need to put the port in VLAN 20 or VLAN 30, depending on whether I connect the computer or the server.”

Good.

Pick an unused access port.

Put it in the correct VLAN.

Connect the device.

Five-minute job.

Or so I thought.

About ten minutes later…

The Entire Network Went Down.

The remote buildings disappeared from monitoring.

The phones stopped working.

The application the staff used was no longer available.

And the IT engineers at the other locations started calling, one after another.

They had no idea someone inside the equipment room had just changed a switchport.

All they knew was that everything had been working a few minutes earlier.

Now, almost nothing was.

I told them a new engineer had just made a change on the switch.

“Whatever happened,” I said, “start with that change.”

A few moments later, the new engineer called me.

“What did you do?” I asked.

“I did exactly what you told me,” he said. “It can’t be because of my change.”

Can’t be?

Before his change, everything was working.

Ten minutes after his change, everything was down.

That didn’t prove exactly what had happened.

But it gave us a very good place to start looking.

I told him, “You made a change, and then the network went down. So yes, we’re starting with what you did. Go back and check it.”

He insisted he’d only configured a normal access port.

Nothing complicated.

Nothing unusual.

Just a couple of switchport commands and the VLAN I’d mentioned.

So I asked him one question.

“Which port did you configure?”

“Port 24,” he said.

Port 24.

That was the main uplink.

It was carrying those VLANs from that switch toward the main building.

And it was one of the ports I’d specifically told him not to touch.

He’d taken an interface configured as a trunk and applied access-port commands to it.

Before his change, that interface had been carrying multiple VLANs between the switches.

After his change, it was no longer operating as the trunk the network depended on.

The VLANs stopped passing correctly.

Traffic started flooding through the network.

The switches struggled to process what was happening.

And the entire operation began disappearing from the network.

By then, I couldn’t SSH into the switch.

My remote access was gone.

The other buildings were still offline.

And when he looked at port 24, the light was blinking so fast it looked like the port was about to explode.

I told him to stop entering commands.

“First, pull the cable out of port 24.”

I didn’t mean shut down the interface.

I didn’t mean try another configuration.

I meant physically pull the cable out.

At that point, the switch CPU was probably already close to one hundred percent.

The last thing I wanted was a nervous engineer entering more commands while the switch was struggling to process everything happening around it.

He pulled the cable.

Then he connected his laptop locally and checked the interface.

And there it was.

GigabitEthernet1/0/24

The access-port commands he’d entered were sitting right there.

The interface that had been configured as a trunk now contained the configuration he’d added.

By then, all the confidence from ten minutes earlier had disappeared.

He was nervous.

And now he was worried the next command might make things even worse.

I told him to slow down.

“Look at every command you entered and put no in front of it.”

First, remove his changes.

Then restore:

switchport mode trunk

He removed the incorrect commands, restored the trunk configuration, and reconnected the uplink.

Then we waited.

One minute passed.

Then another.

A few devices began appearing in the monitoring system.

Then more.

Within about five minutes, the phones came back.

The remote buildings came online.

The application became available.

And the operation slowly returned to normal.

Problem solved.

But the story wasn’t over.

A few minutes later, his manager called me.

The new engineer was standing next to him.

“What happened?” the manager asked. “How did one small change create an outage like this in the middle of the day?”

Now, this wasn’t an ordinary office where one hour of downtime meant a few employees couldn’t check their email.

Goods were arriving constantly.

Large trucks would enter the facility, unload, and then leave.

But before a truck could enter, its information had to be registered in the system.

The driver carried a document that acted as an entry permit.

The staff had to verify that document, register the vehicle, and approve its entry.

Without the network, none of that could happen.

And without that process, the trucks couldn’t enter.

So they waited outside.

At first, there might be ten or twenty trucks.

But the line grew quickly.

Within an hour, there could be fifty large trucks parked one behind another.

Some of the roads around the facility were narrow.

When eighteen-wheelers parked along the side of those roads, they blocked half the lane.

A normal queue could suddenly stretch for one or two kilometers.

Keep the network down long enough, and that line could become several kilometers long.

At that point, the outage was no longer hiding inside an equipment room.

Everyone could see it.

The drivers could see it.

The managers could see it.

People passing through the area could see it.

Eventually, even senior officials could hear about it.

All because one engineer changed one switchport from a trunk to an access port.

The manager listened to my explanation.

Then he turned to the new engineer and said something I still remember.

“From now on, before you change an interface, look at it carefully. Check it two or three times. Run show cdp neighbors and see what’s connected to it. Make sure it’s not an uplink before you touch anything.”

He was right.

Had the engineer run:

show cdp neighbors

He would’ve immediately seen that port 24 was connected to another switch.

That one command would’ve told him everything he needed to know.

This wasn’t an unused access port.

It was an uplink.

And that’s exactly what I told them.

In fact, I still do the same thing myself.

Before I change an important interface, I don’t trust my memory.

I don’t glance at the port number once and immediately start entering commands.

First, I physically look at the cable.

Then I confirm the interface number.

I make sure GigabitEthernet1/0/24 isn’t GigabitEthernet1/0/20 or GigabitEthernet1/0/23.

Then I run:

show cdp neighbors

I check what’s connected to the interface.

After that, I run:

show running-config interface GigabitEthernet1/0/24

And I read the current configuration.

Is it an access port?

Is it a trunk?

Are there VLAN restrictions?

Is it part of a port channel?

Is that interface carrying something important?

Only after I know the answers do I make the change.

You may think that’s unnecessary.

After all, checking the same interface two or three times takes longer.

It may add an entire minute to a job that should take ten seconds.

But do you know what takes longer than one minute?

Trying to restore an entire network while fifty trucks are sitting outside because you couldn’t remember whether the uplink was port 23 or port 24.

The new engineer knew the commands.

He understood access VLANs.

He knew how to configure a switchport.

He’d passed his CCNA.

And he was studying for his CCNP.

The problem wasn’t that he didn’t know enough commands.

The problem was that he trusted the port number before he verified the port.

Typing the command takes five seconds.

Checking the interface first takes a minute.

That minute isn’t wasted time.

That minute is part of the job.

Ali

P.S. The next time you’re about to change a switchport, take the extra minute.

Look at the cable.

Confirm the interface number.

Run show cdp neighbors.

Read the current configuration.

It may feel unnecessary.

But somewhere outside that equipment room, there could be fifty trucks waiting to find out whether you skipped it.