Last month a buddy of mine called me in a mild panic. He'd been doing a routine audit of his colo cage and found a small 1U box wedged between two of his servers. Eight cores by the looks of it, no asset tag, no documentation, and the kicker — it wasn't responding to ping and didn't show up in his ARP table.
Welcome to one of the more unsettling problems in infrastructure work. Someone or something put hardware in your rack, and now you need to figure out what it is, what it's doing, and whether it's a threat or just forgotten gear from a contractor who left two years ago.
I've helped track down three of these over the years. The process is mostly the same every time, and once you've done it once, the panic fades. Here's the playbook.
Why a Device Can Be "Invisible"
Before the witch-hunt, understand what you're looking at. A device that won't respond to ping and has no visible MAC isn't necessarily malicious — it might just be configured to stay quiet. Common reasons:
- It's not connected to your L2 network at all. Maybe it's on a separate VLAN, a management port you forgot about, or it's getting power but no data.
- It's a passive tap or span device. Network taps don't have IPs by design. They just mirror traffic. Vendors sell these for legitimate monitoring, but they're also a favorite tool for someone who wants to read your traffic quietly.
- It's listening only. A box configured with no ARP responses, ICMP dropped at the firewall, and outbound-only traffic will look dead to most scans.
- It's a forgotten KVM, IPMI gateway, or out-of-band management thing. This is the most common answer, and the most boring one.
The goal of the investigation is to move from "unknown" to "identified" before you make assumptions about intent.
Step 1: Find Its Network Footprint Without Touching It
Don't unplug it yet. If it's logging anything or has a heartbeat to somewhere, pulling the cable will tip off whoever owns it. Start passive.
First, dump every MAC address your switches have seen recently. If you're running anything sane on the switch side, this is a one-liner. From a Linux box on the same segment, you can also use arp-scan to actively probe — but again, that's noisy.
Go to the switch the device is plugged into and check the MAC address table for that specific port:
# On most Cisco-ish switches
show mac address-table interface GigabitEthernet0/24
# On Arista / generic Linux switches with FRR or similar
bridge fdb show | grep <port>
If nothing shows up, the device truly hasn't sent a frame. That's rare — even a quiet device usually leaks something during boot. Wait 24 hours and check again before assuming it's L1-only.
If you get a MAC, look up the OUI (the first 3 bytes). The IEEE maintains a public registry of vendor prefixes. There are dozens of CLI tools that do this offline, but you can also pipe it through a local lookup file:
# Quick OUI lookup using a local copy of the IEEE registry
mac="00:1B:21:AB:CD:EF"
oui=$(echo $mac | cut -d: -f1-3 | tr -d ':' | tr a-f A-F)
grep -i "^$oui" /usr/share/ieee-data/oui.txt
# Outputs the vendor — Intel, Supermicro, some random IoT shop, etc.
A legitimate server vendor MAC narrows things down fast. An obscure one is a red flag worth chasing.
Step 2: Find the Physical Switch Port
This is where people waste hours. They have a mystery box and a rack of switches and start tracing cables by hand. Don't. If your switches support LLDP or CDP, the device is probably announcing itself even if it won't talk IP.
# LLDP neighbors from a Linux host running lldpd
lldpcli show neighbors
# From a switch, ask which port a given MAC lives on
show mac address-table address <MAC>
If LLDP is silent, fall back to the MAC table walk. Find the MAC, find the port, label the cable. If the MAC table is empty because the box truly never transmits, your last resort is a port-by-port disconnect test — but do that during a maintenance window, not at 2pm on a Tuesday. Ask me how I learned that.
Step 3: Identify It Without Logging In
Once you've isolated the port, mirror its traffic to a capture host. Even a "silent" device usually does something — DHCP discover, mDNS, NTP, an outbound TLS connection to a vendor cloud. Anything.
# Capture for 10 minutes on the mirrored port and write to a file
sudo tcpdump -i eth1 -w mystery.pcap -G 600 -W 1
# Then look at who it tried to talk to
tshark -r mystery.pcap -q -z conv,ip
A destination IP is often enough. Reverse DNS, WHOIS, and the SNI from any TLS handshake usually tell you the vendor in under a minute. I've identified an "unknown" box as a cloud-managed UPS this way. Embarrassing but harmless.
If it's making no outbound connections at all and accepting no inbound, it's either a hardware tap (read-only by design) or it's broken. Treat hardware taps as a security incident until proven otherwise.
Step 4: Decide What To Do
If you identify the device and it has a legitimate owner (a contractor, a previous admin, a monitoring vendor someone forgot to document), label it and update your asset inventory. If you can't identify it, or the answer is "I have no idea why this is here," pull it and bag it. Don't power it on outside your network. Whoever owns it will surface within a week, and if nobody surfaces, you just solved the problem.
The rule I follow: anything in my rack that I can't tie to a ticket, a vendor, or a person within an hour gets unplugged.
Prevention: Make This Boring Next Time
The reason this is scary the first time is that you don't have the data. Fix that.
- Enable port security on access ports. A simple sticky-MAC config with a one-MAC limit means a new device on a port goes into err-disable instead of silently joining.
- Run 802.1X if you can. Yes, the setup is annoying. Yes, it works. Devices that can't authenticate get dropped onto a guarantine VLAN and you get an alert.
- Inventory by switch port, not by sticker. Every port should map to a known device in a spreadsheet, a Netbox instance, or whatever you use. Drift means a human walks the rack.
- Alert on new MACs. A nightly diff of the switch MAC table against your inventory catches 90% of "how did this get here" moments before they become incidents.
- Lock the cage. This is the most important one and the one people skip. Every rogue-device story I know ends with "...and the cage door was unlocked."
Most mystery devices turn out to be boring. A forgotten serial console server, an IPMI module someone added during a deployment three years ago, a cloud-managed PDU. But the few times it isn't boring, you want the tooling and the process already in place. Don't wait until you're holding an unknown box at midnight to figure out how your MAC tables work.
