So last night, I got it in my head that I would redesign my colo network to eliminate the private "trusted" network assigned to all VMs. Oh, except the two that aren't on my primary VM box but are in the same data center. That created some annoyances in that those VMs were not fully integrated into the network. Getting them connected to the private net would require a whole lot of headache that just wasn't worth it. So... I removed that network from the machines that have public IPs and then switched the gateways on those machines to talk to the router/vpn endpoint/nat box that was the gateway between that private network and the public world. As the VMs on the primary host were physically attached to this router... no big deal.

...and then the murders began.

VPN between the house and the colo worked without issue. All that NATting that needed to happen Just Plain Worked. My machines could talk to the world without issue. Everything was great. Just one problem... I forgot to check inbound. So I make it to the office today and wait a minute... why are all my ssh sessions to my machines dead, and why can't it connect? Turns out I never cleaned up the NAT rules on the router/VPN/NAT device so now everything was dead except for the thing I was monitoring i NAGIOS. Back in the day it was a special NAT setup, which is why it was specifically monitored.

With the rest of the changes I went ahead and cleaned up more of the legacy junk. There is now a clean physical separation between the public and private networks. The VPN connections work as they should. The one remaining service that fully sits behind NAT works as it should as do the non-NAT services. I even labeled the ports on the router and the host.

Everything is fine?

Update: editing this from my sister's house... via afs