Page 1 of 1

Mystery bridges

PostPosted: Jan 26th, '19, 04:36
by jiml8
My workstation has 3 NICs on it. The onboard NIC, eth0 is a realtek device. The other nics (eth1 and eth2) are an intel dual-nic pci-e plugin card.

I use eth0 for my internet connection. The other 2 are used for various development purposes. Usually my workstation is configured as a router, and I will connect eth1 and eth2 to eth0 using iptables with snat. Historically this has worked fine and allows me to conveniently examine packets as they flow through my machine, while giving me full control over the interfaces...which is really helpful during some of my development.

As I write this, I have a device with 2 WANs connected to eth1 and eth2 (one wan to each connection) and each wan is pinging to a different DNS server.

Also as I write this, I have no rules connecting eth1 or eth2 to eth0. There are no bridges defined (by me, anyway). Traffic from that device into these nics should go nowhere.

Nonetheless, pings from the device connected to eth1 and eth2 are going out eth0, with no NAT. In fact, my iptables rules at this time are having no effect at all on these nics. I have even killed ALL my firewall rules, which results in all my internet traffic being killed - except for these pings. Both NICs are bridged to eth0, I didn't do it, and I see nowhere where this shows and I don't know how to stop it short of a reboot (which I will do in a few minutes). When I take down the interfaces, the pings stop. When I bring them back up and assign IP addresses, the pings promptly resume.

There are a couple of host-only virtual lans running in this workstation. This is how I control Windows VMs mostly. These host only VMs are working as expected; I can allow and deny access to the internet by using iptables rules. Only the physical NICs have gone crazy.

I am assuming that the Great, the Wonderful, systemd is doing something. But I have no idea what.

Does anyone here have any idea what might be causing this?

Re: Mystery bridges

PostPosted: Jan 26th, '19, 04:56
by jiml8
I just rebooted and it is still happening. When the box came up, the hotplugged network took a long time to start.

systemd is screwing me up. How the FUCK do I stop this so I can get my work done????

Re: Mystery bridges

PostPosted: Jan 26th, '19, 09:41
by aguador
I hate to say it, but my short answer is I have no idea what is going on. My question would be: are you on Cauldron and have you updated iptables? There have been both upstream changes and some issues in building the current version (1.8.2-4). Assuming Cauldron, update to build 4 of the current iptables. If you have that version, rollback and post the issue on the dev ml so it can be addressed.

Re: Mystery bridges

PostPosted: Jan 26th, '19, 18:46
by jiml8
I am on an up to date 6.1 installation. No way would I try to use cauldron for my workstation; I make my living on this box.

That is, I make my living on it when I don't have problems like this.

It can't be accidental; someone has decided that multiple interfaces on a box should be bridged, and that is what has been pushed out. I could spend many hours figuring this out, and if I'm going to have to spend that time, I'll spend it changing distros, as painful as that will be.

This is stopping my work.

The behavior is new in the last week or so; that was the last time I went to use these interfaces. In the interim, I am thinking of rolling back about two weeks, but that is not a solution.

Re: Mystery bridges

PostPosted: Jan 26th, '19, 20:45
by jiml8
Well, I rooted around in systemctl a bit and figured out that the interfaces are being handled by ifplugd. Not sure I like that in any case; this isn't a laptop.

So, I killed all the instances of ifplugd and manually configured networking.

And the NICs are still bridged to each other.

I guess bridged is not the way to describe it because it seems to be one directional. Traffic from eth1 and eth2 is appearing at eth0, regardless of what I do with iptables. But traffic on eth0 is not appearing at eth1 or eth2.

WTF???

Re: Mystery bridges

PostPosted: Jan 26th, '19, 21:02
by jiml8
So, with just eth0 and eth1 enabled (eth2 disabled), this is my routing table:
Code: Select all
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         gateway         0.0.0.0         UG    0      0        0 eth0
72.183.45.0     0.0.0.0         255.255.255.0   U     0      0        0 vmnet5
172.16.187.0    0.0.0.0         255.255.255.0   U     0      0        0 vmnet1
172.16.188.0    0.0.0.0         255.255.255.0   U     0      0        0 vmnet2
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 vlan0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 vmnet6
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.11.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.218.0   0.0.0.0         255.255.255.0   U     0      0        0 vmnet8


I deleted the entry for eth1 from the table, and this stopped the traffic.

Therefore, non-routable addresses are being routed in my workstation.

After removing the route entry, I enabled my iptables rules for this interface, and they worked. For a minute. Then traffic stopped.

Now, the device connected to eth1 is sending arp queries looking for its gateway (which is eth1, 192.168.11.1) and my box is NOT responding.

Also, lacking a route, I cannot ping the device connected to eth1 from within my workstation.

Something is seriously wrong with the networking stack in this box.

Re: Mystery bridges

PostPosted: Jan 26th, '19, 21:15
by jiml8
My last kernel update was 27 Dec 2018 and I am using 4.14.89-desktop-1.mga6.

I have been using this capability since 27 Dec. Most recently (until now) was somewhere around 14 January, and it did work then.

There have been one or two Mageia updates since that time, but I really don't know what all came through.

Re: Mystery bridges

PostPosted: Jan 26th, '19, 21:20
by jiml8
I just recalled that a few days ago, I settled down to get Pithos working. It stopped when I updated to Mageia 6, and I decided to get it going.

Well, I did get it working. But its installation involved some third party repositories. So I probably should look at those, though the things it did were all python and gtk3 and I don't see how that would impact networking. But something has impacted networking.

Re: Mystery bridges

PostPosted: Jan 27th, '19, 02:20
by doktor5000
Best take a look at the output of rpm -qa --last and scroll through it from top to when you installed/updated something in december ...

Re: Mystery bridges

PostPosted: Jan 27th, '19, 19:49
by jiml8
Because I did get pithos running, which involved installing the meson build system, I diff'ed my running system against a backup from 19 January. I found a fair number of changes, mostly with the meson build system, some flatpack stuff that was needed to build pithos and which did not come from a mageia rpm, as well as some big changes in gir, python and in PHP. Python and gir because of pithos, and PHP because of an update from mageia. I also added dnf in order to play with it.

All of this stuff does not look like anything that would operate low enough in the system, or in the right part of the system.

I also found an update to nss and that does look like something that operates in the part of the system that seems to be affected.

When I looked with rpm -qa --last I found the nss updates as well as an extensive php update, and vnc, dbus, ssh, sqlite, gir, python 3.5 updates on 20 Jan.

So it has to be there somewhere. NSS look like the best candidate.

Re: Mystery bridges

PostPosted: Jan 30th, '19, 02:06
by morgano
Maybe you can swap out your disks temporarily, and install Mageia 6.1 fresh with all updates and only Mageia packages and see if you can reproduce the problem. If no problem then install other stuff you had until you see the problem again. May be a good idea to install using LVM and use snapshots, or another method to snapshot in order to take many snaps and roll back if you find something breaking.

Re: Mystery bridges

PostPosted: Apr 6th, '19, 05:36
by jiml8
I did not have the time to deal with this issue, so I worked around it.

I went into the office (where there are 2 different ISP connections) in order to do the work I needed to get done. I then moved on.

Time has passed, other updates have gone into place, and I once again needed to do some work using both NICs, similar to what I was doing in January when this problem occurred.

Problem is now gone. NICs and networking stack are behaving as expected.

Re: Mystery bridges

PostPosted: Apr 6th, '19, 13:13
by morgano
Nice then :)
Please add [SOLVED] to the beginning of subject line in first post.