sFlow not samples sent if the subinterfaces setup only

[AnDroed09]

It looks like the sFlow agent no longer supports subinterfaces in Packet Sample 2.1.17-1.
In version 2.1.11-5 this functionality was available and working.

sflow {
  polling = 20
  sampling = 128
  sampling.bps_ratio = 0
  agent = eth2
  collector { ip = 10.0.2.2 udpport = 6364 }
  pcap { dev = eth1.20 }
  pcap { dev = eth1.10 }
  dropmon { limit = 50 start = on sw = on hw = off }
  dbus { }
}

Problem:
When using subinterfaces (eth1.10, eth1.20), no packet samples are sent to the collector.
If I instead use the parent interface (eth1), samples appear at the collector.

Could you please check if this is expected change?

Thanks!

[sever-sever]

Some context https://forum.vyos.io/t/sflow-not-sending-to-server-in-latest-rolling-images/
So before users set subinterfaces and after updating the sFlow version in the VyOS we have got this behaviour.

[sflow]

I think it is probably the difference between this (from latest):

host-sflow/src/Linux/mod_pcap.c

Lines 509 to 549 in c600dcb

	static void evt_config_first(EVMod *mod, EVEvent *evt, void *data, size_t dataLen) {
	HSP *sp = (HSP *)EVROOTDATA(mod);
	// the list of pcap {} sections may expand to a longer list of BPFSoc
	// objects if we are matching with patterns or on ifSpeed etc.
	SFLAdaptor *adaptor;
	UTHASH_WALK(sp->adaptorsByName, adaptor) {
	for(HSPPcap *pcap = sp->pcap.pcaps; pcap; pcap = pcap->nxt) {
	if(pcap->dynamic) // ignore dynamic pcaps here
	continue;
	if(pcap->dev
	&& adaptorByName(sp, pcap->dev) != adaptor)
	continue;
	if(pcap->dev_regex
	&& regexec(pcap->dev_regex, adaptor->deviceName, 0, NULL, 0) != 0)
	continue;
	if(pcap->speed_set
	&& !speedTestOK(mod, pcap, adaptor))
	continue;
	// passed speed test, but there may be other reasons to reject:
	HSPAdaptorNIO *nio = (HSPAdaptorNIO *)adaptor->userData;
	if(nio->bond_master) {
	EVDebug(mod, 1, "skip %s (bond_master)", adaptor->deviceName);
	continue;
	}
	if(nio->vlan != HSP_VLAN_ALL) {
	EVDebug(mod, 1, "skip %s (vlan=%u)", adaptor->deviceName, nio->vlan);
	continue;
	}
	if(nio->devType != HSPDEV_PHYSICAL
	&& nio->devType != HSPDEV_OTHER) {
	EVDebug(mod, 1, "skip %s (devType=%s)",
	adaptor->deviceName,
	devTypeName(nio->devType));
	continue;
	}
	// passed all the tests
	addBPFSocket(mod, pcap, adaptor);
	}
	}
	}

And this (from 2.1.11-5):

host-sflow/src/Linux/mod_pcap.c

Lines 443 to 495 in 61094a1

	static void evt_config_first(EVMod *mod, EVEvent *evt, void *data, size_t dataLen) {
	HSP *sp = (HSP *)EVROOTDATA(mod);
	// the list of pcap {} sections may expand to a longer list of BPFSoc
	// objects if we are matching with patterns or on ifSpeed etc.
	for(HSPPcap *pcap = sp->pcap.pcaps; pcap; pcap = pcap->nxt) {
	if(pcap->dev) {
	SFLAdaptor *adaptor = adaptorByName(sp, pcap->dev);
	if(adaptor == NULL) {
	myLog(LOG_ERR, "pcap: device %s not found", pcap->dev);
	continue;
	}
	addBPFSocket(mod, pcap, adaptor);
	}
	else if(pcap->speed_set) {
	if(debug(1)) {
	char sp1[20], sp2[20];
	printSpeed(pcap->speed_min, sp1, 20);
	printSpeed(pcap->speed_max, sp2, 20);
	EVDebug(mod, 1, "searching devices with speed %s-%s", sp1, sp2);
	}
	SFLAdaptor *adaptor;
	UTHASH_WALK(sp->adaptorsByName, adaptor) {
	EVDebug(mod, 2, "consider %s (speed=%"PRIu64")", adaptor->deviceName, adaptor->ifSpeed);
	if((adaptor->ifSpeed == pcap->speed_min && pcap->speed_max == 0)
	|| (adaptor->ifSpeed >= pcap->speed_min
	&& adaptor->ifSpeed <= pcap->speed_max)) {
	EVDebug(mod, 2, "%s speed OK", adaptor->deviceName);
	// passed the speed test, but there may be other
	// reasons to reject this one:
	HSPAdaptorNIO *nio = (HSPAdaptorNIO *)adaptor->userData;
	if(nio->bond_master) {
	EVDebug(mod, 1, "skip %s (bond_master)", adaptor->deviceName);
	}
	else if(nio->vlan != HSP_VLAN_ALL) {
	EVDebug(mod, 1, "skip %s (vlan=%u)", adaptor->deviceName, nio->vlan);
	}
	else if(nio->devType != HSPDEV_PHYSICAL
	&& nio->devType != HSPDEV_OTHER) {
	EVDebug(mod, 1, "skip %s (devType=%s)",
	adaptor->deviceName,
	devTypeName(nio->devType));
	}
	else {
	// passed all the tests
	addBPFSocket(mod, pcap, adaptor);
	}
	}
	}
	}
	}
	}

As you can see, the 2.1.11-5 code is certainly trying to reject sampling-points that are not physical interfaces, because the sFlow standard is written that way. But it seems that something was allowing your sub-interfaces to slip past the checks before? Not sure why. If you add some logging or breakpoints, you might find it is something quite subtle, such as whether this function:

host-sflow/src/Linux/readInterfaces.c

Line 494 in c600dcb

static bool ethtool_get_GDRVINFO(HSP *sp, struct ifreq *ifr, int fd, SFLAdaptor *adaptor)

decides whether the interface is physical or not?

The intent of the sFlow data model is to monitor the physical ports, and distinguish sub-interfaces at the sFlow collector, based on 802.1Q VLAN tags, or MPLS label stacks. If you were to restrict your CLI settings to only allow sFlow to be configured on physical interfaces you would be matching the same restrictions that applies on hardware switches. Would that be a big problem for VyOS? Or would it just be something that needs to be clarified in the CLI?

(You will see in the hsflowd code base that we do sometimes apply the "black box with ports" model to virtual switches that exist within the server, so that we end up applying sampling to virtual ports. For example this happens with a docker bridge or with kubernetes networking. This may seem like a contradiction, but it is consistent with the model of the network being a graph of nodes connected by edges.... it's just that we are adding one more leaf node to the graph. One of the ports on that imaginary node is the NIC, so the graph still has an edge that matches a port on the upstream node.)

Hope this is not too confusing. Another way to think about this is to consider sFlow as a "whole network" monitoring system. It is necessary to extract the dedup'd traffic matrix that represents the total demand on the network, irrespective of the topology that is carrying that traffic. To do that properly, it really helps that every edge in the graph means the same thing to the nodes at either end. Then all layer3 / layer 4 relationships that float on top of this fabric can be captured and distinguished with filters at the collector.

[sflow]

Just committed change (9e29397) to master that will go back to allowing this in the case where the dev is specified explicitly.... along with long comment about why... but it's almost certainly still wrong for VyOS to allow sampling to be configured on a sub-interface. Should really be physical ports only.

[dmbaturin]

We found that Cisco and Arista do support enabling sFlow sample collection only for sub-interfaces, so the idea to allow it per sub-interface seems to be the de facto standard among vendors.

We'll test the update and report back, thanks for this!

[sflow]

Taking a step back.... by far the most important thing is what happens by default, when the customer just sets an sFlow collector and enables sFlow. On Arista, on SONiC, on VPP and on most other platforms this will result in packet-sampling on all physical ports. That way if a link is congested you can see why.

I'm not sure exactly what the simplest possible sFlow config on VyOS looks like. Can you share? I'd like to put it here:
https://github.com/sflow/config/blob/main/VyOS/sflow-config-VyOS.md

Beyond that, if a vendor finds they want to add a corner-case that violates the spec, e.g. just to please a customer, then we tend to turn a blind eye because we understand how that comes about in the real world. We would just ask that it be obscure, and perhaps come with some warning or explanation.

I hope you are persuaded of the value of the spec as it is written. Sampling all traffic on a physical link and distinguishing L3-4 patterns at the collector is really the only way to avoid carrier-grade madness. If you come across a situation where the sFlow export does not contain the information needed to distinguish an important subset of the traffic, then please bring it up on the mailing list because that is where an extra packet-sample annotation may be needed. For existing examples of that, look at the extended_switch, extended_router, extended_gateway and extended_nat structures.