MachineMetadata constarints with %i interpolation doesn't work.

[daniellowtw]

We're using the %i interpolation for dynamic scheduling. Our instance metadata looks as follows:

fleetctl list-machines | grep es-                                         
94356eb9... 10.200.0.6  image=coreos-beta-899-4-0-v20160121,instancename=eu1-staging-es-0,instancetype=elasticsearch,region=europe-west1-d
c8648469... 10.200.0.12 image=coreos-beta-899-4-0-v20160121,instancename=eu1-staging-es-2,instancetype=elasticsearch,region=europe-west1-d
d00b1eae... 10.200.0.8  image=coreos-beta-899-4-0-v20160121,instancename=eu1-staging-es-1,instancetype=elasticsearch,region=europe-west1-d

When we take a unit like:

[Unit]
Description=echo Service
Description=echo Service
After=etcd-cluster-ready.service docker.service dns-resolv-update.service
Requires=etcd-cluster-ready.service docker.service dns-resolv-update.service

[Service]
Restart=always
RestartSec=60
TimeoutSec=0
ExecStartPre=-/opt/bin/echo "whatever %i"
ExecStart=/opt/bin/echo "whatever %i"
ExecStop=-/opt/bin/echo "whatever %i"

[X-Fleet]
MachineMetadata="instancetype=elasticsearch" "instancename=%i"

And we submit two copies of it:
fleetctl start echo@eu1-staging-es-1.service echo@eu1-staging-es-2.service

We'd expect the echo@eu1-staging-es-1.service to be scheduled on d00b1eae and echo@eu1-staging-es-2.service to be scheduled on c8648469.

However, what we're seeing is:

Unit echo@eu1-staging-es-1.service inactive
Unit echo@eu1-staging-es-2.service inactive
Unit echo@eu1-staging-es-1.service launched on d00b1eae.../10.200.0.8
Unit echo@eu1-staging-es-2.service launched on d00b1eae.../10.200.0.8

This happens rarely, but it does happen. We couldn't reproduce it when the constraint instancename=%i was explicit "instancename=eu1-staging-es-2".

It's pretty puzzling, since we know that the only place that deals with Units of Jobs is the jobs.go and that's used by the engine and agents. The interpolation is done within it: https://github.com/coreos/fleet/blob/master/job/job.go#L179

Any ideas what this could be caused by?

[daniellowtw]

It seems like it is only affecting one of our clusters and it is possible that cluster may have been running another fork of fleet that suffers from this issue. Rebooting the machines seems to fixed it.

I'm closing this for now until I see this happening again.

[daniellowtw]

I'm reopening this bug because we updated our fleet to use master branch HEAD and we started seeing this problem again.

[mwitkow]

@jonboulle
We've recently rebased our fleetd that we run to be on the HEAD of master and patched it with the monitoring patches (#1415). The branch is here (https://github.com/improbable-io/fleet/tree/imp_release_20160316_INF-metrics).

We think something changed between fleetd version 0.11.5 and HEAD that caused the instancename=%i interpolation not to work in engine scheduling. We've seen similar problems before when we were experimenting with head before (hence the bug).

We're rolled back to a version branched off of:

3178aed

Merge pull request #1263 from

miekg/followonly-hack

Implement follow_only flag

This version does not have this problem, so this issue has been introduced since then.

[kayrus]

Confirmed.

$ fleetctl list-machines
MACHINE         IP      METADATA
1e1a8655...     coreos3 hostname=coreos3
74dce006...     coreos2 hostname=coreos2
cd29d103...     coreos1 hostname=coreos1

template:

[Service]
ExecStart=/bin/bash -c "while true; do echo Hello, World %i!; sleep 1; done"

[X-Fleet]
MachineMetadata="hostname=%i"

$ fleetctl start hello@random.service

v0.11.5 just hangs when it can not find corresponding machine.
master branch schedules unit randomly independently on metadata.

[daniellowtw]

Can you try with this instead?

[Service]
ExecStart=/bin/bash -c "while true; do echo Hello, World %i!; sleep 1; done"

[X-Fleet]
MachineMetadata=hostname=%i

IIRC, the way the code parses the MachineMetadata is special when there is only one condition and when there are more than one.

Also, doesn't fleetctl start hello@random.service try to schedule this on machines with hostname=random? If so, then it wouldn't be scheduled on any of the machines in the cluster.

[kayrus]

@daniellowtw what do you mean? I use only one condition.

[mwitkow]

I @daniellowtw was considering the case of AND vs OR in: https://github.com/coreos/fleet/blob/master/Documentation/unit-files-and-scheduling.md#schedule-unit-to-machine-with-specific-metadata

[daniellowtw]

@kayrus https://github.com/coreos/fleet/blob/master/job/job_test.go#L323
The test uses

MachineMetadata=hostname=%i

when there is one condition and

MachineMetadata="hostname=%i" "foo=bar"

when there's more than one.

I think it doesn't parse it if there are quotes when there is only one required metadata
i.e. MachineMetadata="hostname=%i"