multitasks continue their execution even if prerequisite fails

[mbunkus]

Problem summary: With the following Rakefile rake v12.0.0 continues building the "apps" even though one of its prerequisites (the "sources") fails.

This Rakefile is a dumbed-down version of my application's build system. There are several applications that all depend on a common library. This library in turn depends on several object files, and for each object file there's a single source file. There's a rule that compiles those source files into object files.

It's the basic C++ application model: several .cpp get turned into .o, those .o are linked into a .a, several applications link against that .a.

What happens, though, is that even if turning one of those .cpp into an .o fails (and therefore the .a cannot be built) the linker for the applications is still invoked. That linking step fails, as the library isn't found.

Here's the synthetic Rakefile that reproduces this issue:

num_sources   = 20
num_apps      = 8

apps          = (1..num_apps).map { |i| "app#{i}" }
objects       = (1..num_sources).map { |i| "source#{i}.o" }
sources       = (1..num_sources).map { |i| "source#{i}.cpp" }

$stdout_mutex = Mutex.new

def log text
  $stdout_mutex.synchronize {
    puts "#{Time.now} #{text}"
  }
end

task :default => apps

task :clean do
  (apps + objects + ["library"]).each do |name|
    File.unlink(name) if FileTest.exists?(name)
  end

  system "touch #{sources.join(' ')}"
end

compiler = lambda do |*args|
  log "Compiling #{args.first.name}"
  system "sleep 2"

  if args.first.name == "source4.o"
    log "Will throw an exception for source4.o"
    fail
  end
end

rule '.o' => '.cpp', &compiler

file "library" => objects do
  log "Creating library"
  system "touch library"
end

(1..num_apps).each do |i|
  file "app#{i}" => "library" do
    log "Creating app#{i}"
    system "touch app#{i}"
  end
end

First, run rake clean. This will create the dummy source files source1.cpp through source20.cpp. Next, run rake -m. You'll observe the following output:

2017-01-21 12:56:07 +0100 Compiling source1.o
2017-01-21 12:56:09 +0100 Compiling source2.o
2017-01-21 12:56:11 +0100 Compiling source3.o
2017-01-21 12:56:13 +0100 Compiling source4.o
2017-01-21 12:56:15 +0100 Will throw an exception for source4.o
2017-01-21 12:56:15 +0100 Creating app2
rake aborted!

/home/mosu/tmp/Rakefile:34:in `block in <top (required)>'
Tasks: TOP => default => app1 => library => source4.o
2017-01-21 12:56:15 +0100 Creating app4
(See full trace by running task with --trace)
2017-01-21 12:56:15 +0100 Creating app5

You can see that Rake aborts building app1, but it it still tries to create app4 and app5. Neither of those should have been continued as their prerequisite library has failed. Which of those other appX targets is continued depends on timing; it's not always 4 and 5.

Just for fun I've tried the same Rakefile with drake -j8 instead of rake -m. drake handles this case correctly and doesn't continue building any of the other appX targets.

[aelsabbahy]

I wonder if this is the same issue I'm experiencing #189.

For what it's worth, in my contrived example the behavior is intermittent.

[mbunkus]

I don't know, but my log above shows another issue: the items in my example aren't actually executed in parallel. Look at the timestamps of the "Compiling" lines — they're emitted by the rule, and those are definitely not in parallel.

Again, drake handles this correctly and executes those rules in parallel. In general I'm pretty disappointed by rake's multitask support. drake has had better and more stable support for parallelism for a number of years now.

[grzuy]

The following PR attempts to fix this issue
#252