Cml runner long job

[DavidGOrtega]

Introduces restart all the workflows with jobs not yet finished.

• Adds restart workflows with tracked and unfinished jobs
• Solves 72 hours max time in Github
• Refactors the code in cml parseRunnerLog and cml-runner shutdown
• solves unknown bug in logs due to incomplete string comparison success:

closes #174
closes #208 partially?

[DavidGOrtega]

#208 is pending because:

• We might find desirable to still handle timeouts
• In GH is near imposible to know if the job is canceled due to timeout, at least from the logs.

The only solution (and actually the original one) is to control the timer to not reach the MAX GH timeout time and restart a few min before reach it. This would solve the max time limit in GH respecting every timeout defined by the users.

What do you think?

@casperdcl @0x2b3bfa0 @Suor @mnrozhkov @dmpetrov

[casperdcl]

we should also maybe add an option (default true) to auto-post a warning comment saying "restarting due to timeout?"

[DavidGOrtega]

we should also maybe add an option (default true) to auto-post a warning comment saying "restarting due to timeout?"

I liked this idea and has been explored before but the runner needs to know the SHA. Aside that I think that what we should do is do proper logging #22 That way they could setup the logging config and being able to log everything and this would also be nice for DVC viewer

[casperdcl]

I think it should be configurable (--retry-timeout, similar to --idle-timeout) - here's the gist...

[0x2b3bfa0]

@DavidGOrtega, do you want a release manager review? They're free this week!

[DavidGOrtega]

@0x2b3bfa0 please!