Add a dump logs command for debugging user issues

[jafreck]

This can be a flag for aztk spark cluster run or something like that.

It should:

1. ssh into each node (create 'aztk' user on the fly)
2. get stdout.txt
3. get the docker logs
4. executor logs
5. driver logs

[jaley]

Just saw this ticket coming and thought it looked pretty useful - nice idea. Just wanted to add a checklist of logs that I've been finding useful while debugging our ETL pipeline:

• Driver application stderr
• Stderr for Spark services that write to $SPARK_HOME/logs on each node
• Executor output available from the worker Web UI, but I believe also written to $SPARK_HOME/work/*

I would mention though that iterating through each node with SSH might get pretty slow on a big cluster. I tried using the existing "copy" command on a cluster of 50-60 nodes and basically had to kill it after waiting 10-15 minutes. There's also the problem that the user needs to be around when the job fails, or leave their cluster running so they can salvage the logs.

In future, an alternative might be something like what EMR does. All logs are periodically uploaded from each node to an S3 bucket, and on cluster shutdown. I guess we could do something similar by running a script in the background on each node, monitoring folders of interest and copying to a storage container?

[jafreck]

Thanks for the suggestions for logs. I'll update the current PR to include $SPARK_HOME/work/* as it is not currently being pulled.

I am a little concerned here that the copy command is so slow. The command actually pulls from all nodes concurrently, so likely if you are experiencing a 10-15 minute period, then one of the nodes is unreachable. This is probably a timeout/retry issue that needs to be resolved.