Make plugins Docker containers

[jafreck]

Use Docker compose to orchestrate the containers

Each plugin can either be :

• a simple script as it is now(Run on the spark container or on the host)
• a docker compose file(We then use docker compose -f spark.compose.yml -f plugin1.compose.yml .... )

[jafreck]

I feel like plugins shouldn't be just scripts without a container. Then, what is the difference between a plugin and a custom-script?

[timotheeguerin]

well thats the thing we get rid of custom scripts.
Also it gives options to open ports and I feel like some people might not want to setup a full docker container for a plugin when they just need a few command a script can do

[jafreck]

ah okay, that makes a lot of sense if we totally deprecate custom scripts