The purpose of this repository is to implement the Dempster-Shafer Theory of Evidence for decision-level model fusion using image segmentation predictions. The models we use in this repository ingest pre- and post-flood remote sensing RGB satellite images, which are then segmented to identify and label flooded and non-flooded roads and buildings. We then fuse outputs from two different models to improve performance and calculate the uncertainty of the fused predictions, providing some explainability through the uncertainty map of the original images.

We use the SpaceNet 8 dataset, which provides curated and labeled MAXAR images in Louisiana and Germany. We also implement the SpaceNet 8 baseline Twin Neural Network (TNN) model, the Ultralytics YOLOv8 model, and a modified version of the Siamese Nested UNet (SNUNet) model.

Clone the repository and in the root of the directory run the following command to create the Python virtual environment:

$ python3.10 -m venv env

Source the new environment:

$ source env/bin/activate

Update pip and install the required packages:

$ python -m pip install --upgrade pip
$ pip install -r requirements.txt

Installing GDAL

$ sudo apt update
$ sudo apt-get install gdal-bin
$ sudo apt-get install libgdal-dev
$ pip install gdal=="$(gdal-config --version).*"
$ pip install pygdal=="$(gdal-config --version).*"

If there are issues with using the environment with jupyter notebooks, then make sure ipykernel is installed and that the environment registered:

$ pip install ipykernel
$ python -m ipykernel install --user --name env

Create a specified data directory then set the data_dir path in ./config/paths/default.yaml to match the path to the data directory. For example, if you create a directory called data in the root of this repo, then the path can be set to ${paths.work_dir}/data/.

The SpaceNet 8 dataset is publicly available through Amazon Web Service (AWS) and can be accessed via the command line tool aws, which can be installed for Linux as follows, or check Installing or updating to the latest version of the AWS CLI for instructions related to your operating system.

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ sudo ./aws/install

The training datasets can be downloaded with the following commands:

$ mkdir data
$ mkdir data/Germany
$ mkdir data/Louisiana-East
$ aws s3 cp --no-sign-request s3://spacenet-dataset/spacenet/SN8_floods/tarballs/Germany_Training_Public.tar.gz data/Germany/. 
$ aws s3 cp --no-sign-request s3://spacenet-dataset/spacenet/SN8_floods/tarballs/Louisiana-East_Training_Public.tar.gz data/Louisiana-East/. 

The testing dataset is also available, but we did not use it because there are no ground truth labels available.

Uncompress the data files, then delete the old tarball to save space:

$ cd data/Germany
$ tar -xzvf Germany_Training_Public.tar.gz
$ rm Germany_Training_Public.tar.gz
$ cd ../Louisiana-East
$ tar -xzvf Louisiana-East_Training_Public.tar.gz
$ rm Louisiana-East_Training_Public.tar.gz

If you downloaded the data to data/ in the root directory then you do not need to change the data_dir path in the configurations. If you did download the data to a directory with a different name or different relative location from this repository's root directory, then you need to set the path in config/paths/default.yaml:

data_dir: ${paths.work_dir}/data/

${paths.work_dir} is the root directory of the repository.

After the data directory is created, the data is downloaded, and the data_dir path is set we can process the data for the models and create the splits by running the command:

$ python data_processing.py

After the script is done, you will find txt and CSV files that list the train, val, and test images for each of the seven folds (labeled 0-6) located in data/folds/; and the Germany and the Louisiana-East directories will have an images and labels directories, each with a subdirectory for PRE-event and POST-event where the image png file and label txt files are stored and will be used by the models.

Models can be trained by calling the main.py file and specifying the model configuration to use. For example, to train the baseline SN8 model, we use the resnet34_siamese configuration, which is saved at config/model/resnet34_siamese.yaml but we only need the file name (see below). We can also specify which fold of the data to hold out for testing with the argument datamodule.fold, where <K> below specifies the fold ID. Training is run separately for each fold, but the scripts/run_train.sh automates training with all the folds (and does the training for the other models too). We can also specify batch size (datamodule.batch_size), and training epochs (trainer.max_epochs).

$ python main.py task_name=train model=resnet34_siamese datamodule.batch_size=4 datamodule.fold=<K> trainer.max_epochs=50

Logging information will be printed to the standard output, but is also stored in logs/train/Resnet34_siamese/<K>/<DATE>_<TIME>/main.log. In the same directory as the log file, there should be a CSV file of test results with the held out data, a confusion matrix, and a lightning_logs directory that contains the saved model weights (logs/train/Resnet34_siamese/<K>/<DATE>_<TIME>/lightning_logs/version_0/checkpoints/epoch=49-step=7150.ckpt).

We recommend copying model weights to a models directory in the root of the repository with the model configuration name and fold for easy reference, for example, the above weights should be moved to models/Resnet34_siamese_<K>.ckpt. There is a script that will automate this described in the Training and Evaluation section below.

After the model is trained and weights are saved we run it on the test data separately, or run it on new data. NOTE: when using folds of the data you MUST specify the same fold for the data (datamodule.fold=<K>) as what was specified during training to ensure that you do not test the model with data it was trained on. That is why we highly recommend changing the name of the model weights to include the fold ID.

The main.py script is also used for evaluation, but we set task name to eval and provide model weights with ckpt=models/Resnet34_siamese_<K>.ckpt, so the trained model is loaded instead of training a new model. NOTE: if you don't provide a file for weights the script will try to train a new model.

$ HYDRA_FULL_ERROR=1 python main.py task_name=eval model=resnet34_siamese\
     datamodule.fold=<K> ckpt=models/Resnet34_siamese_<K>.ckpt

After the evaluation is run, like training, there will be a directory in logs that contains the logs and test results: logs/eval/Resnet34_siamese/<K>/<DATE>_<TIME>/

Our Dempster-Shafer Fusion uses a discount factor (DF) that weighs model predictions based on how reliable they are. We set reliability based on the Mean Average Precision (mAP) metric of the model on the validation data. DF is calculated as the complement of reliability: $DF = 1 - reliability$.

We can run the model on validation data that same way we ran it on the test data for evaluation in the previous section, except we must change the file path of the csv to use with the argument datamodule.test_csv=/home/jovyan/shared/Data/folds/sn8_data_val_<K>.csv. The commands below run the SN8 baseline and SNUNet models on the validation data to calculate the mAP for reliability

$ HYDRA_FULL_ERROR=1 python main.py task_name=val model=resnet34_siamese\
     datamodule.test_csv=/home/jovyan/shared/Data/folds/sn8_data_val_<K>.csv\
     datamodule.fold=<K> ckpt=models/Resnet34_siamese_<K>.ckpt

$ HYDRA_FULL_ERROR=1 python main.py task_name=val model=snunet_medium\
     datamodule.test_csv=/home/jovyan/shared/Data/folds/sn8_data_val_<K>.csv\
     datamodule.fold=<K> ckpt=models/SNUNet_medium_<K>.ckpt

After the models are run with the validation data, we can pull the mAP values from logs/val/Resnet34_siamese/<K>/<DATE>_<TIME>/main.log and logs/val/SNUNet_medium/<K>/<DATE>_<TIME>/main.log. Near the end of the log file there is a list of metrics, including test/mAp which is the one we want. For example, we could expect to have

Using these values and the DS_flood_snunet model configuration, we can fuse these two models with the DF by calling the command below, which includes specifying both the model weights to use and the reliability as arguments.

$ python main.py model=DS_flood_snunet model.name=DS_Resnet34_siamese_SNUNet\
    ckpt=None task_name=eval_DF_mAP datamodule.fold=<K>\
    model.module.litnet1.checkpoint_path=models/Resnet34_siamese_<K>.ckpt\
    model.module.litnet2.checkpoint_path=models/SNUNet_medium_<K>.ckpt\
    model.module.litnet1.reliability=0.656\
    model.module.litnet2.reliability=0.582

The outputs for running the fused model are the same as running the evaluation script, there will be a log and csv of the results saved in logs/eval_DF__mAP/DS_Resnet34_siamese_SNUNet/<K>/<DATE>_<TIME>/. If we were to check the metrics of the main.log here, we should expect to see improved performance.

In our experiments, for fold 0, we have the above values for mAP of the SN8 and SNUNet models, and the mAP for the fused model was 0.718 which is a 9.45% increase in performance, although not every fold will see this magnitude of an increase.

This section provides a more detail on each of the major components of this repository and the configurations they use. It is meant to provide some insights for users to understand how the PyTorch Lightning framework is used and how the model outputs are fused together, and how a user could use the code for different datasets or experiments.

The datasets (train, val, and test) are created by extending the Dataset class from Pytorch, as defined in the SN8Dataset class in src/datamodules/sn8_dataset.py. This class reads the data based on paths listed in a CSV file (created in the Data Download and Processing section above) based on a list of which images and labels of the SpaceNet 8 data to use (e.g., we use the pre- and post-event images and the flood labels). This object then serves the data in batches to the model. The user does not need to work with this class as it is instantiated and used but the datamodule class.

The datamodule class SN8DataModule in src/datamodules/sn8_datamodule.py extends the LightningDataModule class and follows the PyTorch Lightning Framework for handling data. This class creates and stores the train, val, and test datasets for a particular fold, which are used by all the models. This does require providing three separate CSV files paths, one each for train, val, and test, but these files are created in the Data Download and Processing section above.

Configurations for the datamodule are set in the config/datamodule/sn8_flood_fold.yaml file. The user can set the fold here and it automatically populates that into the file paths for the train, val, and test sets. The user can also specify the batch size and number of workers based on the system you are using (i.e., GPU size). There is also a setting for region to limit the data read and used to either Germany or Louisiana, but we never implemented this in our work as this further reduced the training dataset size too much.

Models are also created using the PyTorch framework for model architectures in src/architecture and the PyTorch Lightning framework for storing model specific behaviors and metrics, simplifying the main.py script. The architecture files came from the SpaceNet 8 and SNUNet repositories, while the YOLO architecture is define solely in the Ultralytics package. These architectures are used to create a Neural Network (NN) which is contained in the associated PyTorch Lightning module found in src/modules. For each model, the NN is wrapped by the module which contains code for running the model, calculating evaluation metrics, and saving results.

Model configurations are set by the files in config/model, where there is a file for each individual model and the Dempster-Shafer (DS) Fusion of SN8 + SNUNet and SN8 + YOLO. Each model is different, so configurations also differ. However, this is where the user can set learning rate, the optimizer, the learning rate scheduler, or any of the other YOLO parameters. Note that while the DS model configurations include all the parameters for training, these models are NOT used for training; this is just for properly creating model objects before loading trained weights.

Following the PyTorch Lightning Framework, training and evaluation (i.e., testing) procedures are set by the PyTorch Lightning modules in src/modules. For example, the LitSN8FloodAttribution in lit_sn8_flood_attribution.py contains methods for training_step, validation_step, and test_step which all use the basic step method which defines what happens each time data is passed to the model, how ground truth data is pre-processed for loss calculations and outputs are post-processed for prediction belief to be used with DS Fusion. The test_step also defines how all the metrics are calculated and saved. Each model has their own as they process data in a slightly different way, but these modules standardize the outputs for calculating metrics and DS fusion, while still allowing for model specific training.

The module defines these behaviors, so in main.py we only need to create the datamodule, model, and trainer then tell the trainer to fit the model with the datamodule (trainer.fit(model=model, datamodule=datamodule)) or to evaluate a pretrained model by also specifying its weights (trainer.test(model=model, datamodule=datamodule, ckpt_path=weights)). This oversimplifies main.py a little bit, as it needs to account for the particular way YOLO needs to be run and how to account for running DS fusion models.

To train all of the models you can run the training bash script run_train.sh:

$ bash scripts/run_train.sh

which automates training the SN8, SNUNet, and YOLO models with each fold of the data. After the training is done you can move the model weights into the models directory with the script copy_weights.py. From the root directory run:

$ mkdir models
$ python scripts/copy_weights.py

Once the weight files are moved to models/ then we can run the validation and evaluation of the singular models with the run_eval.sh script:

$ bash scripts/run_eval.sh

The validation results can be read and compiled with the compile_val_results.py script:

$ python scripts/compile_val_results.py

This saves performance metrics for each of the models on the validation split of the data, which we will need for the discount factor during the decision fusion process of FADS-Fusion.

There is a specific DS fusion PyTorch Lightning module (src/modules/lit_DS.py) that takes two sub-PyTorch Lightning models as input (e.g., a trained LitSN8FloodAttribution and a trained LitSNUNet). This class does not perform training (though it has a placeholder method), rather it performs evaluation only by serving the data to each sub-model and then combines their outputs. Once fused, it calculates and saves the evaluation metrics like the other modules for the individual models.

$ python scripts/eval_fusion_discount.py

The evaluation results can be compiled with the compile_eval_results.py script:

$ python scripts/compile_eval_results.py

Then you can use the analysis notebooks to compare the model-level fold results and the individual image-level results:

• notebooks/mode_level_results_analysis.ipynb
• notebooks/image_level_results_analysis.ipynb

Note: that because initial model weights are randomized, the final model results may differ slightly than what is presented in the paper.

Finally, we can plot the uncertainties of the fused prediction by running the plot_uncertainty.py script:

$ python scripts/plot_uncertainty.py

This will run the SN8 and SNUNet models on a sample of images and plot the total uncertainty. It also plots a comparison of model predictions for the individual and fused models, the ground truth, and the uncertainties, where each row is the predictions (or weighted uncertainty) for the classes non-flooded building, flooded building, non-flooded road, flooded road, and null (or empty).

• Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., Chauhan, G., Chourdia, A., Constable, W., Desmaison, A., DeVito, Z., Ellison, E., Feng, W., Gong, J., Gschwind, M., … Chintala, S. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 929–947. https://doi.org/10.1145/3620665.3640366
• Falcon, W., & The PyTorch Lightning team. (2019). PyTorch Lightning. Lightning-AI. https://www.pytorchlightning.ai
• Fang, S., Li, K., Shao, J., & Li, Z. (2022). SNUNet-CD: A Densely Connected Siamese Network for Change Detection of VHR Images. IEEE Geoscience and Remote Sensing Letters, 19, 1–5. https://doi.org/10.1109/LGRS.2021.3056416
• Hänsch, R., Arndt, J., Lunga, D., Gibb, M., Pedelose, T., Boedihardjo, A., Petrie, D., & Bacastow, T. M. (2022). SpaceNet 8 - The Detection of Flooded Roads and Buildings. 1472–1480. https://openaccess.thecvf.com/content/CVPR2022W/EarthVision/html/Hansch_SpaceNet_8_-_The_Detection_of_Flooded_Roads_and_Buildings_CVPRW_2022_paper.html
• Jocher, G., Chaurasia, A., & Qiu, J. (2023). Ultralytics YOLOv8 (Version 8.0.0). Ultralytics. https://github.com/ultralytics/ultralytics
• Warmerdam, F., & Dezhin, A. (2022). pygdal (Version 3.4.1). https://pypi.org/project/pygdal/
• Yadan, O. (2019). Hydra - A framework for elegantly configuring complex applications. Facebook. https://github.com/facebookresearch/hydra

The structure of this repository is based on the following PyTorch Lightning Templates:

• ashleve/lightning-hydra-template
• Lightning-AI/deep-learning-project-template

If you use the code in this repository, please cite this paper:

@article{sobien2026fads,
  title={FADS-Fusion: A Post-Flood Assessment Using Dempster--Shafer Fusion for Segmentation and Uncertainty Mapping},
  author={Sobien, Daniel and Sobien, Chelsea},
  journal={Remote Sensing},
  volume={18},
  number={5},
  pages={714},
  year={2026},
  publisher={MDPI},
  issn={2072-4292},
  doi={10.3390/rs18050714}
}

If you used the data and/or data processing tools, please cite this paper:

@inproceedings{hansch2022spacenet,
  title={Spacenet 8-the detection of flooded roads and buildings},
  author={H{\"a}nsch, Ronny and Arndt, Jacob and Lunga, Dalton and Gibb, Matthew and Pedelose, Tyler and Boedihardjo, Arnold and Petrie, Desiree and Bacastow, Todd M},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={1472--1480},
  year={2022}
}

If you use the YOLO model, please cite:

@book{jocher2023yolo,
  title={Ultralytics YOLOv8},
  url={https://github.com/ultralytics/ultralytics},
  publisher={Ultralytics},
  author={Jocher, Glenn and Chaurasia, Ayush and Qiu, Jing},
  year={2023},
  month=jan
}

And finally, if you use the SNUNet model, please cite:

@article{fang2021snunet,
  title={SNUNet-CD: A densely connected Siamese network for change detection of VHR images},
  author={Fang, Sheng and Li, Kaiyu and Shao, Jinyuan and Li, Zhe},
  journal={IEEE Geoscience and Remote Sensing Letters},
  volume={19},
  pages={1--5},
  year={2021},
  publisher={IEEE}
}