Skip to content

Feature/sbra 360 training on r2 combustion#15

Merged
rmslm merged 6 commits intomainfrom
feature/SBRA-360-training-on-R2-combustion
Apr 8, 2022
Merged

Feature/sbra 360 training on r2 combustion#15
rmslm merged 6 commits intomainfrom
feature/SBRA-360-training-on-R2-combustion

Conversation

@rmslm
Copy link
Contributor

@rmslm rmslm commented Mar 12, 2022

couple of refactoring R2 and cnf for gnns
Create symbolic links to data where ever the data are on Atos clusters to avoid duplicating large data
config files can take the link of the data as parameter

@github-actions
Copy link

github-actions bot commented Mar 12, 2022

Coverage report for weather_forecast/gwd/ use-case

File Coverage Missing
All files 94%
data.py 98% 157
models.py 89% 37-40 90 99
trainer.py 88% 75-76 80

Minimum allowed coverage is 80%

Generated by 🐒 cobertura-action against 0de4a87

@github-actions
Copy link

github-actions bot commented Mar 12, 2022

Coverage report for combustion/gnns/ use-case

File Coverage Missing
All files 80%
config.py 66% 74-75 78-82 87-92 97-104
data.py 79% 116-143 174 229 276 287
models.py 68% 85 116-131 156-160 210-214 234-238
plotters.py 98% 55-56
trainer.py 76% 70-75 79-80

Minimum allowed coverage is 80%

Generated by 🐒 cobertura-action against 0de4a87

@A669015
Copy link
Contributor

A669015 commented Mar 17, 2022

There is a pycache file that should not be pushed.

# See the License for the specific language governing permissions and
# limitations under the License.

fit:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the UC no more functional on this file ?

# See the License for the specific language governing permissions and
# limitations under the License.

- DNS1_00116000.h5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the UC no more functional on these data ?
Could we support the both dataset ? Maybe having several filenames.yaml files ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i agree. both should coexist.

Copy link
Contributor

@alxyok alxyok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

combustion/pycache/init.cpython-38.pyc should also be removed?

num_workers: 0
y_normalizer : 342.553
y_normalizer: 3295.4
origin_path: '/wksp/data/raise/R2_flame/combustiondatabase/R2-filtered/R2-data'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this system-dependent?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The config file are always system-dependent IMHO since they are an extension of the CLI... they are logged in git only for educational purpose.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't agree. the rest of the arguments from those files can be run anywhere with no changes. not that one.
i don't think origin_path should be in the config files, the logic should be pulled out from the main flow

Creates the main Dataset and splits the train, test and validation Datasets from the main
Dataset. Currently the repartition is respectively, 80%, 10% and 10% from the main Dataset
size.
size. Creates symbolic links from origin data.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is an optional preprocessing step.
it should be run outside the main logic from a bash script maybe.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with that, we should find a way to keep that out of the core code, it's config-oriented.

self.local_raw_data = os.path.join(data_path, 'raw')
if os.path.exists(self.local_raw_data):

try :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be mv outside the main execution flow

# See the License for the specific language governing permissions and
# limitations under the License.

- DNS1_00116000.h5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i agree. both should coexist.




class R2Process(CombustionDataset) :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather call it R2Dataset (I think the name should rather denote the fact that it's a dataset)



def get(self, idx: int) -> pyg.data.Data:
class CnfProcess(CombustionDataset) :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Contributor

@leonicoletti leonicoletti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the subclassing of CombustionDataset.

num_workers: 0
y_normalizer : 342.553
y_normalizer: 3295.4
origin_path: '/wksp/data/raise/R2_flame/combustiondatabase/R2-filtered/R2-data'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't agree. the rest of the arguments from those files can be run anywhere with no changes. not that one.
i don't think origin_path should be in the config files, the logic should be pulled out from the main flow

accelerator: "gpu"
devices:
- 0 No newline at end of file
callbacks:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea indeed


with h5py.File(raw_path, 'r') as file:
c = file["/filt_8"][:]
c = file["/c_filt"][:]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
c = file["/c_filt"][:]
c = file["/c_filt"][:]
sigma = file["/c_grad_filt"][:]
if self.y_normalizer:
sigma /= self.y_normalizer

@rmslm rmslm force-pushed the feature/SBRA-360-training-on-R2-combustion branch from bf8547d to 063249c Compare March 28, 2022 10:31
@github-actions
Copy link

github-actions bot commented Mar 28, 2022

Coverage report for combustion/unets/ use-case

File Coverage Missing
All files 93%
data.py 93% 61 72 74 113 116
models.py 87% 41-42 45 48
trainer.py 77% 59-61 65-66 69

Minimum allowed coverage is 80%

Generated by 🐒 cobertura-action against 0de4a87

@rmslm rmslm merged commit 531185a into main Apr 8, 2022
@rmslm rmslm deleted the feature/SBRA-360-training-on-R2-combustion branch April 8, 2022 08:06
ai4sim-pr-app bot pushed a commit that referenced this pull request Feb 27, 2026
github-actions bot pushed a commit that referenced this pull request Mar 2, 2026
* Adding pangu weather (#1)

* re-add pangu-weather after branch cleanup

* reduced spatial resolution for tests

* fixed isort issue

* fixed black issue

* fixed accelerator issue

* added .coveragerc and .gitignore

* removed unnecessary requirements, and testing github repository name

* changed pangu-weather to global-emulation, updated requirement with pip freeze, added global-emulation to images-ci-cd matrix

* changed image ci cd to only push the Docker images in model-collection

* fixed requirements, removed weatherlearn mentions from readme, renamed pangu into panguweather

* added nox back in requirements

* fixed panguweather import, removed unused imports

* removed debug test from images-ci-cd, deleted config_pangu_micro, updated DropPath to use timm directly, removed unused models (MockPanguModel and CustomPanguModel), removed generate_3d_attention_mask_old and its associated test, aligned trainer.py with the others in weather-forecast

* removed unused Dockerfiles

* added git to dockerfile

* made the init of transformer_module more pythonic

* renamed global emulation to nwp emulation

* renamed global emulation to nwp emulation in the cicd

* [SBRA-837] Add mypy automatic checking to CI/CD (#4)

* Add a common mypy.ini file

* Add specific mypy config for pangu, Add mypy session to nox

* add mypy_cache to gitignore

* Add mypy session to ci/cd

* Improve PanguWeather type hints

* fix pangu lint

* Add mypy to gravity-wave-drag

* fix test_utils import

* fix unit test on data

* Add mypy checks to ecrad-3d-correction

* Add mypy checks to combustion unet

* Add mypy check to combustion gnn

* fix format

* fix unit test wrt previous code modification

* [SBRA-870] Polish code for open access release (#5)

* fix time_horizon error print

* fix typo

* fix casual comments

* remove useless comments

* improve comment

* improve print info

---------

Co-authored-by: mikael jacquemont <mikael.jacquemont@eviden.com>

* fix format after black update (#6)

* Bump mlflow from 2.17.2 to 2.20.3 in /weather-forecast/nwp-emulation/panguweather in the pip group across 1 directory (#3)

* Bump mlflow

Bumps the pip group with 1 update in the /weather-forecast/nwp-emulation/panguweather directory: [mlflow](https://github.com/mlflow/mlflow).


Updates `mlflow` from 2.17.2 to 2.20.3
- [Release notes](https://github.com/mlflow/mlflow/releases)
- [Changelog](https://github.com/mlflow/mlflow/blob/master/CHANGELOG.md)
- [Commits](mlflow/mlflow@v2.17.2...v2.20.3)

---
updated-dependencies:
- dependency-name: mlflow
  dependency-version: 2.20.3
  dependency-type: direct:production
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <support@github.com>

* Apply black

* Fix mlflow-skinny version

* Revert "Fix mlflow-skinny version"

This reverts commit 1bf04081ea5f324e91993f9ac24cf718dac5a6f7.

* Revert "Apply black"

This reverts commit 5c1ab8075cb29dfa13694168f3048dff1b8d62f5.

* Fix mlflow-skinny version

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mikaël Jacquemont <mikael.jacquemont@atos.net>

* Refactor projects install (#8)

* use uv instead of pip and python

* Let uv manage python and dependencies

* fix python version for all nox sessions

* fix find wheel version

* remove useless uv sync

* update pangu

* update cnns

* update ecrad unets

* update cnf unets

* update gnns

* update ci/cd workflow

* update docker build

* update doc

* use large runner for docker image workflow

* add dependabot.yml file

* Bump the uv group across 4 directories with 1 update (#9)

* Bump the uv group across 4 directories with 1 update

Bumps the uv group with 1 update in the /weather-forecast/nwp-emulation/panguweather directory: [torch](https://github.com/pytorch/pytorch).
Bumps the uv group with 1 update in the /weather-forecast/gravity-wave-drag/cnns directory: [torch](https://github.com/pytorch/pytorch).
Bumps the uv group with 1 update in the /weather-forecast/ecrad-3d-correction/unets directory: [torch](https://github.com/pytorch/pytorch).
Bumps the uv group with 1 update in the /reactive-flows/cnf-combustion/unets directory: [torch](https://github.com/pytorch/pytorch).


Updates `torch` from 2.6.0 to 2.8.0
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](pytorch/pytorch@v2.6.0...v2.8.0)

Updates `torch` from 2.6.0 to 2.8.0
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](pytorch/pytorch@v2.6.0...v2.8.0)

Updates `torch` from 2.6.0 to 2.8.0
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](pytorch/pytorch@v2.6.0...v2.8.0)

Updates `torch` from 2.6.0 to 2.8.0
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](pytorch/pytorch@v2.6.0...v2.8.0)

---
updated-dependencies:
- dependency-name: torch
  dependency-version: 2.8.0
  dependency-type: direct:production
  dependency-group: uv
- dependency-name: torch
  dependency-version: 2.8.0
  dependency-type: direct:production
  dependency-group: uv
- dependency-name: torch
  dependency-version: 2.8.0
  dependency-type: direct:production
  dependency-group: uv
- dependency-name: torch
  dependency-version: 2.8.0
  dependency-type: direct:production
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix mypy issue

* replacing float casting by item()

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: VINCENT, LIONEL <lionel.vincent.dm@atos.net>
Co-authored-by: mikael10j <mikael.jacquemont@eviden.com>

* improve install using explicit index and find-link for torch and pyg (#10)

* Bump the uv group across 5 directories with 8 updates (#11)

Bumps the uv group with 8 updates in the /reactive-flows/cnf-combustion/gnns directory:

| Package | From | To |
| --- | --- | --- |
| [certifi](https://github.com/certifi/python-certifi) | `2022.12.7` | `2024.7.4` |
| [filelock](https://github.com/tox-dev/py-filelock) | `3.20.0` | `3.20.3` |
| [idna](https://github.com/kjd/idna) | `3.4` | `3.7` |
| [pillow](https://github.com/python-pillow/Pillow) | `12.0.0` | `12.1.1` |
| [requests](https://github.com/psf/requests) | `2.28.1` | `2.32.4` |
| [setuptools](https://github.com/pypa/setuptools) | `70.2.0` | `78.1.1` |
| [urllib3](https://github.com/urllib3/urllib3) | `1.26.13` | `2.6.3` |
| [werkzeug](https://github.com/pallets/werkzeug) | `3.1.5` | `3.1.6` |

Bumps the uv group with 1 update in the /reactive-flows/cnf-combustion/unets directory: [werkzeug](https://github.com/pallets/werkzeug).
Bumps the uv group with 1 update in the /weather-forecast/ecrad-3d-correction/unets directory: [werkzeug](https://github.com/pallets/werkzeug).
Bumps the uv group with 4 updates in the /weather-forecast/gravity-wave-drag/cnns directory: [filelock](https://github.com/tox-dev/py-filelock), [idna](https://github.com/kjd/idna), [setuptools](https://github.com/pypa/setuptools) and [werkzeug](https://github.com/pallets/werkzeug).
Bumps the uv group with 5 updates in the /weather-forecast/nwp-emulation/panguweather directory:

| Package | From | To |
| --- | --- | --- |
| [certifi](https://github.com/certifi/python-certifi) | `2022.12.7` | `2024.7.4` |
| [filelock](https://github.com/tox-dev/py-filelock) | `3.20.0` | `3.20.3` |
| [idna](https://github.com/kjd/idna) | `3.4` | `3.7` |
| [pillow](https://github.com/python-pillow/Pillow) | `12.0.0` | `12.1.1` |
| [setuptools](https://github.com/pypa/setuptools) | `70.2.0` | `78.1.1` |



Updates `certifi` from 2022.12.7 to 2024.7.4
- [Commits](certifi/python-certifi@2022.12.07...2024.07.04)

Updates `filelock` from 3.20.0 to 3.20.3
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst)
- [Commits](tox-dev/filelock@3.20.0...3.20.3)

Updates `idna` from 3.4 to 3.7
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v3.4...v3.7)

Updates `pillow` from 12.0.0 to 12.1.1
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@12.0.0...12.1.1)

Updates `requests` from 2.28.1 to 2.32.4
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.28.1...v2.32.4)

Updates `setuptools` from 70.2.0 to 78.1.1
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](pypa/setuptools@v70.2.0...v78.1.1)

Updates `urllib3` from 1.26.13 to 2.6.3
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.13...2.6.3)

Updates `werkzeug` from 3.1.5 to 3.1.6
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.1.5...3.1.6)

Updates `werkzeug` from 3.1.5 to 3.1.6
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.1.5...3.1.6)

Updates `werkzeug` from 3.1.5 to 3.1.6
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.1.5...3.1.6)

Updates `filelock` from 3.20.0 to 3.20.3
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst)
- [Commits](tox-dev/filelock@3.20.0...3.20.3)

Updates `idna` from 3.4 to 3.7
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v3.4...v3.7)

Updates `setuptools` from 70.2.0 to 78.1.1
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](pypa/setuptools@v70.2.0...v78.1.1)

Updates `werkzeug` from 3.1.5 to 3.1.6
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.1.5...3.1.6)

Updates `certifi` from 2022.12.7 to 2024.7.4
- [Commits](certifi/python-certifi@2022.12.07...2024.07.04)

Updates `filelock` from 3.20.0 to 3.20.3
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst)
- [Commits](tox-dev/filelock@3.20.0...3.20.3)

Updates `idna` from 3.4 to 3.7
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v3.4...v3.7)

Updates `pillow` from 12.0.0 to 12.1.1
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@12.0.0...12.1.1)

Updates `setuptools` from 70.2.0 to 78.1.1
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](pypa/setuptools@v70.2.0...v78.1.1)

---
updated-dependencies:
- dependency-name: certifi
  dependency-version: 2024.7.4
  dependency-type: indirect
  dependency-group: uv
- dependency-name: filelock
  dependency-version: 3.20.3
  dependency-type: indirect
  dependency-group: uv
- dependency-name: idna
  dependency-version: '3.7'
  dependency-type: indirect
  dependency-group: uv
- dependency-name: pillow
  dependency-version: 12.1.1
  dependency-type: indirect
  dependency-group: uv
- dependency-name: requests
  dependency-version: 2.32.4
  dependency-type: indirect
  dependency-group: uv
- dependency-name: setuptools
  dependency-version: 78.1.1
  dependency-type: indirect
  dependency-group: uv
- dependency-name: urllib3
  dependency-version: 2.6.3
  dependency-type: indirect
  dependency-group: uv
- dependency-name: werkzeug
  dependency-version: 3.1.6
  dependency-type: indirect
  dependency-group: uv
- dependency-name: werkzeug
  dependency-version: 3.1.6
  dependency-type: indirect
  dependency-group: uv
- dependency-name: werkzeug
  dependency-version: 3.1.6
  dependency-type: indirect
  dependency-group: uv
- dependency-name: filelock
  dependency-version: 3.20.3
  dependency-type: indirect
  dependency-group: uv
- dependency-name: idna
  dependency-version: '3.7'
  dependency-type: indirect
  dependency-group: uv
- dependency-name: setuptools
  dependency-version: 78.1.1
  dependency-type: indirect
  dependency-group: uv
- dependency-name: werkzeug
  dependency-version: 3.1.6
  dependency-type: indirect
  dependency-group: uv
- dependency-name: certifi
  dependency-version: 2024.7.4
  dependency-type: indirect
  dependency-group: uv
- dependency-name: filelock
  dependency-version: 3.20.3
  dependency-type: indirect
  dependency-group: uv
- dependency-name: idna
  dependency-version: '3.7'
  dependency-type: indirect
  dependency-group: uv
- dependency-name: pillow
  dependency-version: 12.1.1
  dependency-type: indirect
  dependency-group: uv
- dependency-name: setuptools
  dependency-version: 78.1.1
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix README (#12)

* fix README

* Add missing line

Co-authored-by: Mikaël Jacquemont <mikael.jacquemont@atos.net>

---------

Co-authored-by: Mikaël Jacquemont <mikael.jacquemont@atos.net>

* add workflow to automate PR proposal in public when merging in private (#13)

* fix branch name (#14)

* fix dependabot.yaml (#15)

* Automate PR proposal in public - fix credentials  (#16)

* fix job name

* Add branch and PR creation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: MarcMichouxEviden <106808701+MarcMichouxAtos@users.noreply.github.com>
Co-authored-by: Mikaël Jacquemont <mikael.jacquemont@atos.net>
Co-authored-by: mikael jacquemont <mikael.jacquemont@eviden.com>
Co-authored-by: Lionel VINCENT <97742788+A669015@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: VINCENT, LIONEL <lionel.vincent.dm@atos.net>
github-actions bot pushed a commit that referenced this pull request Mar 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants