Conversation
|
Coverage report for combustion/gnns/ use-case
Minimum allowed coverage is Generated by 🐒 cobertura-action against 0de4a87 |
|
There is a pycache file that should not be pushed. |
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| fit: |
There was a problem hiding this comment.
Is the UC no more functional on this file ?
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| - DNS1_00116000.h5 |
There was a problem hiding this comment.
Is the UC no more functional on these data ?
Could we support the both dataset ? Maybe having several filenames.yaml files ?
There was a problem hiding this comment.
i agree. both should coexist.
alxyok
left a comment
There was a problem hiding this comment.
combustion/pycache/init.cpython-38.pyc should also be removed?
combustion/gnns/configs/gat.yaml
Outdated
| num_workers: 0 | ||
| y_normalizer : 342.553 | ||
| y_normalizer: 3295.4 | ||
| origin_path: '/wksp/data/raise/R2_flame/combustiondatabase/R2-filtered/R2-data' |
There was a problem hiding this comment.
The config file are always system-dependent IMHO since they are an extension of the CLI... they are logged in git only for educational purpose.
There was a problem hiding this comment.
i don't agree. the rest of the arguments from those files can be run anywhere with no changes. not that one.
i don't think origin_path should be in the config files, the logic should be pulled out from the main flow
| Creates the main Dataset and splits the train, test and validation Datasets from the main | ||
| Dataset. Currently the repartition is respectively, 80%, 10% and 10% from the main Dataset | ||
| size. | ||
| size. Creates symbolic links from origin data. |
There was a problem hiding this comment.
this is an optional preprocessing step.
it should be run outside the main logic from a bash script maybe.
There was a problem hiding this comment.
I agree with that, we should find a way to keep that out of the core code, it's config-oriented.
combustion/gnns/data.py
Outdated
| self.local_raw_data = os.path.join(data_path, 'raw') | ||
| if os.path.exists(self.local_raw_data): | ||
|
|
||
| try : |
There was a problem hiding this comment.
this should be mv outside the main execution flow
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| - DNS1_00116000.h5 |
There was a problem hiding this comment.
i agree. both should coexist.
combustion/gnns/data.py
Outdated
|
|
||
|
|
||
|
|
||
| class R2Process(CombustionDataset) : |
There was a problem hiding this comment.
I would rather call it R2Dataset (I think the name should rather denote the fact that it's a dataset)
combustion/gnns/data.py
Outdated
|
|
||
|
|
||
| def get(self, idx: int) -> pyg.data.Data: | ||
| class CnfProcess(CombustionDataset) : |
leonicoletti
left a comment
There was a problem hiding this comment.
I like the subclassing of CombustionDataset.
combustion/gnns/configs/gat.yaml
Outdated
| num_workers: 0 | ||
| y_normalizer : 342.553 | ||
| y_normalizer: 3295.4 | ||
| origin_path: '/wksp/data/raise/R2_flame/combustiondatabase/R2-filtered/R2-data' |
There was a problem hiding this comment.
i don't agree. the rest of the arguments from those files can be run anywhere with no changes. not that one.
i don't think origin_path should be in the config files, the logic should be pulled out from the main flow
| accelerator: "gpu" | ||
| devices: | ||
| - 0 No newline at end of file | ||
| callbacks: |
|
|
||
| with h5py.File(raw_path, 'r') as file: | ||
| c = file["/filt_8"][:] | ||
| c = file["/c_filt"][:] |
There was a problem hiding this comment.
| c = file["/c_filt"][:] | |
| c = file["/c_filt"][:] | |
| sigma = file["/c_grad_filt"][:] | |
| if self.y_normalizer: | |
| sigma /= self.y_normalizer |
bf8547d to
063249c
Compare
* Adding pangu weather (#1) * re-add pangu-weather after branch cleanup * reduced spatial resolution for tests * fixed isort issue * fixed black issue * fixed accelerator issue * added .coveragerc and .gitignore * removed unnecessary requirements, and testing github repository name * changed pangu-weather to global-emulation, updated requirement with pip freeze, added global-emulation to images-ci-cd matrix * changed image ci cd to only push the Docker images in model-collection * fixed requirements, removed weatherlearn mentions from readme, renamed pangu into panguweather * added nox back in requirements * fixed panguweather import, removed unused imports * removed debug test from images-ci-cd, deleted config_pangu_micro, updated DropPath to use timm directly, removed unused models (MockPanguModel and CustomPanguModel), removed generate_3d_attention_mask_old and its associated test, aligned trainer.py with the others in weather-forecast * removed unused Dockerfiles * added git to dockerfile * made the init of transformer_module more pythonic * renamed global emulation to nwp emulation * renamed global emulation to nwp emulation in the cicd * [SBRA-837] Add mypy automatic checking to CI/CD (#4) * Add a common mypy.ini file * Add specific mypy config for pangu, Add mypy session to nox * add mypy_cache to gitignore * Add mypy session to ci/cd * Improve PanguWeather type hints * fix pangu lint * Add mypy to gravity-wave-drag * fix test_utils import * fix unit test on data * Add mypy checks to ecrad-3d-correction * Add mypy checks to combustion unet * Add mypy check to combustion gnn * fix format * fix unit test wrt previous code modification * [SBRA-870] Polish code for open access release (#5) * fix time_horizon error print * fix typo * fix casual comments * remove useless comments * improve comment * improve print info --------- Co-authored-by: mikael jacquemont <mikael.jacquemont@eviden.com> * fix format after black update (#6) * Bump mlflow from 2.17.2 to 2.20.3 in /weather-forecast/nwp-emulation/panguweather in the pip group across 1 directory (#3) * Bump mlflow Bumps the pip group with 1 update in the /weather-forecast/nwp-emulation/panguweather directory: [mlflow](https://github.com/mlflow/mlflow). Updates `mlflow` from 2.17.2 to 2.20.3 - [Release notes](https://github.com/mlflow/mlflow/releases) - [Changelog](https://github.com/mlflow/mlflow/blob/master/CHANGELOG.md) - [Commits](mlflow/mlflow@v2.17.2...v2.20.3) --- updated-dependencies: - dependency-name: mlflow dependency-version: 2.20.3 dependency-type: direct:production dependency-group: pip ... Signed-off-by: dependabot[bot] <support@github.com> * Apply black * Fix mlflow-skinny version * Revert "Fix mlflow-skinny version" This reverts commit 1bf04081ea5f324e91993f9ac24cf718dac5a6f7. * Revert "Apply black" This reverts commit 5c1ab8075cb29dfa13694168f3048dff1b8d62f5. * Fix mlflow-skinny version --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mikaël Jacquemont <mikael.jacquemont@atos.net> * Refactor projects install (#8) * use uv instead of pip and python * Let uv manage python and dependencies * fix python version for all nox sessions * fix find wheel version * remove useless uv sync * update pangu * update cnns * update ecrad unets * update cnf unets * update gnns * update ci/cd workflow * update docker build * update doc * use large runner for docker image workflow * add dependabot.yml file * Bump the uv group across 4 directories with 1 update (#9) * Bump the uv group across 4 directories with 1 update Bumps the uv group with 1 update in the /weather-forecast/nwp-emulation/panguweather directory: [torch](https://github.com/pytorch/pytorch). Bumps the uv group with 1 update in the /weather-forecast/gravity-wave-drag/cnns directory: [torch](https://github.com/pytorch/pytorch). Bumps the uv group with 1 update in the /weather-forecast/ecrad-3d-correction/unets directory: [torch](https://github.com/pytorch/pytorch). Bumps the uv group with 1 update in the /reactive-flows/cnf-combustion/unets directory: [torch](https://github.com/pytorch/pytorch). Updates `torch` from 2.6.0 to 2.8.0 - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](pytorch/pytorch@v2.6.0...v2.8.0) Updates `torch` from 2.6.0 to 2.8.0 - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](pytorch/pytorch@v2.6.0...v2.8.0) Updates `torch` from 2.6.0 to 2.8.0 - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](pytorch/pytorch@v2.6.0...v2.8.0) Updates `torch` from 2.6.0 to 2.8.0 - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](pytorch/pytorch@v2.6.0...v2.8.0) --- updated-dependencies: - dependency-name: torch dependency-version: 2.8.0 dependency-type: direct:production dependency-group: uv - dependency-name: torch dependency-version: 2.8.0 dependency-type: direct:production dependency-group: uv - dependency-name: torch dependency-version: 2.8.0 dependency-type: direct:production dependency-group: uv - dependency-name: torch dependency-version: 2.8.0 dependency-type: direct:production dependency-group: uv ... Signed-off-by: dependabot[bot] <support@github.com> * fix mypy issue * replacing float casting by item() --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: VINCENT, LIONEL <lionel.vincent.dm@atos.net> Co-authored-by: mikael10j <mikael.jacquemont@eviden.com> * improve install using explicit index and find-link for torch and pyg (#10) * Bump the uv group across 5 directories with 8 updates (#11) Bumps the uv group with 8 updates in the /reactive-flows/cnf-combustion/gnns directory: | Package | From | To | | --- | --- | --- | | [certifi](https://github.com/certifi/python-certifi) | `2022.12.7` | `2024.7.4` | | [filelock](https://github.com/tox-dev/py-filelock) | `3.20.0` | `3.20.3` | | [idna](https://github.com/kjd/idna) | `3.4` | `3.7` | | [pillow](https://github.com/python-pillow/Pillow) | `12.0.0` | `12.1.1` | | [requests](https://github.com/psf/requests) | `2.28.1` | `2.32.4` | | [setuptools](https://github.com/pypa/setuptools) | `70.2.0` | `78.1.1` | | [urllib3](https://github.com/urllib3/urllib3) | `1.26.13` | `2.6.3` | | [werkzeug](https://github.com/pallets/werkzeug) | `3.1.5` | `3.1.6` | Bumps the uv group with 1 update in the /reactive-flows/cnf-combustion/unets directory: [werkzeug](https://github.com/pallets/werkzeug). Bumps the uv group with 1 update in the /weather-forecast/ecrad-3d-correction/unets directory: [werkzeug](https://github.com/pallets/werkzeug). Bumps the uv group with 4 updates in the /weather-forecast/gravity-wave-drag/cnns directory: [filelock](https://github.com/tox-dev/py-filelock), [idna](https://github.com/kjd/idna), [setuptools](https://github.com/pypa/setuptools) and [werkzeug](https://github.com/pallets/werkzeug). Bumps the uv group with 5 updates in the /weather-forecast/nwp-emulation/panguweather directory: | Package | From | To | | --- | --- | --- | | [certifi](https://github.com/certifi/python-certifi) | `2022.12.7` | `2024.7.4` | | [filelock](https://github.com/tox-dev/py-filelock) | `3.20.0` | `3.20.3` | | [idna](https://github.com/kjd/idna) | `3.4` | `3.7` | | [pillow](https://github.com/python-pillow/Pillow) | `12.0.0` | `12.1.1` | | [setuptools](https://github.com/pypa/setuptools) | `70.2.0` | `78.1.1` | Updates `certifi` from 2022.12.7 to 2024.7.4 - [Commits](certifi/python-certifi@2022.12.07...2024.07.04) Updates `filelock` from 3.20.0 to 3.20.3 - [Release notes](https://github.com/tox-dev/py-filelock/releases) - [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst) - [Commits](tox-dev/filelock@3.20.0...3.20.3) Updates `idna` from 3.4 to 3.7 - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst) - [Commits](kjd/idna@v3.4...v3.7) Updates `pillow` from 12.0.0 to 12.1.1 - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@12.0.0...12.1.1) Updates `requests` from 2.28.1 to 2.32.4 - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.28.1...v2.32.4) Updates `setuptools` from 70.2.0 to 78.1.1 - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@v70.2.0...v78.1.1) Updates `urllib3` from 1.26.13 to 2.6.3 - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@1.26.13...2.6.3) Updates `werkzeug` from 3.1.5 to 3.1.6 - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@3.1.5...3.1.6) Updates `werkzeug` from 3.1.5 to 3.1.6 - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@3.1.5...3.1.6) Updates `werkzeug` from 3.1.5 to 3.1.6 - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@3.1.5...3.1.6) Updates `filelock` from 3.20.0 to 3.20.3 - [Release notes](https://github.com/tox-dev/py-filelock/releases) - [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst) - [Commits](tox-dev/filelock@3.20.0...3.20.3) Updates `idna` from 3.4 to 3.7 - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst) - [Commits](kjd/idna@v3.4...v3.7) Updates `setuptools` from 70.2.0 to 78.1.1 - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@v70.2.0...v78.1.1) Updates `werkzeug` from 3.1.5 to 3.1.6 - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](pallets/werkzeug@3.1.5...3.1.6) Updates `certifi` from 2022.12.7 to 2024.7.4 - [Commits](certifi/python-certifi@2022.12.07...2024.07.04) Updates `filelock` from 3.20.0 to 3.20.3 - [Release notes](https://github.com/tox-dev/py-filelock/releases) - [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst) - [Commits](tox-dev/filelock@3.20.0...3.20.3) Updates `idna` from 3.4 to 3.7 - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst) - [Commits](kjd/idna@v3.4...v3.7) Updates `pillow` from 12.0.0 to 12.1.1 - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@12.0.0...12.1.1) Updates `setuptools` from 70.2.0 to 78.1.1 - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@v70.2.0...v78.1.1) --- updated-dependencies: - dependency-name: certifi dependency-version: 2024.7.4 dependency-type: indirect dependency-group: uv - dependency-name: filelock dependency-version: 3.20.3 dependency-type: indirect dependency-group: uv - dependency-name: idna dependency-version: '3.7' dependency-type: indirect dependency-group: uv - dependency-name: pillow dependency-version: 12.1.1 dependency-type: indirect dependency-group: uv - dependency-name: requests dependency-version: 2.32.4 dependency-type: indirect dependency-group: uv - dependency-name: setuptools dependency-version: 78.1.1 dependency-type: indirect dependency-group: uv - dependency-name: urllib3 dependency-version: 2.6.3 dependency-type: indirect dependency-group: uv - dependency-name: werkzeug dependency-version: 3.1.6 dependency-type: indirect dependency-group: uv - dependency-name: werkzeug dependency-version: 3.1.6 dependency-type: indirect dependency-group: uv - dependency-name: werkzeug dependency-version: 3.1.6 dependency-type: indirect dependency-group: uv - dependency-name: filelock dependency-version: 3.20.3 dependency-type: indirect dependency-group: uv - dependency-name: idna dependency-version: '3.7' dependency-type: indirect dependency-group: uv - dependency-name: setuptools dependency-version: 78.1.1 dependency-type: indirect dependency-group: uv - dependency-name: werkzeug dependency-version: 3.1.6 dependency-type: indirect dependency-group: uv - dependency-name: certifi dependency-version: 2024.7.4 dependency-type: indirect dependency-group: uv - dependency-name: filelock dependency-version: 3.20.3 dependency-type: indirect dependency-group: uv - dependency-name: idna dependency-version: '3.7' dependency-type: indirect dependency-group: uv - dependency-name: pillow dependency-version: 12.1.1 dependency-type: indirect dependency-group: uv - dependency-name: setuptools dependency-version: 78.1.1 dependency-type: indirect dependency-group: uv ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix README (#12) * fix README * Add missing line Co-authored-by: Mikaël Jacquemont <mikael.jacquemont@atos.net> --------- Co-authored-by: Mikaël Jacquemont <mikael.jacquemont@atos.net> * add workflow to automate PR proposal in public when merging in private (#13) * fix branch name (#14) * fix dependabot.yaml (#15) * Automate PR proposal in public - fix credentials (#16) * fix job name * Add branch and PR creation --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: MarcMichouxEviden <106808701+MarcMichouxAtos@users.noreply.github.com> Co-authored-by: Mikaël Jacquemont <mikael.jacquemont@atos.net> Co-authored-by: mikael jacquemont <mikael.jacquemont@eviden.com> Co-authored-by: Lionel VINCENT <97742788+A669015@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: VINCENT, LIONEL <lionel.vincent.dm@atos.net>
couple of refactoring R2 and cnf for gnns
Create symbolic links to data where ever the data are on Atos clusters to avoid duplicating large data
config files can take the link of the data as parameter