chore(ci): migrate compatibility indexes from git branch to GitHub Releases (0.16)#1754
chore(ci): migrate compatibility indexes from git branch to GitHub Releases (0.16)#1754LHT129 merged 1 commit intoantgroup:0.16from
Conversation
…leases - Modify generate_old_version_index.yml to upload indexes to Release - Modify check_compatibility.yml to download indexes from Release - Remove git clone of old_version_index branch (961 MB) - Use gh release download/upload for faster CI execution - Add --prerelease flag when creating new release - Improve shell script safety with bash arrays Backport of antgroup#1692 and antgroup#1708 to 0.16 branch Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
|
Note Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported. |
There was a problem hiding this comment.
Pull request overview
Migrates compatibility test index artifacts from a large old_version_index git branch to a GitHub Release (compatibility-indexes) to reduce CI time and repository cloning overhead on the 0.16 branch.
Changes:
- Update index generation workflow to upload generated artifacts to a GitHub Release (and download the dataset from it when available).
- Update compatibility check workflow to download indexes from the GitHub Release instead of cloning a dedicated branch.
- Adjust concurrency grouping to work for both PR and non-PR events.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| .github/workflows/generate_old_version_index.yml | Replaces index-branch checkout + PR creation with gh release download/upload/create flow. |
| .github/workflows/check_compatibility.yml | Replaces cloning old_version_index branch with downloading artifacts from the Release; updates concurrency key. |
Comments suppressed due to low confidence (1)
.github/workflows/generate_old_version_index.yml:1
- Previously this workflow copied
create_old_version_index.cppinto the checked-out tag and amendedtools/CMakeLists.txtbefore building. Those steps were removed, but the workflow still executes./build-release/tools/create_old_version_index. If older tags don’t contain/build this tool,make releasewon’t produce the binary and the job will fail. Consider restoring the logic that injects/builds the tool (e.g., copy it from the workflow’s own source checkout) or build/run the generator from a stable revision independent of the target tag.
name: Generate Old Tag Vesion Index
| - name: Install GitHub CLI | ||
| run: | | ||
| (type -p wget >/dev/null || (apt update && apt-get install wget -y)) \ | ||
| && mkdir -p -m 755 /etc/apt/keyrings \ | ||
| && wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | tee /etc/apt/keyrings/githubcli.gpg >/dev/null \ | ||
| && chmod go+r /etc/apt/keyrings/githubcli.gpg \ | ||
| && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list >/dev/null \ | ||
| && apt update \ | ||
| && apt install gh -y |
There was a problem hiding this comment.
Installing gh via apt in the job adds a network- and distro-dependent dependency (APT repo availability, keyserver/HTTPS, base image compatibility) and repeats on every run, which can cause flaky CI and increased runtime. Prefer using a pinned standalone gh binary install, or baking gh into the vsaglib/vsag:ci-x86 image (or using a dedicated setup action) to make the workflow more deterministic and faster.
| - name: Install GitHub CLI | |
| run: | | |
| (type -p wget >/dev/null || (apt update && apt-get install wget -y)) \ | |
| && mkdir -p -m 755 /etc/apt/keyrings \ | |
| && wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | tee /etc/apt/keyrings/githubcli.gpg >/dev/null \ | |
| && chmod go+r /etc/apt/keyrings/githubcli.gpg \ | |
| && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list >/dev/null \ | |
| && apt update \ | |
| && apt install gh -y | |
| - name: Setup GitHub CLI | |
| uses: cli/cli-action@v2 | |
| with: | |
| version: '2.58.0' |
| gh release download compatibility-indexes \ | ||
| --repo antgroup/vsag \ | ||
| --dir /tmp \ | ||
| --pattern "*.index" \ | ||
| --pattern "*.bin" \ | ||
| --pattern "*.json" \ | ||
| || echo "Warning: Failed to download compatibility indexes" |
There was a problem hiding this comment.
This step masks any download failure and continues the job. If the compatibility check requires these artifacts, the job may later fail with less actionable errors (or behave differently than intended). Consider failing fast when downloads fail, or explicitly verifying that expected files exist in /tmp and exiting with a clear error if they don’t.
| gh release download compatibility-indexes \ | |
| --repo antgroup/vsag \ | |
| --dir /tmp \ | |
| --pattern "*.index" \ | |
| --pattern "*.bin" \ | |
| --pattern "*.json" \ | |
| || echo "Warning: Failed to download compatibility indexes" | |
| set -euo pipefail | |
| gh release download compatibility-indexes \ | |
| --repo antgroup/vsag \ | |
| --dir /tmp \ | |
| --pattern "*.index" \ | |
| --pattern "*.bin" \ | |
| --pattern "*.json" | |
| # Verify that at least one expected compatibility index artifact was downloaded | |
| if ! ls /tmp/*.index /tmp/*.bin /tmp/*.json >/dev/null 2>&1; then | |
| echo "Error: Failed to find downloaded compatibility index artifacts in /tmp." | |
| echo "Ensure the 'compatibility-indexes' release contains *.index, *.bin, or *.json files." | |
| exit 1 | |
| fi |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| gh release download compatibility-indexes \ | ||
| --repo antgroup/vsag \ |
There was a problem hiding this comment.
The repository is hard-coded as antgroup/vsag. Using ${{ github.repository }} (or a single workflow-level env var) would avoid duplication across workflows and keep the workflows working if the repo is renamed or the workflow is reused in a different repository.
| --repo antgroup/vsag \ | |
| --repo ${{ github.repository }} \ |
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## 0.16 #1754 +/- ##
==========================================
- Coverage 92.08% 92.02% -0.06%
==========================================
Files 295 295
Lines 15708 15708
==========================================
- Hits 14464 14456 -8
- Misses 1244 1252 +8
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Summary
Backport of #1692 and #1708 to 0.16 branch.
Migrate compatibility test index files from the
old_version_indexgit branch to GitHub Releases for improved CI performance and maintainability.Changes
generate_old_version_index.ymlto upload indexes to Release instead of creating PRcheck_compatibility.ymlto download indexes from Release instead of git cloneold_version_indexbranch (961 MB)gh release download/uploadcommands for faster CI execution--prereleaseflag when creating new releaseExpected Improvements
Related Issues