Skip to content

Conversation

@zees-dev
Copy link
Collaborator

@zees-dev zees-dev commented Feb 18, 2025

Description

This PR introduces the LLM module for host runtime as a driver.
Llamafile is used as the LLM provider to download models and provide inference on demand for guest wasm apps.

Other changes

  • Added unit test for llamafile to test e2e lifecycle
  • Cargo.toml updates
  • cargo fmt --all
  • Added a generic HandleMap as a singleton for handling instances of LLM models loaded in guest wasm apps
  • Added rust-toolchain.toml to repo

Dependent PRs

@coveralls
Copy link

Pull Request Test Coverage Report for Build 13380616647

Details

  • 79 of 289 (27.34%) changed or added relevant lines in 5 files are covered.
  • 24 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.02%) to 15.066%

Changes Missing Coverage Covered Lines Changed/Added Lines %
crates/blockless-drivers/src/error.rs 0 1 0.0%
crates/blockless-drivers/src/llm_driver/models.rs 27 100 27.0%
crates/blockless-drivers/src/llm_driver/llamafile.rs 7 143 4.9%
Files with Coverage Reduction New Missed Lines %
../../../.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-core-0.1.33/src/field.rs 3 43.75%
../../../.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-core-0.1.33/src/metadata.rs 6 50.0%
../../../.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-core-0.1.33/src/subscriber.rs 6 16.67%
../../../.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-core-0.1.33/src/dispatcher.rs 9 25.0%
Totals Coverage Status
Change from base Build 12861905583: 0.02%
Covered Lines: 6078
Relevant Lines: 12445

💛 - Coveralls

@Joinhack Joinhack merged commit 68ca215 into main Mar 19, 2025
5 checks passed
@zees-dev zees-dev deleted the llm-module-driver branch March 19, 2025 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants