Integrate cbq

[noamteyssier]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @noamteyssier, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the bqtools utility by integrating the new CBQ BINSEQ format, which offers superior compression and flexibility for DNA sequencing data. The underlying encoding architecture has been unified and simplified, leading to a more robust and extensible codebase. Additionally, the existing count command has been upgraded to a comprehensive info command, providing richer metadata and statistics for all supported BINSEQ file types.

Highlights

• New BINSEQ Format Integration (CBQ): The pull request introduces and fully integrates the new CBQ (Columnar BINSEQ) format, which is now the recommended default for most applications due to its improved compression, lossless nature, and support for variable-length sequences. This includes updates across the CLI, encoding, and file handling logic.
• Refactored Encoding Logic: The internal encoding mechanism has been significantly refactored. The previous BinseqProcessor and VBinseqProcessor have been replaced by a unified Encoder struct and BinseqWriterBuilder, streamlining the process for all BINSEQ formats (BQ, VBQ, CBQ) and improving maintainability.
• Enhanced Information Command: The count command has been renamed to info and significantly expanded. It now provides detailed statistics and metadata for BQ, VBQ, and the new CBQ files, including options to display index information and CBQ block headers.
• Dependency Updates and Version Bump: The project's version has been bumped to 0.5.0, accompanied by updates to several core dependencies such as binseq, clap, libc, nix, paraseq, bon, and tempfile, ensuring compatibility and leveraging the latest features.
• README Documentation Updates: The README.md has been thoroughly updated to reflect the new CBQ format, its advantages, and the deprecation of VBQ as the primary variable-length format. It also updates command examples to use the new info command.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request is a significant and well-executed refactoring to integrate the new cbq format. The introduction of a generic Encoder and the use of BinseqWriterBuilder greatly simplify the codebase and reduce duplication, which is excellent. The CLI is also updated consistently with the new info command and updated options. I have a few suggestions to further improve the code, mainly regarding some leftover debugging settings, commented-out code, and opportunities to reduce duplication.

I am having trouble creating individual review comments. Click here to see my feedback.

Cargo.toml (50-52)

These changes to the release profile appear to be for debugging. Enabling debug symbols (debug = true) in a release build is generally not recommended for production as it increases binary size and can have a minor performance impact. More importantly, commenting out lto = true and codegen-units = 1 will likely cause a significant performance regression in release builds. These settings are crucial for optimizations. Please revert these changes before merging.

lto = true
codegen-units = 1

src/cli/output.rs (321-325)

The enum variants Binseq and VBinseq are being renamed to Bq and Vbq respectively. This is a good change for consistency. However, the old names are still used in BinseqMode::determine. Please update them to use the new variant names.

    Binseq,
    #[clap(name = "vbq")]
    #[default]
    Vbinseq,

README.md (11)

This update is correct. However, please also update line 14, which still incorrectly states that there are only two variants. It should be updated to mention three variants now that cbq is supported.

src/cli/info.rs (6-7)

The struct was correctly renamed to InfoCommand, but its documentation comment is outdated and still describes the old CountCommand. Please update the documentation to reflect the new purpose of showing information and statistics about a BINSEQ file.

/// Show information and statistics about a BINSEQ file.
pub struct InfoCommand {

src/commands/cat/mod.rs (126-142)

There's a good amount of code duplication in this module that could be refactored.

• The functions run_cbq (here) and run_vbq (lines 108-124) are almost identical. The common logic for processing files could be extracted into a helper function that accepts a BinseqWriter.
• Similarly, record_cbq_header (lines 91-106) and record_vbq_header (lines 74-89) are duplicates. This could be solved with a generic function.

Reducing this duplication would make the code more maintainable. For example, run_vbq and run_cbq could be simplified by using a shared helper:

fn run_cat_variable<W: std::io::Write + Send>(
    args: &CatCommand,
    writer: binseq::BinseqWriter<W>,
) -> Result<()> {
    let mut processor = Encoder::new(writer)?;
    for path in &args.input.input {
        let reader = BinseqReader::new(path)?;
        reader.process_parallel(processor.clone(), args.output.threads())?;
    }
    processor.finish()?;
    Ok(())
}

src/commands/encode/encode.rs (185-284)

This large block of commented-out code appears to be from the previous implementation. It's great that you've refactored it into a more generic solution. To keep the codebase clean, please remove this commented-out code before merging.

src/types.rs (3-4)

The BoxedWriter type alias is now marked with #[allow(unused)]. If it's no longer used anywhere in the crate due to the refactoring, it would be cleaner to remove it entirely rather than suppressing the warning.