[multimodal] Add token support to MultimodalInput

[larryliu0820]

This pull request adds support for tokenizer-encoded input (as vectors of token IDs) to the MultimodalInput class, enabling more flexible and efficient handling of multimodal data. The update includes new constructors, type checks, getters, and factory functions for token inputs, as well as unit tests to ensure correct behavior and compatibility with existing code paths.

MultimodalInput class changes:

• Added a new TOKENS type to the MultimodalInput::Type enum and updated the internal std::variant to support storing std::vector<uint64_t> as token data. [1] [2]
• Implemented new constructors, type checks (is_tokens()), getters (get_tokens()), and safe accessors (try_get_tokens()) for token inputs, along with static and instance methods for type name conversion. [1] [2] [3] [4] [5]
• Added factory functions make_token_input for easily creating token-based inputs.

Integration and logging:

• Updated MultimodalPrefiller::prefill to handle both text and token inputs, bypassing tokenization when tokens are provided directly.
• Added logging in MultimodalRunner::generate to include the type name of each input for easier debugging.

Tests:

• Introduced a comprehensive suite of unit tests covering construction, type checking, getters, copy/move semantics, and edge cases for the new token input functionality in MultimodalInput.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14451

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 7 Pending

As of commit 71dea2a with merge base 07d1092 ():

NEW FAILURES - The following jobs have failed:

• pull / android / run-emulator (gh)
  The process '/usr/bin/sh' failed with exit code 255
• pull / test-samsung-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 10c0f561422b69829191728fe8180c429e60f5dbc7e44b5197e7eb8255a784f1 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.