fix: re-add litellm, fix ingestion without langflow for all providers, fix search functionality for all providers

[lucaseduoli]

This pull request introduces a centralized and robust approach to embedding model configuration and provider detection, improving maintainability and clarity across the codebase. The main changes include extracting embedding model constants into a dedicated module, refactoring model name formatting into a utility function, and updating dependencies and usages throughout the project to use these new utilities.

Embedding Model Configuration and Utilities:

• Introduced a new module embedding_constants.py to define embedding model names and their vector dimensions for both OpenAI and WatsonX providers, removing duplication from settings.py. [1] [2]
• Added a utility function get_formatted_model_name in model_utils.py to standardize model naming conventions for LiteLLM/MCP compatibility, replacing scattered logic with a single source of truth. [1] [2] [3]

Refactoring and Codebase Cleanup:

• Refactored imports and usages throughout the codebase (settings.py, search_service.py, embeddings.py) to use the new embedding_constants.py and get_formatted_model_name utility, eliminating redundant or duplicated code. [1] [2] F13f5c3cL13R1)
• Improved thread safety and clarity by using the imported threading module directly in client initialization.

Dependency and Client Initialization Improvements:

• Added the agentd dependency and removed unused dependencies (openai, pyyaml, tiktoken) from pyproject.toml to streamline the environment. [1] [2]
• Enhanced OpenAI-compatible client initialization to use a dummy API key if none is provided, and ensured all client instantiations are routed through the MCP patch for provider-agnostic access. [1] [2] [3]

Service Enhancements:

• Decorated the search_tool method with the @tool decorator to enable agentd tool integration.

These changes collectively improve the maintainability, clarity, and extensibility of the embedding model infrastructure and provider integration.

• Embedding Model Configuration:

  • Added embedding_constants.py to centralize embedding models and vector dimensions, removing hardcoded values from settings.py. [1] [2]
  • Updated imports in settings.py, search_service.py, and embeddings.py to use the new constants. [1] [2] F13f5c3cL13R1)

• Model Name Formatting Utility:

  • Added get_formatted_model_name in model_utils.py and refactored all model formatting logic to use this utility for consistent provider prefixing. [1] [2] [3]

• Dependency and Client Initialization:

  • Added agentd as a dependency and removed unused dependencies from pyproject.toml. [1] [2]
  • Improved OpenAI client initialization to support dummy API keys and ensure all clients are patched for multi-provider compatibility. [1] [2] [3]

• Service and Decorators:

  • Applied the @tool decorator to search_tool in SearchService for agentd integration.

• Thread Safety:

  • Switched to using the imported threading module for lock initialization in settings.py.

  Closes [Bug]: Issue in knowledge search with Ollama embedding model #1173

  Duplicate of fix: re-add litellm, fix ingestion without langflow for all providers, fix search functionality for all providers #1347