This project fine-tunes the Llama 3.2 3B Instruct model using the Unsloth framework for efficient LoRA-based adaptation. It trains on a small Taskd dataset to teach the model how to answer questions about the AI automation company Taskd, and deploys the fine-tuned model using vLLM to create an OpenAI-compatible API server for inference.

Before you begin, ensure you have the following installed:

• Python 3.10+
• Git
• pip (Python package manager)
• vllm (can be installed via pip)

Optional but recommended:

• GPU with CUDA for faster training
• A Weights & Biases (W&B) account for run tracking
• uv is a fast Python package and environment manager.

  pip install uv

git clone https://github.com/Arpnik/taskd-technical-challenge.git

uv venv .venv
source .venv/bin/activate   # for macOS / Linux

Update the Weights and biases account API key here in the script: wandb.login(key="xyz")

python fine_tune_taskd-v1.py

To override defaults via command-line arguments:

python fine_tune_taskd-v1.py \
  --epochs 200 \
  --lr 2e-4 \
  --lora_rank 128 \
  --lr_type cosine \
  --weight_decay 0.02 \
  --run_name "taskd-llama-v5" \
  --max_new_tokens 256

The fine-tuned model will be saved in ./taskd_lora_model/ by default and the training loss curve will look something like this:

After fine-tuning your model, you can test it in two ways:

This script directly loads your fine-tuned LoRA model (taskd_lora_model/) using Unsloth, runs inference, and extracts clean multiline assistant responses.

python test_fine_tuned_model.py [--max_new_tokens 512] [--temperature 0.01]

• Loads your LoRA fine-tuned model in 4-bit precision for fast inference.
• Applies the same Llama-3.1 chat template used during training.
• Cleans raw model outputs using a regex-based extractor for multiline responses.
• Tests a list of example prompts automatically to verify model performance.

If you’re running your fine-tuned model using vLLM (for efficient inference serving), you can use this client script to query the model through its OpenAI-compatible API.

First, launch your vLLM server:

vllm serve ./taskd-technical-challenge/taskd_merged_model --port 8000

Then, in another terminal, run:

python run_inference_on_vllm.py

Once started, the client enters interactive Q&A mode.

• Sends your prompts to the vLLM server via its /v1/chat/completions endpoint.
• Returns the assistant’s message using OpenAI-compatible JSON schema.
• Handles connection errors gracefully and supports adjustable generation parameters.
• The chatbot does not remember previous prompts between messages when using this script.

In a new terminal, verify the server is running and the model is loaded:

curl http://localhost:8000/v1/models

Expected response format:

{
  "object": "list",
  "data": [
    {
      "id": "unsloth/Llama-3.2-3B-Instruct",
      "object": "model",
      "created": <timestamp>,
      "owned_by": "vllm",
      "root": "unsloth/Llama-3.2-3B-Instruct",
      "parent": null,
      "permission": [...]
    }
  ]
}

Install jq for JSON formatting (optional but recommended):

apt install jq

Send a test request to the chat completion endpoint:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "unsloth/Llama-3.2-3B-Instruct",
    "messages": [
      {"role": "user", "content": "Say hi"}
    ]
  }' | jq -r '.choices[0].message.content'

Expected response: A friendly greeting from the model.

Out of Memory Error:

• Reduce --gpu-memory-utilization to 0.7 or 0.8
• Reduce --max-model-len to 2048

Connection Refused:

• Ensure you're using 0.0.0.0 as the host
• Check that port 8000 is not already in use
• Wait for the model to fully load before making requests

Model Download Fails:

• Verify internet connectivity
• Confirm you're using unsloth/Llama-3.2-3B-Instruct (no HuggingFace login needed)