Guidance/plans to add encoder/decoder model support for e.g T5 model?

[mustavikhan05]

I'm trying to adapt the bitdistiller code for encoder-decoder models.

Are there any plans to add support for this? Can some guidance be provided what parts need adaptation?

We're running a project to test the findings found in Table 5 where Llama 7B performed better as the teacher than 13B. We're testing the hypothesis you put forward across OPT models and now expanding our experiment to encoder-decoder models. Further, we're also running an experiment to sequentially introduce larger teachers. I.E self-distillation followed by a bigger model as teacher on the self-distilled model.

[DD-DuDa]

It seems that all current cutting-edge models are decoder-only LLMs, so we might not have plans to support them. However, it may not be difficult to do so. You might just need to modify the loss calculation function. You can refer to the implementation in mytrainer.py.

The hypothesis is an interesting finding and worth further exploration. If you have any questions, feel free to contact me.

[mustavikhan05]

Thanks for the response and I've started work on adapting to encoder-decoder models to test the hypothesis across other architectures. We've already run tests on the OPT models to explore the hypothesis across varying quantization levels. Also, another thing to note the choice of dataset may also be a potential clue hence we've expanded the training datasets to other datasets on top of the standard pipeline. One thing we encountered was the numerical stability of bf16 over fp16 on this particular pipeline.

It would be great if we could get on a 20 min call to discuss this with my professor. I'm emailing you on the email provided on your landing page.

Thanks, again!

[BrownianNotion]

@mustavikhan05 I'd also like to look into this hypothesis for a project, are you able to share your findings somewhere on github if not already? What results did you get - was the hypothesis of same architecture teacher -> quantised student better than larger teacher -> quantised student? Would love to discuss if you have some time :)