It is possible to fine-tune CodeGen using Huggingface Transformers! Then you'd be able to fine-tune it on your own code and use the resulting model. However, training is more expensive -- you'd need an A6000 or better to train the 6B model. Something like the following should work:
Colab will give you a T4, K80, V100 or P100 (alternatively their own TPUs) for free - $50 for 24h uninterrupted background jobs, Gradient will give you free A6000s and sometimes even free A100s for a $40 subscription for 6 hours (repeatable ad infinitum), Lambdalabs gives you a RTX 6000 for 0.50/hour and A6000 for 0.80/hour and Vultr GPU will give you 1/7th of an A100 for 0.37/hour
Thank you for sharing the command for finetuning! Is it possible to share your ds_config.json? I tried to finetune the 2B model on A100 (40GB) using your command, but got a CUDA out of memory error. The ds_config I used was the one from huggingface (https://github.com/huggingface/transformers/blob/main/tests/...).
Have a look at the datasets library [1], but as a shortcut, you can just create a file named "my_code.json" in jsonlines format with one line per source file that looks like:
It might be doable to set this up on an AWS machine with a beefy GPU or two. I haven't tried it yet though.
Once you have a model trained in Huggingface Transformers you'd be able to convert it using this script:
https://github.com/moyix/fauxpilot/blob/main/converter/huggi...