Running aitextgen model training in a Docker container
source link: https://www.kevinhooke.com/2021/10/10/running-aitextgen-model-training-in-a-docker-container/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Running aitextgen model training in a Docker container
I’m setting up an approach to run text generation model training jobs on demand with aitextgen, and the first approach I’m looking at is to run the training in a Docker container. Later I may move this to an AWS service like ECS, but this is my first step.
I’ve built a Docker image with the following dockerfile:
FROM amazonlinux RUN yum update -y RUN yum install -y python3 RUN pip3 install aitextgen ADD source-file-for-fine-tuning.txt . ADD generate.py . ADD train.py .
.. and then built my image with:
docker build -t aitextgen .
I then run a container passing in the cmd I want to run, in this case ‘python3 train.py’:
docker run --volume /data/trained_model:/trained_model:rw -d aitextgen sh -c "cd / && python3 train.py"
I’m also attaching a bind point where the model output is being written to during the run, and -d to run the container in the background.
CategoriesKev's BlogTagsAI, aitextgen, aws, cloud, docker, ML, text generation
Leave a Reply Cancel reply
Your email address will not be published. Required fields are marked *
Comment
Name *
Email *
Website
Save my name, email, and website in this browser for the next time I comment.
Notify me of follow-up comments by email.
Notify me of new posts by email.
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Post navigation
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK