Making Pythia an Image and CI

3 minute read

Anyone who watches my github repo for Pythia, one needs more to do with their time, and two will have seen a major overhaul to the project lately. Basically the old Munch based config and data management that was a hang over from well, mostly ML code, was gutted and instead there is now a key file and a database. One of the big motivators for this was wanting to move to image-based deployment for bots in general and having a good way to give persistent memory to Pythia, with this move Delphi becomes a code free host and just has to hold a few scripts that pull images. This seperation was easy enough to achieve with an SQLite database and that key sitting in a persistent volume that gets attached to the image. I’ve the option to move to a more fullfat database later, but honestly it feels like overkill right now, the use of SQLAlchemy as an interface though so this is a change that shouldn’t be too painful if I do it later.

But, honestly, all of this is preamble. What really took my energy yesterday was CI, or more specifically docker deployment. Wanting to have automatic docker builds on the core branch was the goal; latest build from any update, and versioned builds from version tags, GitHub Actions was going to be the mechanism. That’s as far as the nice plan went, and honestly I think my main takeaway from this has to be “Google better” because a lot of time went on trying to use a workflow that wasn’t going to work. Honestly this was rather frustrating as it was a lot of debugging to get to the point of realising that this workflow that was geared to work with Dockerhub and the Github Container Registry not, as I am currently using, The other thing that made me suffer for this was just trying to do string concatenation, and credit to Aaron for eventually saving me from something that is frankly just me being bad. So, spending an amount of time that I won’t admit to but you can dig up from commits on this was painful because this came while working on the original workflow that proved to be wrong for my purpose. Honestly truly annoyingly this was about a 10-15 minute problem, build times not included, once I’d found the right workflow the process of setting up the workflow was almost trivial and is something that’ll be rolling out to all of the bots I work with most likely.

In fact this pattern is likely to be rolling out to all of them in general, working with image builds is really easy and maintaining the Dockerfiles makes me a bit more concious of my dependancies. As a side note here some use of multistage builds has cut the image size from something that was capping at well over a gigabyte, down to less than 100 megabytes, which makes remote pushes way less painful. Reading this blog you may realise certain obstacles become more mental hurdles that practical ones to me. Moving bots to databases was definitely one of these, I’ve worked with bots that use databases before, but this still felt like it was going to be tough. In the end though this happened, much like the website build on the weekend, far faster than I expected and has helped get me engaged with my personal projects again, which is what the last week or so of post exam tech focus has meant to do, so a definite win! Next step will be extending this to the QuizBot project I’m contributing to.