I’m currently diverting development attention away from forc (our WIP font creation tool) and onto what has become one of the roadblocks to making outputting standard Mutant Standard releases easy, which is our lovely workhorse exporting program called orxporter.
Like a lot of things in Mutant Standard, reaching for new technical heights or features (crushed PNGs, fonts, etc.) easily tips our infrastructural capacity to deal with it gracefully, necessitating another round of changes to our infrastructure.
The war on export time
For a project with as many images as Mutant Standard, it can be pretty easy for software to spend hours and hours converting all the images to the right formats without the right approaches and tools. Especially when stressed by new demands.
August 2018 – January 2019
- Mutant Standard 0.3.0 & 0.3.1
- Orxporter 0.1.0
Mutant Standard’s final emoji count tripled as a result of the colour modifier update in 0.3.0, this jumped the export time from an okayish 45 minutes to more than 3 hours.
The solution to this was to use a much faster renderer than what was being used before, add multithreading, and batch exif metadata embedding at the end of exporting instead of calling the exif embedder once for every file.
This took an export time of about 3.5 hours down to about 15 minutes. But unfortunately these time saves quickly became irrelevant again…
April 2019 – now
- Mutant Standard 0.4.0
- Orxporter 0.2.0
For Mutant Standard 0.4.0, I wanted to use optimised SVGs and crushed PNGs in this release to reduce bandwidth of those hosting Mutant Standard, and to reduce the size of font builds. I also wanted to vary what PNG and SVG exports got licenses or not (because images embedded in fonts don’t need licenses, the font itself has it’s own metadata system for licenses).
I also started playing with next-generation image formats, like WebP and FLIF. For these, the emoji have to be rendered as PNGs first, and then converted.
For various reasons (as I will explain in a sec), this has once again resulted in an export time that takes several hours. Unlike the last time, this also required an intricate, long shell script to handle tasks that orxporter wasn’t capable of handling.
Orxporter also doesn’t support optimised SVGs and crushed PNGs at the moment, requiring export tasks taking place outside of orxporter, and to a small degree undermining the point of orxporter.
Isolated, linear exporting tasks
Unlike in 2018 where the solution was how fast the rendering tasks were going, 2019’s problem is the way exporting routines are structured.
Currently in orxporter, when a format is chosen for export, it will do all the tasks required to get that output for one format in a linear order, and every other format pursues their own line of tasks. Everything done in the interim to get to the end product is a temporary file and gets discarded.
This is fine when we just had two simple formats for export – basic PNG and SVG.
Optimised SVG, crushed PNG, and next-generation formats (like WebP) all require multiple stages, many of which are identical to each other. This results in a lot of wasteful exporting with the more complex exporting demands that I have now.
This also happens because orxporter’s export parameters have little room for complexity. If certain parameters are different (like whether something has license metadata or not), orxporter has to be called again in a new command line, creating even more wasted export routines.
Dependency-based exporting
To fix this issue, we need a system whereby orxporter takes stock of every single exporting target given to it, so it can then create a tree of tasks, and procedurally go through each of them, so every render or conversion task is only done once.
In addition to this, we need something to prevent repeat command line uses. So we’re going to add a Parameters file option, for storing exporting formats and parameters in more detail. This means that every format that is desired can be requested in a single command (while also keeping the original command line arguments for people who want to use orxporter in a more simple and immediate way).
This new version will also support crushed PNGs and optimised SVGs. And it’s far more resilient and extensible to add them in this situation, as well as new formats coming in the future!
This in theory should be the last big, necessary update for orxporter, as after this there is not much room for optimisation aside from getting better hardware (in particular an SSD). After this point, and after giving the code a good check, orxporter will probably be ready to be called 1.0! :3
Before that
orxporter (which was originally made by kiilas <3) was not designed to be developed on by multiple people, so I’ve got to tidy things up and make various small changes to make this ready for those big changes. I’ve made a bunch of changes already and released Orxporter 0.2.1 (which is currently in my own fork because it was not expected that I would be working on this, it will be in Mutant Standard’s official repo soon) and it’s almost there!