Building a Builder

Distributing Nova presents some unique challenges because of the genre feature.  SMS was easy to distribute because it used a single genre and it was really easy to distribute because I could just zip it up, throw it on the server and people would download it.  The same isn’t possible with Nova for a couple reasons.  First, if we provided a single zip archive, it’d be massive because we’d need to include all the genre rank images along with the system.  For people with hefty Internet connections that can download and upload those kinds of file sizes, that’s fine, but for the average user it’s just not feasible.  Second, if we broke the genre files out to a separate download, then the user needs to download a second package, unzip it and put the files in the right location.  Sure, maybe that’s not too difficult, but the goal here isn’t “not too difficult”, we’re aiming for easy.  The only conclusion to come to was that Nova needed a download builder.

Fortunately, I have a little experience with the kind of code involved.  At a previous job, I was responsible for developing HTML emails for a client.  Each email was identical save for new images and new content.  On average, it’d take me between 60 and 90 minutes to build an email.  That’s all well and good, but I had other responsibilities as well, so I sat down and started building a simple system that would move files from a source directory into a directory dedicated to email output.  The whole thing started with a simple form where I’d copy and paste the content into text boxes and hit submit to trigger the magic.  Even with all the Photoshop work that needed to be done, I was getting these emails done in about 15 minutes.  I was saving the company between 45 and 75 minutes for every email.

So how did it work?  Pretty simple actually.  On the front page of the system, you’d just paste the content into the appropriate boxes and hit submit.  Once the form was submitted, it’d take those POST variables and put them in the proper locations in a template.  It’d then write the template to a file and move it to the proper location in an output directory.  Once that step was finished, the script would go out and move files from a source directory into the output directory.

When it comes to a download builder for Nova, that seemed the most logical place to start.  Use a source directory with the latest version of Nova and copy files around the server until we have what we need then zip it up and offer the download to the user.  So that’s where I started, but immediately a challenge presented itself.  What if multiple people were building their packages at the same time?  Using a single folder wouldn’t work at that point because one might overwrite another.  That’s no good.  The answer was that the folders needed to be unique, so the next step was to create these directories on the fly with unique names.  CodeIgniter has a grand little string helper that’ll generate random strings for you, so I started using that to programmaticly build directories with random 16 character alpha-numeric names.  Beautiful.  But once that was done, another challenge popped up: file permissions.

Now, on a distributed system like Nova, creating a feature that relies so heavily on PHP modules being turned on would be impossible.  It’s just too hard to work on that kind of thing.  But with a close system like the Anodyne site, it’s a lot easier.  But still, when you’re fighting with file permissions and the server creating files and whatnot, it’s significantly more difficult.  Try as I did, there was just no good way to get around the file permissions stuff.  Add to that a memory usage of about 3MB per request and you can see how a lot of users hitting it simultaneously (say the Nova launch), would cause problems.  Unfortunately, I didn’t see any other options, so I had to keep going.

The next step was zipping the files up.  Fortunately, that was the easy part because CodeIgniter has a zip archive library that helps with zipping files up and offering them to the user for download.  As I read through the CI User Guide about the library, I saw that the library wasn’t just for zipping files up, it actually provided a whole series of methods for creating the zip archive from scratch and adding files and directories to the archive on the fly.  Wait what?  I’d done all this work and it might be possible that CodeIgniter has a built in solution?  Golden.  So I loaded the library up and started working on doing the same thing I’d already done, but through CI’s zip library.

I was just about finished when another problem crept up.  This time, the library maintained the file structure of the source directory when adding directories.  That was a problem because adding the CI core and Nova core would require adding lots of directories and sub directories.  Instead of putting it in all nice and neat, the library would put it in the archive like /some/deep/path/to/source/dir/application.  Pretty useless when a user has to pour through those kinds of folders just to get what they’re looking for.  Lucky for me, the amazing community around CodeIgniter had already come up with a solution.  After downloading the extended zip class, I was on my way and in a matter of minutes, was offering up downloads to myself with the options I wanted.

So how does it work (generally speaking)?  Pretty simple actually.  You select the genres you want and click download.  The builder will take all of your genre selections and move the images and ranks folders where they belong, which is really handy in the event you select more than one genre.  If you only select one genre, the builder will also move a Nova config file into the right place that already has your selected genre set up meaning you only have to update your database settings and then install.

The builder should be a great tool for Nova when it launches and make life significantly easier!

3 Responses to “Building a Builder”

  1. Kirrus says:

    I take it you’re not a sysadmin then ;)

    Any sysadmin reading will be grimacing in severe pain. Make sure you cache the builds, and then just serve out the cache, or just statically build each option. Still use the download builder, but make sure you’re not generating each zip file on the fly, or, depending on the code, and the server, 15-30 simultaneous users and your servers groaning. 30-60 and it’s dead.

  2. There’s no way to cache it because it’s directories that are being pulled, not zip archives. Serving up static zip archives defeats the entire purpose of having a builder. The idea of a download builder is putting everything a user needs in the archive for them and not making them do all the work themselves. I understand what you’re saying, but I’m not sure what other solutions there are other than creating 15 different zip archives for every release that’s ever done.

  3. Kirrus says:

    15 different zip archives would be far better than generating them on the fly.

    The system can only generate so many of these at once, and as soon as you’ve exceeded that number, it will start to overload, and once you’re there, it won’t take too long for apache to take the system down.

    If you must do this, limit the number of simultaneous builds to 15 or so, in your application code. Some may have to wait to get it, but it’s better than having to reboot your server a few times!

    Another course, is adding “monit” to the server, and setting it to restart apache if the server loads go above ~10. It’ll kill all current sessions, but again, better than having to reboot your server.

    Can always try doing a test with the builder system, should be able to script a build so you can hit the server with a specified number of build requests simultaneously.

Back to Top