The Build Pipeline

Engineer goes through a number of steps when building a site. Understanding this order and the steps that are taken might be helpful when building your site. At the very least, it might be interesting.

Basic Flow

Engineer operates almost entirely on something called the ‘output cache.’ This is a location in the CACHE_DIR. Engineer essentially ‘stages’ all content there during processing. Once that’s all done, then it synchronizes the actual OUTPUT_DIR with the cache.

With that in mind, these are the basic steps that Engineer goes through. You might also find it interesting to look at the code for the engineer.engine.build() function - it is the primary entry point for the build process.

  1. Copy base Engineer static content

    First, any static content from the core Engineer libraries is copied to the output cache. This includes things like the built-in jQuery, Modernizr, and Foundation libraries.

  2. Copy theme static content

    Any static content needed by the theme is copied to the output cache.

  3. Generate template pages

    Template Pages are generated and copied to the output cache.

  4. Load posts

    Posts are loaded from the post cache and POST_DIR and stored in memory in a PostCollection.

  5. Generate posts

    Based on settings like PUBLISH_DRAFTS and PUBLISH_PENDING, the list of posts is trimmed to those that should be output, then they are rendered and placed in the output cache.

  6. Generate rollup pages

    The front page and other ‘rollup’ pages are generated and placed in the output cache.

  7. Generate archive page

    The archive page is generated and placed in the output cache.

  8. Generate tag pages

    Tag pages are generated and placed in the output cache.

  9. Generate feed and sitemap

    The RSS feed and sitemap are generated and placed in the output cache.

  10. Copy raw content to output cache

    Any site-specific Raw Content is copied to the output cache.

  11. Compress/minify CSS/JS

    If the COMPRESSOR_ENABLED setting is on, then any CSS or JS files used in the site will be compressed.

  12. Remove source LESS files from output cache

    If the PREPROCESS_LESS setting is on, then the source LESS files will be removed from the output cache. This is safe to do since the LESS preprocessing happens during rendering, so at this point the corresponding CSS file has already been generated. The LESS source file is no longer needed.

  13. Synchronize output directory with output cache

    Finally, the contents of the OUTPUT_DIR is synchronized with the output cache. This approach ensures that the actual site output is disturbed as little as possible. All the major copying/generating/rendering has already happened separately on the output cache, so the actual site output directory has only changes/additions/deletions propagated to it.

Raw Content

Raw content is simply content that you want to include in your site as-is with no processing. Static content such as JavaScript and CSS could also be considered raw content but those sorts of files don’t need to live at a specific place in your site’s URL. The raw content feature is specifically for content where placement matters, such as for robots.txt files and favicons. These files must be in the root of your site, so just treating them like regular static content won’t work.

Raw content should be placed in your site’s CONTENT_DIR, and since it’s the last thing copied in the build pipeline, it will overwrite any content that was generated by the other phases of the build pipeline. Keep this in mind.

The structure of CONTENT_DIR should match your site’s. In other words, if you want something to wind up in the root of your site, you would put it at the root of your CONTENT_DIR. Similarly, if you want something to wind up in /foo/ in your site, you’d put it in a foo folder inside your CONTENT_DIR.

CSS/JS Compression

TODO

LESS Preprocessing

TODO