Building a static site generator

This is the first post of my new personal space on the web. It doesn't look like much right now, it doesn't even have any styling whatsoever. But I intend to improve it while I work on it. You see, the usual trend of me blogging is: 1. get really excited about something I want to write about, 2. set up the blog using a SSG, 3. write the post I got excited about, 4. be really sluggish in writing follow-up posts, and 5. finally lose intrest and let the blog lie dormant. One of the causes to point 4 is not having very much to write about. People say blogging is something you should just start with and get in the habit of doing to keep it going. But I rarely find something interesting enough to write about or think others will care about. Another problem for me is that, because my activity is so low, my repo nearly always gets behind on the SSG packages it uses, and since there's so much time between my activity, I often forget about how the SSG works and updating/migrating to the new version brings some difficulty. So I figured, why not try to solve both these problems by writing my own static site generator? It will give me something to write about. The blog looks monstrously ugly at the moment, so there's incentive to improve it and while doing that, I can take some inspiration for topics. All the while, I guess it can improve my writing skills as well.

How it's made

There's a great tragedy in my developer life, a forbidden love. It's name is F#. I guess like many .net developers, I love F# but rarely get to write it since it's hard to push it at work when I'm the only one who is (kind of) proficient at it. So personal projects, while few and far between, are where it's at when choosing F#, and so it is with my SSG.

My previous experience with static site generators has been exclusively with metalsmith.js. I really like how it works and is a perfect fit to take a similar route when implementing in F#. Metalsmith quotes on its website: "Since everything is a plugin, the core library is just an abstraction for manipulating a directory of files.". In effect, it is a pipeline through which your content files are fed, with a plugin being a step within the pipeline. I guess I needn't say more about how naturally this fits for F#?

I've set up my project to consist of 2 parts: the first is a bunch of libraries making up the SSG and the second is a FSI script that references these libraries and wires it all up to build this website.

The script to generate the site looks like the following:

WssgConfig.init
    // base config:
    |> source "./content"
    |> output "./public"

    // add processors:
    |> add (clean true)
    |> add frontmatter
    |> add markdown
    |> add (templates {
            Directory = "./layouts"
            DefaultTemplate = "index.hbs"
        })

    |> run Loggers.console

It starts out with an empty config object. I can then thread this object through a series of functions that each work on a part of it in order to set up the pipeline. In the code snippet above, I call the functions source and output to respectively configure the directory where my content files are stored and the directory where the generated output should go. Then there are several add invokations, each adding a step to the pipeline. Finally, I call run to execute the pipeline. As it is with metalsmith, the trick is to know which steps to add to the pipeline. Here, I add frontmatter, markdown and templates. This should tell you that I am writing my blog in markdown format and apply a template to it to produce a full web page for them (and the .hbs extension probably tells you that it's handlebars). The "frontmatter" bit might not be a familiar term (it wasn't for me). It is a way of defining metadata about a markdown file and is appended as a header in the file it is describing. Its name is literally taken from how books add a preliminary section before the actual content (cfr. Wikipedia).

These are probably the bare minimum to have a working static site generator. If you're reading this now, it means that I've succesfully built my site using it and I don't see how I could have done it with anything less than that. But let's look at the other part, the code where these functions live.

Let's see what we can tell from the types that are defined in my core library:

type Item = Item of FileInfo * Map<string, string>

type WssgConfig = WssgConfig of {| Source : string; Output : string; Processors : Processor list |}
and Processor = Async<Wssg> -> ILogger -> Async<Wssg>
and Wssg = Wssg of WssgConfig * Item array with
    static member fromAsync config = async { return Wssg (config, Array.empty) }

The Item type holds information about a file, tupled with a map with both string key and value. This map is the metadata describing the file and will, for the most part, be populated with the frontmatter I specify in the file itself. I say for the most part because steps in the pipeline may add their own entries if they have a need for it.
Then I have the WssgConfig type which I'm configuring in the FSI script above. It holds the source directory, output directory and a list of processors to execute. This Processor type is a function that takes a type called Wssg and then returns it. I think it is obvious that both passing this type in and returning it, indicates that these functions are the steps in the pipeline and that you can chain them together. Wssg is my final type and it is used to store information when the pipeline is running. It holds the config object, tupled with the sequence of items.

Next I'd like to show you the run function that I call on the last line of my FSI script:

let run logger config =
    let (WssgConfig c) = config in
        c.Processors
        |> List.fold (fun wssg proc -> proc wssg logger) (Wssg.fromAsync config)
        |> Async.RunSynchronously

Okay I probably didn't even need to show you this as running a pipeline obviously couldn't have been anything other than a fold. But, whether or not surprising, these types and this fold, some 10 lines of code, are all that are needed for the core of the SSG.

Now all it needs are some Processor functions. 10 lines of code couldn't generate much of a website without them. I won't list them all, but let's look at how we get the Item array populated. I have coded this to just be another Processor function, with the only difference being that it will always be the first processor function to be called; it is not configurable in the FSI script.

let listFiles : Processor =
    fun asyncWssg logger ->
        async {
            let! (Wssg (WssgConfig config, _)) = asyncWssg

            let dir = new DirectoryInfo(Path.Combine(Directory.GetCurrentDirectory(), config.Source))
            return Wssg (WssgConfig config, dir.EnumerateFiles("*", SearchOption.AllDirectories) |> Seq.map Tools.toItem |> Seq.toArray)
        }

It is nothing special. It just takes the configured source directory and enumerates all the files in it. But I do think it shows just how nice and concise these functions are. They all are very small, mostly doing just one thing. Making them part of a pipeline to perform a bigger task that, on the whole, is more complex, probably captures the true attraction of functional programming for me.

Sorry if that sounded a bit sentimental. Let's have a look at another processor and then I'll call it a post. Here's the code for the markdown processor:

let transformAsync (asyncInput : Async<Item * string>) =
    async {
        let! (item, contents) = asyncInput             

        return (item,
            markdig()
            |> frontmatter true
            |> useSoftlineBreakAsHardlineBreak()
            |> toHtml contents
        )
    }

let prepareOutputAsync (WssgConfig config) (asyncInput : Async<Item * string>) =
    async {
        let output = new DirectoryInfo(config.Output)
        if (output.Exists |> not) then output.Create()
    
        let! (Item (f, map), html) = asyncInput
        let fo = new FileInfo(Path.Combine(config.Output, sprintf "%s.html" (f.Name.TrimEnd(f.Extension.ToCharArray()))))

        return (Item (fo, map), html)
    }
    
    
let markdown : Processor =
    fun asyncWssg logger ->
        async {
            let! (Wssg (config, items)) = asyncWssg

            let! markdownFiles =
                items
                |> Array.filter (Tools.filter <| ByExtension ".md")
                |> Array.map (Tools.readAsync >> transformAsync >> (prepareOutputAsync config) >> Tools.writeAsync)
                |> Async.Parallel

            return (Wssg (config, items |> Array.append markdownFiles))
        }

That's a bit more code but still manageable, I think (even though I haven't listed all the helper functions). The transformAsync function takes Item * string tuple (wrapped in an async). The Item part of this tuple, is not needed here but I'm just threading it through for easier composition in the last function. The string part of the tuple is what is interesting here as this contains the contents of the file, which should be markdown content. So I'm using an external library called Markdig to perform the actual conversion. I've written some helper functions (not listed here) to provide a more functional friendly API. I configure it to skip the frontmatter part of the file, render linebreak as <br /> tags, and then call toHtml. Below that I have the prepareOutputAsync function. This one will prepare a FileInfo object that points to a file to create in the output directory. It needs information from the WssgConfig object for this, and uses the name of the source file for the eventual file name, but with an .html extension instead. Tee returned result is similar to the input tuple, but the file is the output file, rather than the input file. The final function, markdown, is the actual Processor function that wires the former two functions together. You can see this where the Array.map is called. There's a function composition going on there that first reads a file, then transforms the markdown, prepares the output file and finally writes the contents to the output file. The read and write functions, Tools.readAsync and Tools.writeAsync, are not listed here but they essentially use a System.IO.StreamReader and System.IO.StreamWriter to do their work.

The future

I think that's my post. I won't bore you with all the details of my code. I think I just needed to lay out how my little project will work. I can keep other mentionable details for separate posts. However, I can't write about them yet as my SSG currently can only generate a single file, the one you're reading now. I need to come up with other processor functions to do more interesting things. I won't set myself any timeframe in which to do this, but I am somewhat excited to work on it. Whenever I have progress, you'll see my site change and I think I should write about what I did to get there.

Further down the line, I'd like to use my site to share more about all the projects I'm working on. I plan for it to not just be about programming. But I'll share in due time, and when my SSG is ready for it.