- Posted on
- 2026-08-07T07:48:00Z
- Tags
- Content warnings
- None
Table of Contents
A big follow-up to Zola is dead to me. Long live Kusachi!.
I was working on this website again to add font support and some toki pona to it, when I felt struck with dissatisfaction with the state it was in! The transformers and components and their disparate nature felt jarring and overcomplicated!
§ Moving away from pandoc; the prototype "ma kasi lili "
The project started with a prototype and looking into getting rid of using pandoc for the markdown preprocessor and moving to a (more) native Clojure CommonMark parser. I settled upon nextjournal/markdown but I'm not sure whether this is a sane decision to stick with long term. I like that it has a library of functions to which it delegates rendering of individual types of markdown feature with and this was inspiring!
It rendered directly into hiccup-style data structures and this was convenient, however being able to parse the embedded HTML within markdown documents still required rendering the hiccup into HTML first (before turning it back into data and unescaping what hickory had just escaped for later work).
Here's a look at the renderers I'm using at the current point of writing this:
(def renderers
(assoc md/default-hiccup-renderers
:toc render-toc
:code (fn [_ {:keys [language] :as m}]
[:pre.kxcode
[(if language
(keyword (str "code.language-" language))
:code)
[:hiccup/raw-html (sintech language (-> m :content first :text ))]]])
:image (fn [{:as _ctx ::keys [parent]} {:as node :keys [attrs]}]
(let [url (:src attrs)
thumb (ft/add-thumb-to-filename url)
img-markup [:a {:href url} [:img (merge attrs {:src thumb :alt (md/node->text node)})]]]
(if (= :doc (:type parent))
[:p img-markup]
img-markup)))
:html-inline (comp rawy md/node->text)
:html-block (comp rawy md/node->text)))
It pretty cleanly facilitates making the image statements into thumbnails and providing links! It also helps with how I now do syntax highlighting but we will get into that later on!
§ No pandoc, no template to include frontmatter into, no selectors‽
Without pandoc and the template I was giving it to produce HTML to read frontmatter out of and then remove (ew, gross), I realised I would have to read in the frontmatter and split the document on it, so now I do.
I feel, however inscrutable the below may seem, it ends up being pretty elegant.
; note: s/ is clojure.string ^^
(defn yaml-grabby [sp]
(let [[lx ax] (split-with #(not (re-matches #"---\s*" %)) (drop 1 sp))
ls (s/join "\n" lx)
as (s/join "\n" (drop 1 ax))
dat (yaml/parse-string ls)]
[dat, as]))
(defn edn-grabby [sp]
(let [[lx ax] (split-with #(not (re-matches #"\s*" %)) sp)
ls (s/join "\n" lx)
as (s/join "\n" ax)
dat (edn/read-string ls)
date (:date dat)
; this is done for rfc3339 / atom :3
pdate (inst/read-instant-date date)
dat2 (assoc dat :date pdate)]
[dat2, as]))
(defn metadater [data]
(let [sp (s/split-lines data)]
(cond
(re-matches #"---\s*" (first sp)) ; yaml
(yaml-grabby sp)
(re-matches #"\{.*" (first sp)) ; edn
(edn-grabby sp)
§ Rethinking the tree transformers / zippers that this site uses
I disliked the ad-hoc nature of the prior system for transforming the pandoc output and how much was being duplicated amongst site features alongside the added complexity.
For each feature, a hickory-update call would zip over the entire tree and rewrite it (while requiring that the change made the selector function invalid for the location it was called upon!). Having to traverse over each entire document as many times as there were functions to transform it was likely a bad move architecturally.
Additionally, I was using hickory's own data structure for HTML selection and updating instead of hiccup-style, which I use for the rest of the website's components/"templates" (so, now we have two formats instead of one!).
A comparison between the two:
; hickory
{:type :element
:tag :p
:attrs {:class "hello"
:id "world"}
:content [
{:type :element
:tag :a
:attrs {:href "bar"}
:content [
"meow!" "wow!"
]}
]}
; hiccup-style
[:p.hello#world>a.foo {:href "bar"} "meow!" "wow!"]
One of these is arguably far better for ergonomics for the end-user and one of these is easier for hickory to do selectors over (admittedly, having things that expand out like the hiccup-style would be a lot harder to handle, alongside the fact that one can put ids and classes both into the keyword and into the attrs section). Normalizing this is, obviously a bother.
However, as an intermediate form for zipping over? Hickory provides a zip function for working with hiccup-style structures in this way and provides a means of taking HTML and turning it into hiccup-style data. In such a situation, the normalization or complexity of hiccup as produced by a user is irrelevant since it would be produced unambiguously by hickory and it would be produced by the markdown preprocessor.
Looking over my actual wants for this and having discussions with several friends at the time, I figured out what I actually wanted was a way to register components for programmatic replacement of parts of the produced tree.
§ The basis for components
; a place to store component functions
(def component-library (atom {}))
; a way to register them, a pattern is just what element it applies to
; theoretically a pattern could be a function that you call against the loc
; that returns a boolean to decide whether to run the component function
; upon it, even. this would not be that much worse other than an additional
; function call. it was not considered necessary here.
(defn register-component [pattern component]
(swap! component-library assoc pattern component))
; a way to register component functions that edit instead of removing
; (a removal type never actually manifested!)
(defn register-edit [pattern component]
(println (str "Registered edit pattern: " pattern))
(register-component pattern (fn [loc & args] (zip/edit loc component args))))
; self-explanatory :3
(defn tree-walk [f z]
(if (zip/end? z)
(zip/root z) ; root presents the new (potentially) modified tree
(recur f (zip/next (f z)))))
; if the location is a branch, check if the location contains a registered component, if it does, call it and hand it the frontmatter too
(defn hiccup-transform [fm loc]
(let [node (zip/node loc)
tag (nth node 0 nil)]
(if (zip/branch? loc)
(let [component (get @component-library tag)]
(if (some? component)
(component loc :frontmatter fm)
loc))
loc)))
; zip the tree, walk it and transform it
(defn hiccup-transformer [fm hickory-tree]
(let [zip (hz/hiccup-zip hickory-tree)]
(tree-walk (partial hiccup-transform fm) zip)
))
§ What do components look like?
Clojure is a beautiful language and has an amazing system for destructuring arguments! This is used greatly to my advantage for component functions. From the code within the basis, you can see components are called with a positional argument loc (their entire node) and a keyword argument :frontmatter. This is how Clojure makes that beautiful:
; destructuring the loc argument
; this makes headings have a link to their own section :3
(defn headings [[tag {id :id :as attrs} & rest :as lit] & _]
; this check is because the table of contents also contains headings
; *those* headings do not have an id attr so this is used to disambiguate them
; because i do not want them to have section links (it would be silly!)
(if (some? id)
[tag attrs [:a {:href (str "#" id)} "?"] " " rest]
lit))
(doseq [tag [:h1 :h2 :h3 :h4 :h5 :h6]]
(register-edit tag headings))
; destructuring the :frontmatter argument
; this makes the header containing frontmatter data for every post
(defn frontmatter-interpolation [_ {{:keys [date draft tags cws]} :frontmatter}]
; reagent-style fragment (<3 huff)
[:<> [:dl {:id "frontmatter"}
[:dt "Posted on"]
[:dd (kh/datf date)]
[:dt "Tags:"]
[:dd {:id "tags"}
(if (some? tags)
[:ul {:class "inline-list" :id "taglist"} (for [el tags]
[:li [:a {:href (str "/tags/" el ".html")} (str el)]])]
"None"
)]
[:dt "Content warnings"]
[:dd {:id "cws"}
(if (some? cws)
[:ul {:class "inline-list" :id "tags"} (for [el cws]
[:li [:a {:href (str "/tags/" el ".html")} (str el)]]
)]
"None"
)]
]
; a when becomes a nil on falsity and nils do not become anything in the HTML
(when draft [:i "This page is a draft! Content may not yet be finalized."])
])
(register-edit :frontmatter frontmatter-interpolation)
§ Live reload; re-examining the static site generator as build-system adjacent tooling
I started to feel somewhat like the generation times on kusachi were a little long, at approximately 9s for a run (with image hashing) and so I wanted to look again into building a persistent dependency graph resolver with hashing/caching (this would also facilitate me having partial recompiles!).
§ How I Learned to Stop Hashing and Love mtime
However, I decided to look into profiling it (I believe it was at wife's behest) before committing properly to complex rearchitecturing in this way. Looking into the flamegraphs produced by clojure-goes-fast/clj-async-profiler, it was evident that other than the fixed cost of JVM + Clojure compilation a vast majority of the time was spent... computing sha256 digests of each file. Oh.
Wife pointed out to me that a far cheaper "hash" for kusachi could in fact, be the file's modification time and making this change reduced the time spent by kusachi (with images already generated) by 3 seconds (from 9s to 6s!).
From there, I looked into adding partial compilation... not from the perspective of calculating a dependency graph but instead a file-watcher. kusachi has already had a serve subcommand for a while now, using ring, jetty and compojure to serve the static files (so that it could be tested with links to / not going to actual filesystem /).
I was already using one nextjournal project and my TODO list contained a mention of nextjournal/beholder, so I went to check that out. Upon adding a new subcommand behold (later became watch), I noticed that it would do nothing for ...a really long while. It was our old friend, "hashing the contents of every file in every subdirectory given". Upon adopting filipesilva's fork it started instantly! No wonder.
After significant rearchitecting of many things, I was able to get changes in any file associated with a core to show up after a manual reload.
A core (the one that made this markdown file into HTML):
(def post-core {; self-explanatory
:name "post-core"
; what subfolder does this apply to
:path "posts"
; a lens determines what files apply to this core (within the specified path)
:lens {:filter [(fc/glob "*.md")]
:remove []}
; do we always recalculate (for uncacheable cores)
:recalculate (fn [] true)
; hooks are for incremental recompile handling
:hooks [
generate-tags
generate-rss
generate-postlist
generate-index
]
; a router takes the input path and decides where the output goes
:router [fr/output-router (fr/refiletyper "md" "html")]
; a compiler is what transforms an input into an output
:compiler (fn [cont path out-path]
(->> path
(#(let [
[metadata, post] (huffmd %)
mutant (hiccup-transformer metadata post)
templated (fco/blogpost :continuous cont
:metadata metadata
:content mutant
:title (:title metadata))
current-path (.getAbsolutePath (io/file "./"))
output-path (.getAbsolutePath (io/file "./output"))
relative-out (fr/relative-path current-path out-path)
relative-output (fr/relative-path output-path out-path)
context (assoc metadata :rel (s/replace relative-output "index.html" ""))]
(swap! post-map assoc relative-out context)
(assign-post-to-tags relative-out context)
templated))
(fc/simple-writer out-path)))})
During my last post about kusachi, I spoke about a project in Gleam that I had gotten livereload working with. I took that code and adjusted it to be simpler, alongside implementing the server side with ring's async websocket handling.
Clojure (server side):
(def livereload-channel (a/chan))
(def livereload-mult (a/mult livereload-channel))
(defn send-reload []
(println "Live reload requested")
(a/put! livereload-channel "livereload"))
(defn make-livereload-handler []
(let [in (a/chan)
out (a/chan)
err (a/chan)]
(a/go (try
(a/tap livereload-mult in)
(loop []
(when-let [mesg (a/<! in)]
(a/>! out mesg)
(recur)))
(finally
(a/close! out)))) ;; closes the websocket
{:ring.websocket/listener (wsa/websocket-listener in out err)}))
(cc/defroutes app-routes
(cc/GET "/livereload" [] (make-livereload-handler))
(route/files "/" {:root "./output"}))
Javascript (client side):
function connect() {
const protocol = window.location.protocol === 'https:' ? 'wss://' : 'ws://';
const wsUrl = protocol + window.location.host + '/livereload';
var socket = new WebSocket(wsUrl);
socket.onclose = function(event) {
console.log("Websocket connection closed or unable to connect; " +
"starting reconnect timeout");
socket = null;
setTimeout(function() {
connect();
}, 5000)
}
socket.onmessage = function(event) {
console.log("Incoming event: " + event.data);
switch(event.data) {
case "livereload":
socket.close(1000, "Reloading page after receiving reload_request");
console.log("Reloading page after receiving reload_request");
location.reload(true);
break;
default:
console.log(`Don't know how to handle type '${data.type}'`);
}
}
}
connect();
Now, people (ok, who?) might ask why I didn't use ClojureScript for this and the answer is, I tried!
Equivalent-ish ClojureScript (client-side):
(ns kusachi-client.livereload
(:require [cljs.core.async :as a :refer [! go go-loop]]
[clojure.string :as str]
[haslett.client :as ws]
[haslett.format :as wsfmt]))
(defn- start-listener [stream]
(go-loop []
(when-some [message (<! (:in stream))]
(if (= (:message message) "livereload")
(js/console.log "Livereload command sent")
(.reload (.-location js/window)))
(recur))))
(defn- websocket-url [path]
(let [loc (.-location js/window)
proto (if (= "https:" (.-protocol loc)) "wss" "ws")]
(str proto "://" (.-host loc) path)))
(defn- websocket-connect [path]
(ws/connect (websocket-url path) {:format wsfmt/transit}))
(defn- on-load [_]
(js/console.log "Connecting to websocket")
(go (let [stream (<! (websocket-connect "/livereload"))]
(start-listener stream))))
(defn init []
(.addEventListener js/window "load" on-load))
However, even with shadow-cljs set to release, here's the size difference:
kusachi on HEAD (1ae72e8) [✘!?] via ☕ v21.0.12 via via ❄️ impure (nix-shell-env)
at 00:44:59 ❯ ls -la ./output/assets/js/livereload.cljs.js
.rw-r--r-- 245k kat 1 Aug 00:26 ./output/assets/js/livereload.cljs.js
kusachi on HEAD (1ae72e8) [✘!?] via ☕ v21.0.12 via via ❄️ impure (nix-shell-env)
at 00:45:58 ❯ ls -la ./resources/js/livereload.js
.rw-r--r-- 997 kat 31 Jul 22:27 ./resources/js/livereload.js
Oof. That is not worth it. Vanilla javascript is more than fine.
From here, I was relatively satisfied with things but there was one thing left... being able to change the underlying code and see that reflected in the site being updated. I thought a lot about how to do this! I was worried it may require having fancy dependency graphing and breaking the whole thing up into independent segments that each corresponded to a cacheable piece...
...but then I remembered Clojure is a dialect of Lisp.
§ Conjure and the REPL; playing to the strengths of Clojure or Lisps in general
Read–eval–print loop - WikipediaA read–eval–print loop (REPL), also termed an interactive toplevel or language shell, is a simple interactive computer programming environment that takes single user inputs, executes them, and returns the result to the user; a program written in a REPL environment is executed piecewise.
Olical/conjure is a wonderful project from the same person who made nfnl! It's like CIDER is for emacs but for your neovim. It supports Clojure and nREPL.
Using conjure, I could <localleader>eb (or ,eb in my case; my <localleader> is ,) to load and execute the contents of an entire neovim buffer within my nREPL session. Doing this on main.clj would pull in the entirety of kusachi. At the bottom of that file is a section of helpers I've left for myself:
(comment
; nrepl helpers
(continuous) ; watch, livereload and serve ...
(discontinue) ; stop watching, livereloading and serving ...
(generate true) ; regenerate the whole website with continuous (embed livereload javascript)
(stop-watchers) ; stop all of the watchers
(fs/stop-server) ; stop the webserver
(fs/start-server) ; start the webserver
)
With ,ee I could execute the S-expression highlighted under the cursor; by choosing a form to execute I could start the entire livereload + serve system.
What does this resolve? When an edit is made in a separate namespace, that buffer can be re-evaluated (,eb) and thus the namespace reloaded. By causing a (generate true) the entire site would then regenerate as necessary. Since this is a long-lived process, the JVM + Clojure compile (which took 4s typically) would already be done for all functions that had not changed! A rebuild of the site was actually fast (and also produced a livereload!) to the point where this was it, this was what I wanted!
It might not be perfect, it might not be automagic, but it works.
§ Atonement for Arborium; we'll take sintech
In the last blog post about kusachi, I spoke high-praise of arborium before learning it may as well be slopware. I later removed it from the website and yet, we have syntax highlighting and not just syntax highlighting, but better syntax highlighting than prior. Syntax highlighting that could even be considered incorrect because there is no way the blocks give a full indication of scope (what macros? what is in scope here?)!
I have decided to join the regex lover club, I have put together for myself sintech. It's simpler than arborium (it doesn't require cargo-xtask nor require me to produce a flake to manage cargo-xtask producing actual cargo crates!) for me. It's not perfect but it highlights dialects of lisp better than arborium did, for me. You've been seeing its output althroughout this post!
§ Conclusion
In many ways, I wouldn't want a (meta-)build system built in something like Clojure. I don't think it leverages much of what makes Clojure worthwhile! Sure, one can GraalVM their Clojure, embed sci, ... but like, I think I would rather keep a process alive. If I wanted something else, perhaps I would use salsa-rs/salsa or something like it! One could still embed a scripting language interpreter and maintain the advantages of a Lisp as such, including interpolation clauses being self-enclosing and only requiring a single opening character.
I can have livereload, I can have code reloading, I can be efficient enough such that I do not have to care and I can have this without maximal big braining. I like writing my Clojure data structures (hiccup is fine! why do I need to write HTML outside of the markdown? my "templates" (big hiccup data outputting functions) are fine. :D