mikkokoski.com

jj fix 🩷 standard-clj

Tue, 26 May 2026 17:30:29 +0000

For the past few weeks, I've been using Jujutsu (jj) as my primary version control system instead of git. I'm still learning, but I think I'm past the point of no return.

jj has a bunch of cool things, but one of my favorite commands is jj fix.

In this article, I'll describe what jj fix does and why it's better than doing the same with git rebase + git amend workflow, and how it integrates with my Clojure formatter of choice, standard-clj.

`jj fix` in a nutshell

jj fix updates files with formatting fixes. You can hook it up with a formatter of your liking, e.g. prettier or standard-clj or any tool that can take content from stdin and output it to stdout.

At Sharetribe, we started using standard-clj as the formatter for our Clojure code, and it works really well with jj fix.

Let's see how jj fix works.

So you forgot to run the formatter...

Here's the problem.

Let's say you made a commit "Hello, world", where you added the following piece of code:

(ns myapp)

(defn hello []
     (println "Hello, world")
    
    )

Notice the indentation, which is broken.

After that, you made a second commit "Bye bye", where you changed the "Hello, world" to "Bye bye":

(ns myapp)

(defn hello []
     (println "Bye bye!")
    
    )

You are now ready to commit your changes and open a PR. But then you realize: You forgot to run the code formatter.

With Git, you have two options:

Run the formatter and add a third commit just for the formatting fixes.
Do a git rebase --interactive and fix the formatting in the commits that introduced them.

The professional in you thinks that of course option 2 is the right thing to do, but since the interactive rebase with Git is kinda cumbersome, it's tempting to just be lazy and pick option 1.

The problem with the git rebase option is the following.

Let's do git rebase --interactive to edit the "Hello, world" commit

edit 36dc4f9 # Hello, world
pick 0057aed # Bye bye

# Rebase 0057aed onto 049568a (2 commands)

Now while rebase is ongoing and we are editing 36dc4f9, let's run standard-clj fix:

➜  git:(36dc4f9) ✗ standard-clj fix myapp.clj
standard-clj fix v0.28.0

F /myapp.clj [3.47ms]

1 file formatted with Standard Clojure Style 👍 [1067.49ms]

standard-clj successfully fixed (F) /myapp.clj

Let's amend the formatting fixes and continue rebase.

➜  git:(36dc4f9) ✗ git add myapp.clj
➜  git:(36dc4f9) ✗ git commit --amend

Now let's continue the rebase:

➜  git:(b98ec2c) git rebase --continue
Auto-merging myapp.clj
CONFLICT (content): Merge conflict in myapp.clj
error: could not apply 0057aed... Bye bye
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Could not apply 0057aed... # Bye bye

Oh no... as you might have expected, we have a conflict:

(ns myapp)

(defn hello []
<<<<<<< HEAD
  (println "Hello, world"))
=======
     (println "Bye bye!")
    
    )

>>>>>>> 0057aed (Bye bye)

What happened? We formatted the earlier "Hello, world" commit. Then we continued with the rebase. The "Bye bye" commit touches the same line that got formatted, so while git tries to apply the "Bye bye" commit diff, it realizes that the underlying "Hello, world" commit had changed, so it doesn't know how to apply the diff.

Enter `jj fix`

Let's do the same with jj fix.

I've initialized a new jj repo with the same two commits. Also, I've configured jj fix to use standard-clj for formatting (we'll see soon how that is done).

Here's how my jj log looks:

➜  jj:(pwl) jj log
@  pwlomtxr mikko@sharetribe.com 2026-05-19 22:58:54 122deafe
│  Bye bye
○  qoxyznmy mikko@sharetribe.com 2026-05-19 22:57:33 9d8ce0ae
│  Hello, world
◆  zzzzzzzz root() 00000000

We have the same two commits: "Hello, world" (qox) and Bye bye (pwl).

Now, let's run jj fix:

➜  jj:(pwl) jj fix
Fixed 2 commits of 2 checked.
Working copy  (@) now at: pwlomtxr 8cb76981 Bye bye
Parent commit (@-)      : qoxyznmy 5fd8b557 Hello, world
Added 0 files, modified 1 files, removed 0 files

Boom! 💥 jj fix found and fixed formatting issues in two commits, without any conflicts!

Let's confirm it actually formatted the code as expected and included the formatting fixes in correct commits. Here's the diff for "Hello, world" (qox) commit:

➜  jj:(pwl) jj diff -r qox
Added regular file myapp.clj:
        1: (ns myapp)
        2:
        3: (defn hello []
        4:   (println "Hello, world"))

Yep, the "Hello, world" commit is now correctly formatted.

And let's also see the diff for "Bye bye" (pwl) commit:

➜  jj:(pwl) jj diff -r pwl
Modified regular file myapp.clj:
   1    1: (ns myapp)
   2    2:
   3    3: (defn hello []
-  4     :   (println "Hello, world"))
+       4:   (println "Bye bye!"))

Looks good! No formatting fixes in this commit. The only change is the text from "Hello, world" to "Bye bye".

And of course, this works with any number of commits. Two commits are still easy to handle, but with 10+ commits where all of them have formatting issues, handling all this with git rebase would be very cumbersome.

Why no conflict with `jj fix`?

So you might wonder, how does jj fix do this without introducing conflicts as git did?

Here's what jj fix --help says about this (emphasis mine):

The changed files in the given revisions will be updated with any fixes determined by passing their file content through any external tools the user has configured for those files. Descendants will also be updated by passing their versions of the same files through the same tools, which will ensure that the fixes are not lost. This will never result in new conflicts.

What the help text is saying is that jj fix does not care about diffs. It edits a commit and formats the file content of the changed files (the snapshots of the files, not the diffs). Then it repeats that for the descendant commits. And indeed, this never results in conflicts. Neat!

And it's fast too!

Running jj fix with standard-clj is also very fast! The reason for this is that since jj is aware of which files changed, it doesn't run the formatter for the whole codebase, just to the files that changed. Of course, if you want to run it for the whole codebase, you can use the --include-unchanged-files flag.

How to configure `jj fix` to use `standard-clj`

⚠️ Make sure you're on standard-clj v0.28.0 or newer. See the note at the end of the post for why.

jj can be configured at three levels: user, repo and workspace.

The formatting tool configuration goes into the repo level. The repo configuration file is stored outside of the repository. The way to know the path is to run:

➜  jj:(ymq) jj config path --repo
/Users/mikko/.config/jj/repos/3c9fd428d14ad5b1a5e0/config.toml

So that's where you should add the configuration.

jj requires that the tool used for jj fix is able to take the content in stdin and output to stdout. standard-clj fix - does exactly that. In addition to the tool command, you need to specify glob patterns. Here's an example config that you can add to your repo-specific config.toml file:

[fix.tools.standard-clj]
command = ["standard-clj", "fix", "-"]
patterns = ["glob:'src/**/*.{clj,cljs,cljc,edn}'",
            "glob:'test/**/*.{clj,cljs,cljc,edn}'",
            "glob:'deps.edn'"]

Important! Use `standard-clj --version` v0.28.0 or newer

When I first started using standard-clj with jj fix, I noticed a weird bug. Some of my files, especially the longer ones with 1000+ lines, were sometimes just cut off. And since jj fix runs the formatting for each commit, and the files were cut off in some earlier commit, you can probably imagine that this got my repository in a super broken state 😄.

Luckily, I was able to find the bug and provide the fix. This fix was released in standard-clj version v0.28.0, so make sure you use that version or newer.

How to catch derived Vars with a clj-kondo hook

Sun, 04 Feb 2024 14:38:36 +0000

I love Clojure's REPL-driven development workflow.

The fast feedback loop I get when I evaluate the changed code and immediately see the result brings me joy every day.

But... I do have one gripe: derived Vars.

Let's take an example:

(defn welcome-page-handler []
  {:status 200
   :body "Welcome!"})

(def routes
  {"/" welcome-page-handler})

In the example above, we have a very simple web app skeleton with a handler function welcome-page-handler and a routes map from URLs to handler functions.

Let's try it out:

(let [handler-fn (get routes "/")]
  (handler-fn))

;; #=> {:status 200, :body "Welcome!"}

We simulated an HTTP request to the homepage path "/". We did "routing" with get and called the handler function for the "/" route, and the result was as expected. All good!

Now, let's make a small change to the welcome-page-handler function and re-evaluate it.

(defn welcome-page-handler []
  {:status 200
   :body "Welcome, stranger!"})

;; #=> #'user/welcome-page-hander

Let's test the handler again:

(let [handler-fn (get routes "/")]
  (handler-fn))

;; #=> {:status 200, :body "Welcome!"}

Dang! That's not what we wanted to see, right? We were expecting the new "Welcome, stranger!" greeting.

Why did this happen?

The reason is that the routes map still points to the old version of the welcome-page-handler function. To make the routes point to the new version, we must also re-evaluate the routes map too.

So what's the big deal? Just re-evaluate routes!

In a simple example like this, re-evaluating the routes after we change the welcome-page-handler (or any other function handler in the routes map) is a viable option. However, in a larger codebase, the handler function might be in different namespaces where the routes map is. It is cumbersome to re-evaluate routes every time you make a change to a handler. Changes are high that you will forget to do so.

Ok, so this is clearly a problem. Is there a solution?

Well, good news! Yes, there is a solution! Three solutions, actually:

Use reloaded workflow
Var quoting with the #' syntax
Function wrapping

Let's go through these options.

Option 1: Use reloaded workflow

At Sharetribe, we've been developing our Clojure app for over nine years. Since the beginning, we've been using the reloaded workflow and the reloaded.repl library to solve the derived Var issue. It has served us well.

The idea in a nutshell is to have a (user/reset) function that utilizes tools.namespace to reload the namespaces that have changed and the namespaces that depend on the changed namespaces.

So if you have a routes namespace that requires handlers namespace and the handlers namespace changes, reloaded.repl will reload both namespaces. This solves the issue with derived Vars.

This used to work well for us. But after nine years of development and 100k+ LOC of clj/c/s code, things started to get slow.

In the Clojure Workflow Reloaded article, Stuart Sierra writes the following (emphasis mine):

Therefore, after every significant code change, I want to restart the application from scratch. But I don't want to restart the JVM and reload all my Clojure code in order to do it: that takes too long and is too disruptive to my workflow. Instead, I want to design my application in such a way that I can quickly shut it down, discard any transient state it might have built up, start it again, and return to a similar state. And when I say quickly, I mean that the whole process should take less than a second.

This is something that can't be achieved with a large code base. Let me demonstrate.

If I'm in Sharetribe's codebase and I make a change to one of the endpoint handler functions and call refresh in our user namespace, it usually takes 3-4 seconds:

user> (time
       (clojure.tools.namespace.repl/refresh))
:reloading (...)
"Elapsed time: 4680.926875 msecs"
;; => :ok

That's too long, in my opinion.

Options 2: Var quoting

Var quoting means that you use a special reader syntax #' so that the routes map would look like this:

(def routes
  ;; see the #' in front of the welcome-page-handler
  {"/" #'welcome-page-handler})

We can use the routes map precisely the same way we did earlier, with the exception that changes in the welcome-page-handler are reflected without re-evaluating the routes map:

(let [handler-fn (get routes "/")]
  (handler-fn))

;; #=> {:status 200, :body "Welcome, stranger!"}

What happens there is that the value of the routes map isn't a function anymore; it's a Var. Those two aren't equal unless the Var is dereferenced:

welcome-page-handler
;; #=> #function[user/welcome-page-handler]

#'welcome-page-handler
;; #=> #'user/welcome-page-handler

(= welcome-page-handler #'welcome-page-handler)
;; #=> false

(= welcome-page-handler @#'welcome-page-handler)
;; #=> true

Now, if the handler function isn't a function anymore, but a Var, how can we invoke the function without first deref'ing the Var? The reason is that when Var is used in the function position (the first element in the list), it is deref'd automatically. Var implements IFn interface.

Because of this, the two examples below behave the same:

(#'welcome-page-handler)
;; #=> {:status 200, :body "Welcome, stranger!"}

(@#'welcome-page-handler)
;; #=> {:status 200, :body "Welcome, stranger!"}

Option 3: Function wrapping

Function wrapping means that you wrap the handler function with another function, like this:

(def routes
  ;; see the anonymous function #( ) around the welcome-page-handler
  {"/" #(welcome-page-handler)})

Out of these two options, Var quoting and function wrapping, the #' approach is the one I'd prefer in most cases. It is, in my opinion, less noisy syntax.

However, #' should be avoided in CLJS (and thus also in CLJC).

How to find all derived Vars?

Ok, so, now we know what are our options to fix the derived Vars.

But there are still two open questions:

How can I find all the derived Vars in my codebase? Derived Vars are kinda subtle, aren't they?

We've been building our app without worrying about derived Vars for nine years because the reloaded workflow solved the issue. The result is that there are derived Vars here and there in the code base. We have many maps that do some kind of dispatching from a name or a keyword to an appropriate handler function. Derived Vars are sprinkled everywhere.

In addition to finding all the derived Vars, I want solve this issue once and for all. If I now go through our codebase and fix all the derived Vars, I don't want to do it again later. How can I prevent anyone from introducing derived Vars in the future? As I already mentioned, they are subtle and easily introduced by accident.

I once asked about this in the ClojureVerse discussion forum. The conclusion was that there isn't an existing solution out there to fix the issue with derived Vars once and for all, but clj-kondo might be able to get me somewhere.

clj-kondo has a nice feature, hooks, that allows enhancing linting via user-provided code.

And it turned out that it's possible to write a clj-kondo hook that will catch the derived Vars. Here's how.

The hook 🪝

Ok, let's get to it. Let's build the hook, piece by piece.

Let's start by editing the .clj-kondo/config.edn file. We need to define the hook and add the linter:

;; .clj-kondo/config.edn

;; &#9312; Analyze both CLJ and CLJS `def`s with `hooks.def/analyze`.
;; This hook will emit `:fn-sym-in-def` findings.
{:hooks {:analyze-call {clojure.core/def hooks.def/analyze
                        cljs.core/def hooks.def/analyze}}

 :linters {
           ;; &#9313; clj-kondo-config linter will complain about 
           ;; unknown linter `fn-sym-in-def`, unless we ignore it.
           #_{:clj-kondo/ignore [:clj-kondo-config]}
 
           ;; &#9314; Set `fn-sym-in-def` level to warning
           :fn-sym-in-def {:level :warning}}}

That's all we need in the config file for now.

In the config file, we're pointing to hooks.def namespace. Next, we'll create it.

Add a new CLJ file in .clj-kondo/hooks/def.clj, with the following content:

(ns hooks.def
  (:require [clj-kondo.hooks-api :as api]))
  
(defn analyze
  [{:keys [node lang]}]

  ;; &#9312; Print the `node`, just to try things out.
  (println node)

  ;; &#9313; Return the `node` without transforming it.
  node)

That's the skeleton for our hook. You can now try to run clj-kondo and see the defs in your project to be printed out:

✗ clj-kondo --lint src
&lt;list: (def routes {"/" welcome-page-handler})&gt;
linting took 28ms, errors: 0, warnings: 0

Since we are about to analyze the form def, it's good to keep in mind that def can have multiple arities:

;; no initial value
(def foo)

;; initial value
(def bar :bar-value)

;; docstring and initial value
(def quax "Quax docstring" :quax-value)

We're not interested in the first form; we can ignore it. Also, we are not interested in the docstring. We only want to inspect the last child of the node.

Let's add the logic to the analyze function to call an auxiliary function analyze*:

(defn analyze
  [{:keys [node lang]}]

  ;; &#9312; Call `analyze*` with the last child node
  (when (< 1 (count (:children node)))
    (analyze* {:node (last (:children node))
               :lang lang}))

  ;; &#9313; Return the `node` without transforming it.
  node)

Now, let's implement the analyze* function.

The analyze* will have a cond with two branches. Let's start with the first branch in which we call a fn-token-node? predicate, and if the node indeed is a token node pointing to a function, we record the finding and recommend a fix based on the language.

(defn- analyze* [{:keys [node lang]}]
  (cond
    
    ;; &#9312; Call predicate function that returns true for symbols pointing to functions
    (fn-token-node? {:node node 
                     :lang lang})
                     
    ;; &#9313; Record finding if predicate returns truthy
    (api/reg-finding! (assoc (meta node)
                             :type :fn-sym-in-def
                             :message (str "fn-sym-in-def: "
                                           node
                                           (if (= :cljs lang)
                                             " - use function wrapping"
                                             " - use var quoting"))))))

And now, let's implement the fn-token-node? predicate, which is the beef of the hook.

(defn- fn-token-node? [{:keys [node lang]}]
  ;; &#9312; Check that the node is a symbol token node
  (when (and (api/token-node? node)
             (symbol? (:value node)))
             
    ;; &#9313; Resolve the symbol to `ns` and `name`
    (let [{:keys [ns name]} (api/resolve {:name (:value node)})
          
          ;; &#9314; Get the _cached_ analysis data
          analysis (get-in (api/ns-analysis ns {:lang lang}) [lang name])]
          
      ;; &#9315; Ignore `clojure.core` and `cljs.core`
      (when-not (#{'clojure.core
                   'cljs.core} ns)
                   
        ;; &#9316; Return truthy if the analysis data looks like it's a function
        (some (set (keys analysis)) [:fixed-arities :varargs-min-arity])))))

First, we check if the node is a symbol token node.

After that, we use clj-kondo's api/resolve function to resolve the namespace and the name of the symbol.

Then, we use the resolved namespace and name to get the cached analysis data from clj-kondo.

Next, we filter out clojure.core and cljs.core namespaces.

If the returned analysis data looks like a function (i.e. it has either :fixed-arities or :varargs-min-arity key), we return truthy value from the predicate.

Now, we do want to do a bit of traversing in case the value is a Clojure collection. Only then we can catch the example case shown earlier we have a routing map from keys to handler functions. So let's add another predicate function and another branch to the cond and resursively traverse:

(defn- traverse? [{:keys [node]}]
  (or
   (api/vector-node? node)
   (api/map-node? node)
   (api/set-node? node)))

(defn- analyze* [{:keys [node lang]}]
  (cond
    (fn-token-node? {:node node 
                     :lang lang})
    (api/reg-finding! (assoc (meta node)
                             :type :fn-sym-in-def
                             :message (str "fn-sym-in-def: "
                                           node
                                           (if (= :cljs lang)
                                             " - use function wrapping"
                                             " - use var quoting"))))
                          
    ;; &#9312; Check if we should recursively traverse
    (traverse? {:node node}) 
    
    ;; &#9313; Traverse all child nodes.
    ;; Note the `doall` which is needed because `reg-finding!` is
    ;; a side-effecting function inside `map`
    (doall (map #(analyze* % lang) (:children node)))))

Alright, that's pretty much all we need! Now we can try it out to see if it catches the derived Vars in a router map.

➜  clj-kondo --lint src
src/core.clj:8:8: warning: fn-sym-in-def: welcome-page-handler - use var quoting
linting took 27ms, errors: 0, warnings: 1

That's exactly what we wanted to see! We got a warning saying that welcome-page-handler needs to be Var quoted.

To see the whole thing, have a look at the example repository at https://github.com/rap1ds/derived-vars-clj-kondo-hook

Caveats

There are some caveats in the hook:

First, the hook uses clj-kondo.hooks-api/ns-analysis, which utilizes clj-kondo cache. The problem is that if the cache is empty, you get false negative results, i.e. you don't get warnings of derived Vars.

When you're developing on your local environment, you likely have a populated clj-kondo cache. However, in a CI environment, you will probably start from a clean slate with an empty cache. The best workaround that I'm aware of is to run clj-kondo twice. The first time to populate the cache, and then the second time to do the actual linting. Obviously, this doubles the time it takes to lint the project. Luckily, clj-kondo is pretty fast.

The second caveat is that the traversing part is rather naive and only supports the most common Clojure data types. If you wrap the Var in e.g. when or partial etc, you won't get a linter warning. However, I feel that the current implementation is good enough and catches the most common cases.

Conclusion

Derived Vars are, in my opinion, one of the most biggest annoyance in the REPL-driven development workflow.

We can use Var quoting or function wrapping to make derived Vars REPL friendly in a large codebase where namespace reloading with tools.namespace is too slow.

It's quite easy to introduce derived Vars by accident, so a solution like linter check needs to be implemented to make sure we don't accidentally introduce them.

clj-kondo hooks are extremely powerful and can be used to analyze def forms, register findings, and report warnings for derived Vars.

With the help of the clj-kondo hook, I was able to identify and fix over 300 derived Vars from our codebase. The codebase is now a bit more friendly for REPL-driven development!

If you have any questions or comments, I'd love to hear! Please leave a comment on Mastodon.

Clojure set with custom equality semantics

Thu, 07 Sep 2023 19:34:47 +0000

Sets are collections of unique values.

To be able to eliminate the multiplicates, the set needs to know when two values are equal.

Sets use the standard Clojure equality semantics, meaning that if two values are =, then it removes the multiplicates.

Example:

(= :foo :foo)
;; #=> true

#{:foo :bar :foo}
;; #=> #{:foo :bar}
;; One of the two :foo values is removed because they are equal (`=`)

But what if you want to have a set with custom equality semantics?

Say you have a list of person entities, where each person has a :person/id attribute that identifies the person. When you put the person entities in a set, you want entities with unique :person/ids, no matter what other data the entity contains.

Example:

#{{:person/id 1
   :person/firstName "Mikko"}
  {:person/id 2
   :person/firstName "John"
   :person/lastName "Doe"}
  {:person/id 1
   :person/lastName "Koski"}} 
;; #=> #{#:person{:id 1, :firstName "Mikko"}
;;       #:person{:id 1, :lastName "Koski"}
;;       #:person{:id 2, :firstName "John", :lastName "Doe"}}

;; Note that there are two entities with `:person/id` 1, which is not what we want.

How could we create a set that uses the :person/id attribute to decide if two values are equal?

Custom equality with `sorted-set-by`

Sets are unordered data structures, but you can create ordered sets with sorted-set. Using sorted-set-by, you can have a sorted set with your own comparator for sorting.

The sorted-set has another interesting property: In addition to the sort order, the comparator also defines the equality semantics, meaning that if the comparator thinks that two values are equal (i.e., the comparator returns 0), then the set threats them as equal values, that is, it removes multiplicates.

So for example, we can make a comparator that compares only the :person/id attribute and return 0 for two maps with the same :person/id, regarless of the other keys and values in the map:

(defn compare-id [x y]
  (compare (:person/id x) (:person/id y)))
  
(compare-id
 {:person/id 1
  :person/firstName "Mikko"}
 {:person/id 1
  :person/lastName "Koski"})
;; #=> 0
;; Zero means the values are equal from comparator point-of-view

And if we use compare-id as the comparator with sorted-set-by, we get the result we wanted:

(def s
  (sorted-set-by
   compare-id
   {:person/id 1
    :person/firstName "Mikko"}
   {:person/id 2
    :person/firstName "John"}
   {:person/id 1
    :person/lastName "Koski"}))

s
;; #=> #{#:person{:id 1, :firstName "Mikko"} 
         #:person{:id 2, :firstName "John"}}

Do sets with custom equality work with `disj`, `contains?` and `get`?

Yes, they do.

These functions are documented as follows in Clojure reference:

Sets support 'removal' with disj, as well as contains? and get, the latter returning the object that is held in the set which compares equal to the key, if found:
-- Clojure - Data Structures

Here's an example that demonstrates that everything works as documented:

(disj s {:person/id 1 :person/lastName "Koski"})
;; #=> #{#:person{:id 2, :firstName "John"}}

(contains? s {:person/id 1 :person/lastName "Koski"})
;; #=> true

(get s {:person/id 1 :person/lastName "Koski"})
;; #=> #:person{:id 1, :firstName "Mikko"}

What about `clojure.set` operators?

So now that we've set up our set of person entities with custom equality by :person/id and tested that the essential functions disj, contains? and get work as expected, the next question is, do the set operations provided by the clojure.set namespace work with sets with custom equality?

The answer is yes, but with one caveat:

Do not mix different set types.

If we look at the source code of clojure.set, we can see that the functions don't create new sets for the return values; instead, they reuse the ones given as parameters. This is good news because we don't want to lose our custom equality semantics after using the clojure.set operations.

However, the functions in clojure.set use count of the given sets to determine which one will be used as a base for the return value. You shouldn't rely on this logic, so it's safest to use two sets of the same type.

Let's take an example.

First, let's call clojure.set/intersection with a sorted set of two values and a normal unsorted set of one value:

(def result-when-sorted-set-has-more-values
  (clojure.set/intersection
   (sorted-set-by
    compare-id
    {:person/id 1
     :person/firstName "Mikko"}
    {:person/id 2
     :person/firstName "John"})
   #{{:person/id 2
      :person/lastName "Doe"}}))

result-when-sorted-set-has-more-values
;; #=> #{#:person{:id 2, :lastName "Doe"}}

(sorted? result-when-sorted-set-has-more-values)
;; #=> false

(contains? result-when-sorted-set-has-more-values {:person/id 2})
;; #=> false

In the example above, the returned set has the value we expected but is not sorted, nor does it use the equality by :person/id.

Next, let's have a sorted set of one value and an unsorted set of two values:

(def result-when-unsorted-set-has-more-values
  (clojure.set/intersection
   (sorted-set-by
    compare-id
    {:person/id 2
     :person/firstName "John"})
   #{{:person/id 1
      :person/firstName "Mikko"}
     {:person/id 2
      :person/lastName "Doe"}}))

result-when-unsorted-set-has-more-values
;; #=> #{}

(sorted? result-when-unsorted-set-has-more-values)
;; #=> true

(contains? result-when-unsorted-set-has-more-values {:person/id 2})
;; #=> false

Now we did got back a sorted set, but the set didn't have any values, which is not what we wanted.

To get the correct result and correct set type, we have to use the same set type for both arguments:

(def result-when-both-sets-are-sorted
  (clojure.set/intersection
   (sorted-set-by
    compare-id
    {:person/id 2
     :person/firstName "John"})
   (sorted-set-by
    compare-id
    {:person/id 1
     :person/firstName "Mikko"}
    {:person/id 2
     :person/lastName "Doe"})))

result-when-both-sets-are-sorted
;; #=> #{#:person{:id 2, :firstName "John"}}

(sorted? result-when-both-sets-are-sorted)
;; #=> true

(contains? result-when-both-sets-are-sorted {:person/id 2})
;; #=> true

Conclusion

Using the sorted-set-by with custom equality semantics was a fun thing to learn, but I'm still unsure if there are any real world use-cases out there where this would be actually useful.

In the examples above, I had two sets of person entities with slightly different attributes, and the aim was to eliminate the duplicates. In reality, I'd probably do that with something like distinct-by (if it existed), or I'd use group-by and then map the vals with merge to get distinct entities with the attributes merged, which is probably what I'd want anyway:

(->> 
 [{:person/id 1
  :person/firstName "Mikko"}
  {:person/id 2
   :person/firstName "John"}
  {:person/id 1
   :person/lastName "Koski"}]
 (group-by :person/id)
 vals
 (map #(apply merge %)))
;; #=> (#:person{:id 1, :firstName "Mikko", :lastName "Koski"}
;;      #:person{:id 2, :firstName "John"})

But maybe if I'd need to do actual set operations (difference, intersection etc.) with the two sets of entities, then I'd maybe use the sorted-set-by with custom equality. Maybe.

What do you think? Is there any real-world use for this?

Please let me know and comment on Mastodon!

8 tips for Advent of Code 2022 🎄

Wed, 23 Nov 2022 17:46:00 +0000

Advent of Code 2022 starts in 8 days!

I've been participating in AoC since 2017. My highlight was in 2020 when I solved all the puzzles and received 50/50 stars ⭐️ (though that didn't happen in December 2020 but only much later in 2021).

Since 2017, I've learned a lot. I'm sharing some tips/notes to self for the upcoming AoC 2022.

Here are eight tips for AoC 2022:

1. Keep it fun!

Let's start with the most boring yet important tip.

Remember to keep it fun! Don't stress if you can't solve a puzzle in a day. Don't let it ruin your Christmas! Christmas time can be stressful enough even without AoC.

Also, don't stress about the leaderboard. People who are on the leaderboard have practiced these kinds of coding puzzles for years.

2. Use the right data structure

First "real" coding tip:

Using the right data structure might make solving a challenging puzzle a piece of cake.

Let's take an example:

You need to have an XY-coordinate system, a grid
The grid has an infinite size
Initial value for point [0, 0] is 1
Initial value for everything else is 0
You need to have fast random access (e.g., quickly access a value of any given XY-point)

What would be a proper data structure for this?

At first, by intuition, you might think about vectors in a vector (like a 2D array) for an XY-coordinate system. But things get tricky due to the other requirements, i.e., infinite size.

A perfect data structure for this is a map from XY-points to values.

;; Initial value for 0,0 is 1
      (def grid {[0 0] 1})

      ;; Fast random access and an initial value for everything else than 0,0 is 0
      (get grid [0 0] 0) ;; #=> 1
      (get grid [12 34] 0) ;; #=> 0
      (get grid [-56 78] 0) ;; #=> 0

      ;; Infinite size
      (def new-grid (assoc [100 100] grid 100))
      ;; #=> {[0 0] 1, [100 100] 100}

Another useful data structure is a set. You'll find plenty of opportunities to use sets in AoC puzzles.

3. How to make your solution fast? Tip 1/2: `memoize`

AoC puzzles have two parts.

A typical puzzle pattern is this: The first part is easy to solve. You can easily craft a naive, brute-force implementation that does the trick.

The second part is such that you need to change the first part only a bit. It looks effortless until you run it and notice that it takes forever to finish.

For example, in the first part, you need to define a step function (a function that takes as an input the output of the previous step function execution) and run the step 100 times for a given input. Running the step function 100 times takes couple of seconds.

In the second part, you need to run it 1 000 000 times. You notice that this will take hours to complete.

How to make your solution run faster for the second part?

One helpful strategy is memoization: Make your function pure (as we default in Clojure) and throw some memoize in the mix. That itself might make the function fast enough.

4. How to make your solution faster? Tip 2/2: Find a repeating pattern

Not all solutions can be memoized. In those cases, you need to figure out something else.

One thing you can look for is to find a repeating pattern.

Say you have a step function, and you've been asked what the result of running that step function 549027529283424023592234980235235 times (or some other ridiculously high number) is. You can't just repeatedly run the function that many times, or it will take years to execute.

It's possible that you can find a repeating pattern where the output of, say, the 12514th execution is the same as the output of the first run. You can now calculate 549027529283424023592234980235235 modulo 12514 and figure out that it is enough to run the step function only 8633 times to get the result.

5. Make heavy use of the examples

Each AoC puzzle will provide examples that are way smaller than the actual puzzle input.

Make sure you test your implementation with _all_ of the provided examples before even trying to run your solution with the actual input.

6. Put inputs and examples in `resources/`

The organizers of AoC recommend that people refrain from pushing their puzzle inputs to public version control like Github.

I organized my AoC project so that I put the inputs and examples in the resources/ directory and .gitignore it.

I think it's helpful to have the examples also in the resources/ so they are easy to use from there.

Also, it helps if you have a helper function to read the file lines from the resources/.

7. Keep your code modular

Because AoC puzzles have two parts, it's a good idea to keep your solution of the first part modular and split it into small functions that you can replace for the second part.

I've noticed that I often use threading macro ->. A typical puzzle solution might look something like this:

(-> (read-input-from-resources "name-of-the-file")
    (parse-input)
    (calculate-one-thing)
    (calculate-another-thing)
    (calculate-the-final-checksum))

Many times the requirements for part 2 are such that you need to change only one of these functions. A typical solution for part 2 looks something like this:

(-> (read-input-from-resources "name-of-the-file")
    (parse-input)
    (calculate-one-thing)
    (calculate-another-thing-a-bit-differently-for-part-2)
    (calculate-the-final-checksum))

8. If you can't solve a puzzle, it doesn't mean that you're a lousy coder

Final yet crucial tip (repeating the tip #1 a bit)

Don't feel bad if you can't solve a puzzle. It doesn't mean that you're a terrible coder. More likely, it means that you need to practice solving puzzles.

When we do our work as software engineers, we aren't solving puzzles on a daily basis. We are solving real business problems. Solving puzzles is very different from solving business problems and requires different skills. And you'll learn those skills by solving more puzzles.

I've witnessed this myself. When I started doing AoC, I managed to do only a few days and then got stuck. The puzzles just felt way too tricky.

And this felt bad. I thought I knew how to code!

But in 2020, when I completed all the puzzles, many of the puzzles' problems felt familiar and easy. The same patterns repeat, year after year, with only slight changes. I recognized how to solve a problem because I'd seen that same problem in earlier years.

So, I'm repeating myself, but this is important: Keep it fun, and don't feel bad if you can't solve all of them.

ClojureScript REPL Workflow with Figwheel

Tue, 15 Nov 2022 17:50:00 +0000

Roman Liutikov wrote a great piece on bringing REPL workflow to ClojureScript.

In the article, Roman shows a neat trick to override ClojureScript's built-in Browser REPL function for evaluating JavaScript code. The overridden version calls a render function, which will rerender the app after each REPL evaluation. You'll see the changes in your UI component immediately after you've evaluated them in REPL.

But hey, what about Figwheel? I'm not using Browser REPL. I'm using Figwheel! How could I do the same with Figwheel?

Rerendering after eval in Figwheel

The function that needs to be overridden in Figwheel is figwheel.repl.eval-javascript**

If we take Roman's snippet as a base and modify it a bit for Figwheel, we get this:

(comment
  (let [eval-js (.. js/figwheel -repl -eval_javascript_STAR__STAR_)]
    (set! (.. js/figwheel -repl -evaluate_javascript_STAR__STAR_)
          (fn [& args]
            (let [ret (apply eval-js args)]
              (when (= :success (:status ret))
                (render)) ;; your render function
              ret)))))

Call this function once to set it up and start evaluating changes in the REPL. You'll see your app rerendering and reflecting all UI changes immediately.

Async rerendering

If you notice, like I did, that Figwheel REPL starts to time out quite often, your rendering function execution is taking too long.

I was able to fix this by making the rerendering async.

Here's an improved version:

(comment
  (let [eval-js (.. figwheel -repl -eval_javascript_STAR__STAR_)]
    (set! (.. js/figwheel -repl -evaluate_javascript_STAR__STAR_)
          (fn [& args]
            (let [ret (apply eval-js args)]
              (when (= :success (:status ret))
                (js/setTimeout ;; async rerender
                 (fn [] (render)))) ;; your render function
              ret))))))

But why?

Figwheel already implements code hot-loading, so why is this needed?

When the project size grows, Figwheel code hot-loading start to take some time. It's not terrible, but it's not fast, either. It takes seconds for Figwheel to catch up on the newest changes after the file is saved.

The REPL method instead is instant, and that feels pretty great.

My experience has been that rerendering after REPL eval works seamlessly with Figwheel's own hot-loading. You might first eval the change in REPL, see it rerendered, then save the file and soon see the ClojureScript logo on the corner indicating that Figwheel has also caught up on the latest changes.

ClojureScript update from 1.10.439 to 1.11.60

Thu, 25 Aug 2022 19:34:00 +0000

We recently upgraded our ClojureScript version from 1.10.439 to 1.11.60. This is a rather big jump. 1.10.439 was released on Nov 2, 2018, while 1.11.60 was released on Jun 23, 2022. That's almost three and a half years! Although the update went well, there were few things that needed special attention.

Long.fromString instead of Long/fromString

The way how we called the fromString function from goog.math.Long stopped working. We got a compilation error saying that fromString didn't exist.

Here's how we used to require and call the function:

(ns "my.namespace"
  (:require [goog.math.Long]))

(goog.math.Long/fromString "123")

After CLJS upgrade, the compiler threw an error saying that fromString doesn't exist.

We changed the code to this:

(ns "my.namespace"
  (:import [goog.math Long]))

(goog.math.Long.fromString "123")

import + dot works well, and it seems to be also the recommended way, according to the import function examples in the documentation.

language-in: es-next

New Google Closure Library seems to contains JavaScript code that utilizes "Optional catch binding". What this means is that usually, when you catch an error in JavaScript, you bind the error to a variable, like this:

try {
  // something
} catch (error) {
  // the exception is bound to 'error'
}

In the newer JavaScript version (ES10) it's optional to bind the exception to a variable. The following works just fine:

try {
  // something
} catch {
  // no binding
}

Google Closure Library contains JavaScript code that utilizes optional catch binding. Because of that, we got an error "ERROR - [JSC_LANGUAGE_FEATURE] This language feature is only supported for ECMASCRIPT_2019 mode or better: Optional catch binding. ".

The solution is to change the :language-in compiler option to :ecmascript-next. And AFAIK, there's no need to change the :language-out parameter.

The issue has also been discussed in Ask Clojure forum.

Upgrade devtools

The compiler showed a warning about devtoolsusing an undeclared var. This seems to be a bug in ClojureScript.

The warning went away by upgrading devtools to 1.0.6

Change the order in which JS files are required

Okay, now that we fixed all the errors and warnings we saw in development mode, it was time to compile the JS bundle with advanced compilation. This revealed couple more issues.

We use CodeMirror from CLJSJS with XML and Handlebars modes. Apparently, something has changed in the order in which the modules are required.

What we earlier had was this:

(ns code-editor-namespace
  (:require
   [codemirror :as codemirror]
   [cljsjs.codemirror.mode.xml]
   [cljsjs.codemirror.mode.handlebars]
   [cljsjs.codemirror.addon.mode.simple]
   [cljsjs.codemirror.addon.mode.multiplex]

However, this didn't work anymore. The reason was that Handlebars mode requires simple mode, thus, I needed to change the order in which they are required:

(ns code-editor-namespace
  (:require
   [codemirror :as codemirror]

   ;; Load addons first before language modes
   [cljsjs.codemirror.addon.mode.simple]
   [cljsjs.codemirror.addon.mode.multiplex]

   [cljsjs.codemirror.mode.xml]
   [cljsjs.codemirror.mode.handlebars]

The change makes sense. However, what I don't know is why this was working earlier. My guess is that because this appeared in advanced compilation, maybe something has changed in Google Closure Compiler.

goog.object needs to be required before use

Next issue with advanced compilation: goog.object needs to be required in the namespace that uses it. This wasn't the case with the earlier version, but was changed in the ClojureScript 1.10.891 release.

Fix was easy, I just added the requires in the namespaces that didn't have those.

Conclusion

Although a few things needed special attention, I'm suprised that there weren't more of those. As mentioned, the version from which I updated was released more than three and a half years ago!

ClojureScript is heavily using Google Closure Compiler and Google Closure Library. I didn't go through all the changelogs, but my assumption is that many of there "breaking changes" were actually something that were changes in Google Closure instead of changes in ClojureScript itself.

mikkokoski.com

jj fix 🩷 standard-clj

jj fix in a nutshell

So you forgot to run the formatter...

Enter jj fix

Why no conflict with jj fix?

And it's fast too!

How to configure jj fix to use standard-clj

Important! Use standard-clj --version v0.28.0 or newer

How to catch derived Vars with a clj-kondo hook

Option 1: Use reloaded workflow

Options 2: Var quoting

Option 3: Function wrapping

How to find all derived Vars?

The hook 🪝

Caveats

Conclusion

Clojure set with custom equality semantics

Custom equality with sorted-set-by

Do sets with custom equality work with disj, contains? and get?

What about clojure.set operators?

Conclusion

8 tips for Advent of Code 2022 🎄

1. Keep it fun!

2. Use the right data structure

3. How to make your solution fast? Tip 1/2: memoize

4. How to make your solution faster? Tip 2/2: Find a repeating pattern

5. Make heavy use of the examples

6. Put inputs and examples in resources/

7. Keep your code modular

8. If you can't solve a puzzle, it doesn't mean that you're a lousy coder

ClojureScript REPL Workflow with Figwheel

Rerendering after eval in Figwheel

Async rerendering

But why?

ClojureScript update from 1.10.439 to 1.11.60

Long.fromString instead of Long/fromString

language-in: es-next

Upgrade devtools

Change the order in which JS files are required

goog.object needs to be required before use

Conclusion

`jj fix` in a nutshell

Enter `jj fix`

Why no conflict with `jj fix`?

How to configure `jj fix` to use `standard-clj`

Important! Use `standard-clj --version` v0.28.0 or newer

Custom equality with `sorted-set-by`

Do sets with custom equality work with `disj`, `contains?` and `get`?

What about `clojure.set` operators?

3. How to make your solution fast? Tip 1/2: `memoize`

6. Put inputs and examples in `resources/`