Journal 2019.22 - spec updates, tools.deps

Spec 2 - closed spec checking redo

I’ve mostly been doing spec 2 work in the last couple weeks. I finally completed the closed check rewrite that I started a while back. When Ilast visited closed spec checking, we had gone down the path of a stateful approach to managing closed specs. After some reflection and feedback from the community, this has been replaced.

The first big API level change is that all of the conform-oriented API calls ( conform , valid? , explain , explain-data ) now take an optional last argument settings which is a map keyed by setting. Initially there is just one setting available :closed , which takes a set of closed spec names. A brief example of closed spec checking:

(require '[clojure.spec-alpha2 :as s])
(s/def ::f string?)
(s/def ::l string?)
(s/def ::s (s/schema [::f ::l]))

;; "extra" keys are ok normally - open maps are the default
(s/valid? ::s {::f "Bugs" ::l "Bunny" ::x 10})
;;=> true

;; but closed spec checking can be more restrictive
(s/valid? ::s {::f "Bugs" ::l "Bunny" ::x 10} {:closed #{::s}})
;;=> false

(s/explain ::s {::f "Bugs" ::l "Bunny" ::x 10} {:closed #{::s}})
#:user{:f "Bugs", :l "Bunny", :x 10} - failed: (subset? (set (keys %)) #{:user/f :user/l})

One note around this change is its a pretty big change internally to the Spec protocol. It is pretty helpful though in opening up an avenue to other kinds of conditional checks in the future without needing any new api changes.

Spec 2 - symbolic data specs

One use case that we wanted to work on in spec 2 centers around programmatic creation and transformation of symbolic specs. While it’s possible to do this with symbolic spec forms, and the support for that has even been extended in spec 2 to avoid the macro layer entirely, the form-oriented nature has been a barrier to some.

As such, we are working on introducing a symbolic map format for ops that will parallel the symbolic form for ops. Parts of this are in master now but it’s under active work still and will likely change a bit.

But a quick example in its current form:

;; creating a spec with symbolic form
(s/def ::of (s/or :k keyword? :i int?))

;; creating the same spec with symbolic map instead
;; s/spec* is programmatic entry point to create spec objects from symbolic specs
;; s/register is a function variant of s/def
(s/register ::om 
  (s/spec* 
    `{:clojure.spec/op s/or :keys [:k :i] :specs [keyword? int?]}))

;; they produce the same spec in the registry
(s/conform ::of 100)
;;=> [:i 100]

(s/conform ::om 100)
;;=> [:i 100]

(s/explain ::om "abc")
"abc" - failed: keyword? at: [:k] spec: :user/om
"abc" - failed: int? at: [:i] spec: :user/om

(s/form ::om)
;;=> (clojure.spec-alpha2/or :k clojure.core/keyword? :i clojure.core/int?)

;; new operation to expand a form to a map
(s/expand-spec `(s/or :k keyword? :i int?))
;;=> {:clojure.spec/op clojure.spec-alpha2/or, 
;;    :keys [:k :i], 
;;    :specs [clojure.core/keyword? clojure.core/int?]}

We’re continuing to push on the relationship between the symbolic form, symbolic map, and even the conform output and consider more automated conversions between the form and map versions, ways to potentially work/transform specs in the map form, support for datafy, etc. More to come.

tools.deps

I spent some time this week on a few bugs and some perf issues in tools.deps based on some reports. Exclusion symbols were never canonicalized (so compojure wouldn’t exclude compojure/compojure ). That should be fixed in next release. I happened to notice that pom deps were including all transitive deps regardless of scope, instead of narrowing to just compile and runtime deps so I fixed that. And there have been some complaints that tools.deps includes slf4j-nop as a dep. slf4j is a pluggable logging framework and the nop logger is a “don’t log” plugin, which takes away the choice for library consumers to plug as they wish. tools.deps will remove that dependency but it will still be included in the clj installation.

Finally, there were some perf-related issues where it seemed like the same resource was being requested many times. I did some poking at this and we were a) not installing the Maven session repository cache and b) not reusing the Maven session throughout the classpath generation process. In many cases, this doesn’t make that much of a difference, but it is particularly pronounced if you have a large number of deps and if you include slow repositories (like an s3 repo). Anyhow, there is now a session API that will be available for tooling consumers and it’s enabled for use with clj. The next version of clj should have, in some cases, faster classpath generation (but won’t affect anything if cp is already cached). I do not think this is the end of the story - there are still a few additional things that caught my eye but this seemed like a big improvement in the meantime.

I am still testing some of these changes, so new clj is not yet available. tools.deps has been released though - if you’re a tool maker, feel free to follow up on clojurians slack.

Ask Clojure

Lots of questions coming in on the Ask Clojure forum this week! If you haven’t yet, check it out.

Also, I made a Twitter account, @AskClojure that is tweeting any new questions if you want to watch that. There are many RSS feeds available on the Ask Clojure site - if you need a link to something (like recent questions, recent activity, etc), let me know.