Type-safe blogs and books with literate-ts

Years ago when Brett Slatkin wrote Effective Python, he blogged about creating a tool call pyliterate to run all the code samples in his book and verify that their output matched what he'd written in the text. The idea stuck with me and when I started writing Effective TypeScript in early 2019, I thought I'd do something similar. If nothing else, static analysis of a book seems very much in the spirit of TypeScript. Creating literate-ts wound up being a lot of work, but in the end I think it more justified itself, though not quite for the reasons I expected!

TypeScript presents a few challenges for verification. You can compile and run a TypeScript program through Node to check its output, sure, but then you're not testing anything relating to the errors or inferred types. Usually you see these by mousing over a symbol or an error in your editor. But what's "mousing over" in a printed book? Still, inferred types and errors are both essential parts of the TypeScript experience and I wanted to both show them in Effective TypeScript and check them with a verifier.

As I wrote, I eventually settled on a system. To show an error in a code sample, I'd put tildes and the error message in a comment under the line on which the error occurred. This is meant to evoke the squiggly red lines under an error in your editor:

let str = 'not a number';
let num: number = str;
//  ~~~ Type 'string' is not assignable to type 'number'.

In addition to conveying the error to the reader, there are a few things for that can be verified here:

When you run it through tsc, does the code sample produce that one error and no others?
Do the tildes line up with the true error?
Do the error messages match?

For inferred types, I used a slightly different comment syntax, always starting with "type is":

let x = 12;  // type is number
'four score'.split(' ');  // type is string[]

This indicates that if you mouse over x you'll see that the inferred type is number. For the second line, you need to assign the expression to a variable or mouse over split to see the string[] type. It's usually quite intuitive which symbol the "type is" refers to, but getting this right in the verifier took a bit of care. The main thing to verify here, of course, is the type of the symbol or expression. The string representations of the types are matched, character-for-character, ala dtslint.

If the types don't line up, or the errors aren't quite right, literate-ts will complain:

$ cat sample.asciidoc
TypeScript will infer types in the absence of annotations:

[source,ts]
----
const n = 10;  // type is number
----

$ literate-ts --alsologtostderr sample.asciidoc
Verifying with TypeScript 4.0.0-dev.20200629

Code passed type checker.
sample-4: Failed type assertion for const n = 10; (tested n)
  Expected: number
    Actual: 10
  0/1 type assertions matched.

(The issue is here is that the inferred type is the numeric literal type 10 rather than number. Either the comment should be changed or the variable should be declared with let.)

Overall I'm pretty happy with how things turned out! You can check out the results over at the literate-ts repo. All in all, there are about 600 code samples in Effective TypeScript that all get checked. Running these checks takes 5–10 minutes on my Macbook.

So what sorts of errors did literate-ts turn up? Initially… not many! I'd been pretty careful about running code samples through VS Code or the TypeScript playground before committing them. Here was the first mistake that it caught (at the end of Item 7, "Think of Types as Sets of values"):

Finally, it's worth noting that not all sets of values correspond to TypeScript types. There is no TypeScript type for all the integers, or for all the objects which have x and y properties but no others. You can sometimes subtract types using Exclude, but only when it would result in a proper TypeScript type:

type T = Exclude<string|Date, string|number>;  // type is number
type NonZeroNums = Exclude<number, 0>;  // type is still just number

It should be "type is Date", not "type is number". Not a major mistake (the argument is still valid) but a nice validation of literate-ts nonetheless. There were a handful of small mistakes like this.

Where literate-ts really shone was when I started ~~refactoring~~ editing the text. It's exactly the same as in coding. Writing it right the first time isn't nearly so hard as changing things and keeping them right. As I incorporated feedback from my editors and reworked Items, I constantly found myself introducing real mistakes that were caught by the verifier. Score one for literate-ts!

This was doubly true when the O'Reilly editors started editing the text directly in the run-up to publication. Inevitably some mistakes were made, e.g. dropping quotes or comment markers. literate-ts caught all of these. (Since the book is stored in a git repo, git log and git diff were also extremely useful tools for catching these sorts of problems.)

But the biggest benefit came when new TypeScript versions were released. In the week before Effective TypeScript came out, the beta of TypeScript 3.7 was announced. As Anders Hejlsberg has said, semantic versioning isn't very meaningful when it comes to tsc: the whole point of new versions of TypeScript is to break your code! (Or rather to reveal ways in which it was already broken.) So as a book author, this was a bit terrifying. But no big deal, I just updated literate-ts and ran all the code samples.

The biggest issue I found was that I had a code sample showing that interfaces could be recursive but type aliases couldn't. And that was true… until TypeScript 3.7!. So I knew exactly what I had to fix. But just as importantly for my peace of mind, I knew that the rest of the book was OK. As new versions of TypeScript have come out since the book's publication in October of 2019, I've continued to re-run the verifier and make minor revisions as needed.

So what sorts of mistakes didn't literate-ts catch? It's the classic case of tests vs. types. Here's a particularly egregious example:

type SortedList<T> = T[] & {_brand: 'sorted'};

function isSorted<T>(xs: T[]): xs is SortedList<T> {
  for (let i = 1; i < xs.length; i++) {
    if (xs[i] < xs[i - 1]) {
      return false;
    }
  }
  return true;
}

function binarySearch<T>(xs: SortedList<T>, x: T): boolean {
  // ...
}

This comes from Item 37, which discusses "brands", a way of simulating nominal typing in TypeScript to create types for things like "non-empty sets" or "sorted sets" that can't be represented directly.

The implementation of binarySearch is listed earlier in the text and assumes a list that's sorted in ascending order. The isSorted type guard checks that the list is sorted… but in descending order. Oops! This all type checks great, but it produces incorrect results. A type checker can't tell you that < should be > (Haskell people, please tell me if I'm wrong!). You really need a unit test.

At some point I realized that you could create a unit test with literate-ts! The trick is to write some code that calls the function and console.logs the results. Then literate-ts can check the expected output against the actual output. This can all go in a comment so that the reader doesn't see it.

Basically all of the errata have been variations on this: things that weren't checked by the verifier. I'm not aware of a single mistake having to do with the type system itself.

I've recently published literate-ts to npm and added support for Markdown sources (O'Reilly books are written in Asciidoc). And I've started applying it to this blog. If you're writing a TypeScript book or blog yourself, head over to the literate-ts repo, give it a try and let me know how it goes!

If you prefer videos, literate-ts comes up in my 2019 tsconf talk on Testing Types: