Tooling for Script Authors

This guide covers the tools for writing, checking and debugging Schooner scripts:

• Source locations in errors
• Checking scripts before they run
• Tracing and assertions
• Backtraces
• The REPL
• Inspecting macro expansion

The running example

The examples below share one embedder-defined environment. Keeping it in a named, zero-arity function lets the mix tasks and the REPL use the same sandbox your application uses:

defmodule MyApp.Scripts do
  alias Schooner.Host

  def environment do
    Schooner.Environment.new(
      standard_libraries: [:base, :char, :write],
      pre_imports: [["scheme", "base"]],
      libraries: [
        Host.library(
          name: ["myapp", "catalog"],
          primitives: [
            {"unit-price", 1, &unit_price/1}
          ]
        )
      ]
    )
  end

  defp unit_price([sku]), do: MyApp.Catalog.price(Host.to_string!(sku, op: "unit-price"))
end

and one script, scripts/pricing.scm:

(import (myapp catalog))

(define (line-total sku qty)
  (* (unit-price sku) qty))

(define (order-total lines)
  (if (null? lines)
      0
      (+ (line-total (car (car lines)) (cdr (car lines)))
         (order-total (cdr lines)))))

Telling the tools which environment to use

Every mix task in this guide resolves its environment the same way, first match wins:

1. --env MyApp.Scripts.environment on the command line: a zero-arity function returning a %Schooner.Environment{}.

2. Application config:

   # config/dev.exs
   config :schooner, :tooling_environment, {MyApp.Scripts, :environment, []}

3. Otherwise, every shipped standard library, all imported: the surface Schooner.run/1 gives a script that imports nothing. This is fine for experiments, but it is wider than any real sandbox, so prefer 1 or 2.

Source locations in errors

Every script-level exception carries a :location (a %Schooner.Location{file, line, column}, or nil), and when the location names a file its message is prefixed with file:line:col. Pass :file to turn locations on and name the script, or locations: true to turn them on without a name. Tracking positions makes reading and expanding a script about 15% slower but costs nothing while it runs; without either option errors have location: nil.

With locations on, reader errors, malformed special forms and other syntax errors, imports of missing libraries, and unbound variables are located. Errors raised while applying a procedure, such as a primitive's type error, an arity mismatch or an uncaught (error ...), are located only when you pass debug: true, which also turns locations on. Debug mode wraps each primitive call in a try and records every call for a backtrace, which together make call-heavy scripts two to three times slower. Without it those errors have location: nil, so turn it on while you develop and test scripts, and wherever you want precise errors more than that speed:

source = File.read!("scripts/pricing.scm")

{:error, error} =
  Schooner.eval(source <> ~s|(order-total '(("widget" . "3")))|,
                MyApp.Scripts.environment(),
                file: "scripts/pricing.scm", debug: true)

error.location
# => %Schooner.Location{file: "scripts/pricing.scm", line: 4, column: 3}

Schooner.format_error/2 renders the error for people. With source: it adds an excerpt, and with debug: true the error also carries the backtrace printed after it:

IO.puts(Schooner.format_error(error, source: source))

scripts/pricing.scm:4:3: type error in `*`: expected number, got "3"
  |
4 |   (* (unit-price sku) qty))
  |   ^

Scheme backtrace (most recent first):
  *            scripts/pricing.scm:4:3 (tail call)
  line-total   scripts/pricing.scm:9:10
  order-total  scripts/pricing.scm:11:1

Errors raised while expanding a macro point at the macro's use site in your script, not into the library that defines the macro. An error in a procedure defined by a library loaded with Schooner.Library.Loader.load_file/3 points into that library's file; pass debug: true to load_file/3 to locate errors raised while applying procedures in the library's code. Scripts run through Schooner.compile/3 and Schooner.run_compiled/3 keep their locations; pass file: and debug: to compile, or debug: true to run_compiled/3 for one run.

Checking scripts before they run

Schooner.check/3 reads a script, resolves its imports and expands its macros against an environment without evaluating it, and returns a list of Schooner.Diagnostics. An empty list means nothing was found.

Schooner.check(~s|(import (myapp catalog)) (unit-prise "widget")|,
               MyApp.Scripts.environment(),
               file: "draft.scm")
# => [
#      %Schooner.Diagnostic{
#        severity: :error,
#        code: :unbound,
#        message: "unbound variable: unit-prise",
#        location: %Schooner.Location{file: "draft.scm", line: 1, column: 27}
#      }
#    ]

A procedure's arity is known when it is a primitive or procedure bound by the environment or an import, or when the script defines it once with lambda or (define (name ...) ...) and nothing else binds it.

Every form is checked, including branches that never run, so an unbound name in dead code is still reported. When an import fails, the checker can't know which names it would have bound, so it reports no :unbound diagnostics for that script.

Because check/3 never runs the script and leaves the environment unchanged, it is safe to call on untrusted input when a script is saved. Use it to reject broken scripts early and to show diagnostics in your own UI. It does expand the script's macros, and a macro that never stops expanding never returns, so call it under a timeout as you would Schooner.eval/3.

From the command line, or in CI: suppose scripts/ holds pricing.scm, draft.scm with the script above, and discounts.scm, which starts with (import (myapp promos)). Then:

$ mix schooner.check "scripts/**/*.scm" --env MyApp.Scripts.environment
scripts/discounts.scm:1:9: error[unknown_library]: library not found: (myapp promos)
scripts/draft.scm:1:27: error[unbound]: unbound variable: unit-prise
2 errors in 2 files

Arguments can be files, directories or globs; quote globs so the task expands them rather than the shell. The task exits with status 1 when it finds errors. --warnings-as-errors makes warnings fail too, although every diagnostic the checker reports today is an error. --format json prints machine-readable output.

Tracing and assertions

In Schooner, display and write return their rendered text rather than writing it anywhere (see Deviations), so they can't be used for print debugging. Instead, the opt-in (schooner debug) library sends output to a sink the embedder chooses. Like (scheme time), it is not in the default registry. A script can only import it when you pass the library in:

env =
  Schooner.Environment.new(
    pre_imports: [["scheme", "base"]],
    libraries: [Schooner.Debug.library(sink: {:logger, :debug})]
  )

Take scripts/order.scm:

(import (schooner debug))

(define (line-total price qty)
  (trace "line-total" (* price qty)))

(define (order-total lines)
  (print "pricing" (length lines) "lines")
  (let ((total (apply + (map (lambda (line) (line-total (car line) (cdr line)))
                             lines))))
    (assert (> total 0))
    total))

(order-total '((10 . 3))) logs pricing 1 lines and line-total: 30, and returns 30. (order-total '((10 . 0))) fails the assertion.

• (trace label expr) returns expr's value unchanged, so you can wrap any expression in place without restructuring code. It sends label: value, with the label rendered by display and the value by write. Multiple values pass through, and are all written. trace has to see the value, so expr is not in tail position, but tail calls inside expr still are: wrapping a call to a long-running loop is fine.
• (print obj ...) sends the display rendering of its arguments, separated by spaces.
• (assert expr) / (assert expr message) raise a normal Scheme error, catchable with guard, when expr is #f, and otherwise return its value. The error's message is assertion failed: followed by the source text of expr, and then by message, which is only evaluated when the assertion fails.

All three are syntax, so trace, print and assert can't be passed as values.

Sinks:

kind is :trace or :print. The location is the %Schooner.Location{} of the trace or print form, recorded on the same terms as error locations: pass :file, locations: true or debug: true, or it is nil. With the logger sink it is in the :schooner_location metadata, and kind is in :schooner_debug. A sink runs in the evaluating process; if it raises, the script stops with a Schooner.Primitive.Error that a guard can't catch.

The pid sink is convenient in tests:

env =
  Schooner.Environment.new(
    pre_imports: [["scheme", "base"]],
    libraries: [Schooner.Debug.library(sink: self())]
  )

{:ok, 30} =
  Schooner.eval(source <> "(order-total '((10 . 3)))", env, file: "scripts/order.scm")

assert_receive {:schooner_debug, :print, "pricing 1 lines", _location}
assert_receive {:schooner_debug, :trace, "line-total: 30", %Schooner.Location{line: 4}}

A failed assertion is located too:

{:error, error} =
  Schooner.eval(source <> "(order-total '((10 . 0)))", env, file: "scripts/order.scm")

error.message
# => "scripts/order.scm:10:5: uncaught Scheme error: assertion failed: (> total 0)"

Backtraces

Schooner implements proper tail calls on top of the BEAM's last-call optimisation, so there is no Scheme call stack to look at after an error, and the Elixir stacktrace shows only evaluator internals. With debug: true, the evaluator keeps a bounded history of recent procedure calls and attaches it to any error that escapes. Taking the error from Source locations in errors:

{:error, error} =
  Schooner.eval(source <> ~s|(order-total '(("widget" . "3")))|,
                MyApp.Scripts.environment(),
                file: "scripts/pricing.scm", debug: true)

IO.puts(Schooner.format_error(error))

scripts/pricing.scm:4:3: type error in `*`: expected number, got "3"

Scheme backtrace (most recent first):
  *            scripts/pricing.scm:4:3 (tail call)
  line-total   scripts/pricing.scm:9:10
  order-total  scripts/pricing.scm:11:1

Read it from the bottom up: the top-level call to order-total on line 11 called line-total on line 9, which called * from tail position and failed. The history follows calls as they return, so unit-price, which line-total called and which returned, isn't listed. A call marked (tail call) replaced the procedure that made it, so the frame below it is the call it replaced rather than a caller waiting for its result. A tail-recursive loop shows up as a run of tail calls. Given scripts/sum.scm:

(define (sum-firsts lists acc)
  (if (null? lists)
      acc
      (sum-firsts (cdr lists) (+ acc (car (car lists))))))

(sum-firsts '((1) (2) (3) 4) 0)

the fourth iteration fails:

scripts/sum.scm:4:38: type error in `car`: expected pair, got 4

Scheme backtrace (most recent first):
  car         scripts/sum.scm:4:38
  sum-firsts  scripts/sum.scm:4:7 (tail call)
  sum-firsts  scripts/sum.scm:4:7 (tail call)
  sum-firsts  scripts/sum.scm:4:7 (tail call)
  sum-firsts  scripts/sum.scm:6:1

The history is a ring buffer of the last 32 calls; pass :backtrace_depth to keep more or fewer. A loop overwrites its oldest entries rather than growing the buffer, so a script that tail-recurses a million times runs in constant memory with debug: true too.

The raw frames are in error.scheme_backtrace, most recent first, as %Schooner.Frame{name, location, tail?}. name is the name the procedure was defined with, a primitive's name, "<lambda>" for an anonymous procedure, or "<parameter>" for a parameter object. With debug: false (the default), no history is kept and scheme_backtrace is nil. Debug mode is chosen when the program is compiled to closures, so without it they contain no history code at all. A compiled program can run with Schooner.run_compiled(compiled, environment, debug: true), and errors are located if it was compiled with file: or locations: true.

Some calls aren't recorded:

• Arithmetic and comparison on two integers (+, -, *, =, <, >, <=, >=) runs inline and can't fail, so it isn't recorded. With any other arguments these calls are recorded like any other.
• A procedure that a primitive calls, such as the procedure given to map, for-each, apply, call/cc, dynamic-wind or with-exception-handler, has no frame of its own: the primitive's frame stands in for it. The calls it makes are recorded. Tail calls it makes stay in the history until the primitive returns, so a procedure that map calls several times may leave a frame from each call.

Procedures that the evaluator calls directly rather than through a closure, such as a named let loop, are recorded like any other call, under their name.

Escapes keep the history consistent. When guard catches a raise, or a continuation from call/cc is invoked, the calls between the escape and its target are dropped, so a later error doesn't list them as callers. When no guard clause matches, the raise continues with the history it had when it was raised.

Recording every call is most of what makes debug mode two to three times slower on call-heavy scripts. Leave debug off where scripts must run at full speed.

The REPL

mix schooner.repl starts an interactive session against your real environment. Something that works in the REPL works in production, and something that's unbound in production is unbound here too.

$ mix schooner.repl --env MyApp.Scripts.environment --load scripts/pricing.scm
Schooner 1.1.0 — environment: MyApp.Scripts.environment/0. ,help for commands.
schooner> (line-total "widget" 3)
30
schooner> (define (with-shipping x)
     ...>   (+ x 5))
schooner> (with-shipping 30)
35
schooner> (char-upcase #\a)
error: unbound variable: char-upcase
schooner> (import (scheme char))
schooner> (char-upcase #\a)
#\A
schooner> ,env unit
unit-price  (myapp catalog)  procedure, 1 arg
schooner> ,expand (when ok (go))
(if ok (begin (go)))
schooner> ,time (order-total '(("widget" . 2) ("gadget" . 1)))
30  ; 0.1ms, 1650 reductions
schooner> ,quit

char-upcase is in (scheme char). The environment offers that library but doesn't import it, so a script must import it, and so must you.

Definitions, imports and define-syntax macros persist from one entry to the next. An entry that ends inside an open form continues on the next line. In a terminal, each new line starts indented to where the code goes, as Schooner.Pretty lays it out: the body of a define, lambda or let two columns in, the arguments of a call under the first, and the bindings of a let under each other. So you type the code, not the spaces. Tab re-indents a line, Up and Down recall earlier entries, and pasted code keeps its own indentation.

An error is printed and the session carries on. Ctrl-C interrupts an evaluation that runs too long, leaving the session as it was before that entry, and Ctrl-D or ,quit leaves. --debug turns on backtraces for errors.

When standard input isn't a terminal, the REPL reads whole lines without editing or indenting them, so you can pipe a script into it. Ctrl-C then reaches the BEAM, as it does in any mix task.

To build your own console (a web console, an admin-panel console), use the same session API the REPL is built on:

session = Schooner.Session.new(MyApp.Scripts.environment())
{:ok, _, session} = Schooner.Session.eval(session, "(define x 41)")
{:ok, 42, _session} = Schooner.Session.eval(session, "(+ x 1)")

Schooner.Session.bindings/1 lists what's in scope, as ,env does, and Schooner.Session.environment/1 passes the session's macros to Schooner.expand/3.

Inspecting macro expansion

Many of Schooner's standard forms (cond, case, the let family, do, and, or, when, unless) are syntax-rules macros. When they, or your own macros, surprise you, look at the expansion:

{:ok, [form]} = Schooner.expand("(when (> n 0) (go n))", MyApp.Scripts.environment())
IO.puts(Schooner.Pretty.format(form))
# (if (> n 0) (begin (go n)))

Options:

• step: :once expands each macro use that isn't inside another macro use once, and leaves the macro uses in its output unexpanded.
• trace: true also returns each expansion step, in the order it happened, as %{macro:, location:, before:, after:}.

mix schooner.expand takes a file, or the source with -e, and the same --env as mix schooner.check. --trace lists the steps before the result:

$ mix schooner.expand -e "(let loop ((i 0)) (when (< i 3) (loop (+ i 1))))" --trace
[1] let at 1:1
    (let loop ((i 0)) (when (< i 3) (loop (+ i 1))))
 => (letrec*·1 ((loop (lambda (i) (when (< i 3) (loop (+ i 1)))))) (loop 0))

[2] when at 1:19
    (when (< i 3) (loop (+ i 1)))
 => (if (< i 3) (begin (loop (+ i 1))))

(letrec* ((loop (lambda (i) (if (< i 3) (begin (loop (+ i 1))))))) (loop 0))

Identifiers a macro introduces are renamed so they can't clash with yours, and are printed with a number (letrec*·1 above), so you can tell a macro's own tmp from yours. Pass names: :plain to Schooner.Pretty.format/2, or --plain to the task, to print plain names. Like check/3, expand/3 never evaluates the program.