OCaml PreProcessor eXtension framework
PPX is a preprocessor:
source code (*.ml, *.mli)
↓ Preprocessor (PPX, CamlP4, m4)
source code (*.ml, *.mli, or AST binary)
↓ OCaml compiler
* Parsing
* Typing
* Compilation
object file, executable (*.cm*, *.exe)
Not text to text pp like m4
and cpp
#ifdef
, #include
123456z
for Z.of_string "123456"
let
regex "(?P<name>regex)" : <name : string> regex
type t = Foo [@@deriving show]
PPX tends to be context-free.
(You can type-check the input in PPX, if you dare…)
Attributions and extension points
[@...]
OCaml syntax has a way to postfix annotations:
42 [@the_answer] (* for expression *)
let x = 42 [@@the_answer] (* for toplevel *)
[@@@the_module] (* the entire file *)
Removing attributes from a parsable OCaml code is still parsable.
ocamlc
ignores most of the attributes,
except [@warning]
[@tailcall]
[@@inline]
, etc.
Attributes can give hints to PPX.
Postfix looks ugly sometimes. There are some infix ways:
Infix | Postfix equivalent |
---|---|
let [@a] x = .. in .. |
let x = .. [@@a] in .. |
match [@a] x with .. |
(match x with ..) [@a] |
if [@a] e then .. |
(if e then ..) [@a] |
struct [@a] .. end |
(struct .. end)[@a] |
module [@a] X = .. |
module X = .. [@@a] |
.. | .. |
[%...]
Embed something out of OCaml:
[%the_answer] + 42 (* to be an expression *)
module X = struct
[%%self_destruct] (* to be a toplevel *)
end
PPX must replace the extension points with normal code,
otherwise they are rejected at OCaml’s typing:
Each attribute and extension point can have a payload.
It is an OCaml expression, structure, type, signature, or pattern:
[@foo 42] (* expression and structure *)
[%bar fun x -> M.y ()]
[@pee let x = 1;; let y = 2]
[@foo: int -> int] (* type and signature *)
[%bar: [`Foo]] (* prefixed with : *)
[@@baz: val x : int]
[@foo? M.Some _] (* pattern *)
[%bar? 'a'..'z'] (* prefixed with ? *)
[@@@boo? 1 | 2 when true]
They must be parsable but types are not checked.
Payload gives additional information to PPX.
How to handle AST
PPX must implement 2 functions:
*.ml
structure -> structure
*.mli
signature -> signature
The types are defined in OCaml compiler source code.
The main module for the parsed AST:
$OPAM_SWITCH_PREFIX/lib/ocaml/compiler-libs/parsetree.mli
or $OPAM_SWITCH_PREFIX/.opam-switch/build/ocaml-base-compiler.4.xx.y/parsing/parsetree.mli
Data types:
structure
: module implementationsignature
: signature declarationexpression
pattern
Available in compiler-libs.common library\(^†\):
Location
: source code location and error reportingAsttypes
: constants and flags: Const_int n
, Recursive
, etcLongident
: Iden.Tifier.s
Pprintast
: printerCheck parsing/
of OCaml compiler source:
$OPAM_SWITCH_PREFIX/.opam-switch/build/ocaml-base-compiler.4.xx.y/parsing/
\(†\) But we do not directly use compiler-libs.common.
Also available in compiler-libs.common:
Ast_helper
: buildersAst_iterator
: iteratorAst_mapper
: mapper for recursive code transformationAttr_helper
: attribute handlingBuild ty -> ty
for various types of Parsetree.
Your own mapper inheriting default
:
let super = default
let rec structure_item self sitem =
match sitem.pstr_desc with
| Pstr_eval (e, atrs) -> .. expr self e ..
| _ -> super.structure_item self sitem (* recursive! *)
and expr self e = ..
let my_mapper = { super with structure; expr }
(ppxlib provides class based mapper.)
PPX must be compiled to an executable:
*.ml
or *.mli
as an inputPPX executable must be given to ocamlc
:
$ ocamlc -ppx ./ppx_my_own.exe code.ml
But do not do it by hand. Use ppxlib and Dune.
Writing and using PPX easier.
Now a defacto standard library for PPX.
Ast_traverse
: class based AST mapper/iterator/foldPPX which does nothing:
; dune
(library
(kind ppx_rewriter) ; kind: ppx_rewriter
(name ppx_my_own)
; (public_name ppx_my_own) ; will need ppx_my_own.opam
(libraries ppxlib)) ; library: ppxlib
$ dune build ppx_my_own.cma
(library
(name my_cool_library)
(public_name my_cool_library)
(preprocess (pps ppx_my_own ; <== add this
ppx_not_my_own)) ;
(libraries whatever))
PPX with ppxlib can be linked together with (pps ..)
AST mapper, iterator, and folder
open Ppxlib
class my_map = object
inherit Ast_traverse.map as super (* default *)
method! expression e = match e.desc with
| ... (* transform e *)
| _ -> super#expression e (* visit recursively *)
end
let () = Ppxlib.Driver.register_transformation
~impl: (new my_map)#structure
~intf: (fun i -> i)
"ppx_my_own"
Conversome to write AST for 42 + 1
:
ppxlib.metaquot: useful extensions to lift ASTs:
[%expr 42 + 1]
More constructs:
Available at https://github.com/camlspotter/ppx_my_own
Random tips for PPX
ppxlib exposes Parsetree related modules.
In most cases, you need not use compiler-libs directly.
The AST types of ppxlib are wrapped.
They are incompatible with those of compiler-libs.
Ppxlib_ast.Selected_ast.to_ocaml
Ppxlib_ast.Selected_ast.of_ocaml
-dsource
and -dparsetree
Good to see how things are parsed:
$ ocaml -dsource
# let [@a] x [@b] = 1 + 1 [@c] [@@d];;
let ((x)[@b ]) = ((1 + 1)[@c ])[@@a ][@@d ];;
val x : int = 2
$ ocaml -dparsetree
# 1;;
Ptop_def
[
structure_item (//toplevel//[1,0+0]..[1,0+1])
Pstr_eval
expression (//toplevel//[1,0+0]..[1,0+1])
Pexp_constant PConst_int (1,None)
]
- : int = 1
NO easy way for the moment!
Write a test:
Build a preprocessed file *.pp.ml*
and find it:
$ dune build --verbose ./test.pp.ml
...
- _build/default/ppx_simple/tests/test.pp.ml
Print the pp’ed file using ocamlc -dsource
:
$ ocamlc -dsource ../../_build/default/ppx_simple/tests/test.pp.ml
Use Pprintast.*
for -dsource
output:
or Ocaml_common.Printast.*
for -dparsetree
output:
Type inspection, definition jump and more. Use it already!
let impl sitems =
List.map (fun { pstr_desc; _ } ->
match pstr_desc with
| _(*<-cursor*) -> assert false) sitems
Pattern destruction (C-c C-d
in Emacs):
let impl sitems =
List.map (fun { pstr_desc; _ } ->
match pstr_desc with
| Ppxlib.Pstr_eval (_, _)|Ppxlib.Pstr_value (_, _)|Ppxlib.Pstr_primitive _
|Ppxlib.Pstr_type (_, _)|Ppxlib.Pstr_typext _|Ppxlib.Pstr_exception _
|Ppxlib.Pstr_module _|Ppxlib.Pstr_recmodule _|Ppxlib.Pstr_modtype _
|Ppxlib.Pstr_open _|Ppxlib.Pstr_class _|Ppxlib.Pstr_class_type _
|Ppxlib.Pstr_include _|Ppxlib.Pstr_attribute _|Ppxlib.Pstr_extension
(_, _) -> assert false) sitems