{0 How to drive [odoc]}

This 'live' document describes how to use [odoc] to produce the documentation of [odoc] itself. The aim is
to show a short, simple example of how [odoc] can be used, covering most of the important features.
The document built here includes not only the documentation of [odoc] itself, but it also builds the
docs for a subset of [odoc]'s dependent libraries to show how this may be done. For a much more
complete and comprehensive use of [odoc], see the {{: https://github.com/ocaml-doc/voodoo} Voodoo project}, the tool that is being used to build
the package docs for
{{: https://v3.ocaml.org/} v3.ocaml.org}.

First we need to initialise MDX with some libraries and helpful values.

{[
(* Prelude *)
#require "bos";;
#install_printer Fpath.pp;;
#print_length 100;;
#print_depth 10;;
open Bos;;
let (>>=) = Result.bind;;
let (>>|=) m f = m >>= fun x -> Ok (f x);;
let get_ok = function | Ok x -> x | Error (`Msg m) -> failwith m
]}

{1 Desired Output}

[odoc] produces output files (html or others) in a structured directory tree, so before running [odoc], the structure of the output must be decided. For these docs, we want the following structure:
{ul {- [odoc/index.html] : main page
}{- [odoc/\{odoc_for_authors.html,...\}] : other documentation pages
}{- [odoc/odoc_model/index.html] : [odoc] model library subpage
}{- [odoc/odoc_model/Odoc_model/index.html] : Module page for the module [Odoc_model]
}{- [odoc/odoc_model/Odoc_model/...] : Further pages for the submodules of [Odoc_model]
}{- [odoc/odoc_.../index.html] : other [odoc] library pages
}{- [odoc/deps/stdlib/index.html] : stdlib main page
}{- [odoc/deps/stdlib/Stdlib/index.html] : Module page for the module [Stdlib]
}{- [odoc/deps/astring/index.html] : astring main page
}{- [odoc/deps/...] : other dependencies
}}

The [odoc] model for achieving this is that we have {e pages} ([.mld] files) that have {e children} which are either {e further pages} ([.mld] files) or {e modules} (from [.cmti] files). This {{!page-parent_child_spec} parent/child relationship} is specified on the command line. Parent pages must be {e compiled} by [odoc] before their children. Then compiling a page [mypage.mld] will produce the file [page-mypage.odoc].

In the example below, there will be a file [odoc.mld] that corresponds with the top-level directory [odoc/]. It will be compiled as follows:

{[
odoc compile odoc.mld --child page-odoc_model --child deps ...
]}

The file [deps.mld] which corresponds with the sub-directory [odoc/deps/], will be compiled as follows:

{[
odoc compile deps.mld -I . --parent `odoc` --child page-stdlib --child page-astring ...
]}

The file [odoc_model.mld] will have a child module [Odoc_model]. It will be compiled as follows:

{[
odoc compile odoc_model.mld -I . --parent `odoc` --child module-Odoc_model
]}

When compiling any [.mld] file, the parent and all children must be specified. Parents can only be pages from other [.mld] files, and children may be pages (from [.mld] files) or modules (from [.cmti]/[.cmt] or [.cmi] files).

The parent page must exist before the child page is created, and it must have had the child specified when it was initially compiled.

{1 Document Generation Phases}

Using [odoc] is a three-phase process:
{ol {- Compilation: [odoc compile]

This takes the output from the compiler in the form of [.cmti], [.cmt], or [.cmi] files (in order of preference), translates it into [odoc]'s internal format, and performs some initial expansion and resolution operations. For a given input [/path/to/file.cmti] it will output the file [/path/to/file.odoc] unless the [-o] option is used to override the output file. If there were [.cmi] dependencies required for OCaml to compile these files, then there will be equivalent [.odoc] dependencies needed for the [odoc compile] step. [odoc] will search for these dependencies in the paths specified with the [-I] directive on compilation. [odoc] provides a command to help with this: [odoc compile-deps]:

As an example we can run [odoc compile-deps] on the file [../src/xref2/.odoc_xref2.objs/byte/odoc_xref2__Compile.cmti]:

{[
$ `odoc` compile-deps ../src/xref2/.odoc_xref2.objs/byte/odoc_xref2__Compile.cmti | tail -n 5
Stdlib__result 2ba42445465981713146b97d5e185dd5
Stdlib__seq d6a8de25c9eecf5ae9420a9f3f8b2e88
Stdlib__set 5d365647a10f75c22f2b045a867b4d3e
Stdlib__uchar ab6f1df93abf9e800a3e0d1543523c96
Odoc_xref2__Compile e0d620d652a724705f7ed620dfe07be0
]}

so we can see we will need to run [odoc compile] against several [Stdlib] modules before we can compile [odoc_xref2__Compile.cmti]
}
{- Linking: [odoc link]


This takes the [odoc] files produced during the compilation step and performs the final steps of expansion and resolution. It is during this phase that all the references in the documentation comments are resolved. In order for these to be resolved, everything that is referenced must have been compiled already, and their [odoc] files must be on the
include path as specified by the [-I] arguments to [odoc link]. In this example, we achieve that by compiling all modules and [.mld] files before linking anything. The output of the
link step is an [odocl] file, which is in the same path as the original [odoc] file by default.

Please note: it's only necessary to link the non-hidden modules (i.e., without a double underscore).
}
{- Generation: [odoc html-generate]


Once the compile and link phases are complete, the resulting [odocl] files may be rendered in a variety of formats. In this example we output HTML.
}}

{1 [odoc] Documentation}

In this section [odoc] is used to generate the documentation of [odoc] and some of its dependent packages. We can make a few simplifying assumptions here:
{ol {- Since we're working with one leaf package, we can assume that there can be no module name clashes in the dependencies. As such, we can afford to put all of our [.odoc] files into one directory and then hard-code the include path to be this directory. When using [odoc] in a context where there may be module name clashes, it requires more careful partitioning of output directories.
}{- We'll do all of the compiling before any linking.
}}

Let's start with some functions to execute the three phases of [odoc].

Compiling a file with [odoc] requires a few arguments: the file to compile, an
optional parent, a list of include paths, a list of children for [.mld] files,
and an output path. Include paths can be just ['.'], and we can calculate the
output file from the input because all of the files are going into the same directory.

Linking a file with [odoc] requires the input file and a list of include paths. As
for compile, we will hard-code the include path.

Generating the HTML requires the input [odocl] file and an output path. We will hard-code the output path to be [html].

In all of these, we'll capture [stdout] and [stderr] so we can check it later.

{[
let odoc = Cmd.v "../src/odoc/bin/main.exe"

let compile_output = ref [ "" ]

let link_output = ref [ "" ]

let generate_output = ref [ "" ]

let add_prefixed_output cmd list prefix lines =
  if List.length lines > 0 then
    list :=
      !list
      @ Bos.Cmd.to_string cmd :: List.map (fun l -> prefix ^ ": " ^ l) lines

let compile file ?parent ?(ignore_output = false) children =
  let output_file =
    let ext = Fpath.get_ext file in
    let basename = Fpath.basename (Fpath.rem_ext file) in
    match ext with
    | ".mld" -> "page-" ^ basename ^ ".odoc"
    | ".cmt" | ".cmti" | ".cmi" -> basename ^ ".odoc"
    | _ -> failwith ("bad extension: " ^ ext)
  in
  let open Cmd in
  let cmd =
    odoc % "compile" % Fpath.to_string file % "-I" % "." % "-o" % output_file
    |> List.fold_right (fun child cmd -> cmd % "--child" % child) children
  in
  let cmd =
    match parent with
    | Some p -> cmd % "--parent" % ("page-\"" ^ p ^ "\"")
    | None -> cmd
  in
  let lines = OS.Cmd.(run_out ~err:err_run_out cmd |> to_lines) |> get_ok in
  if not ignore_output then
    add_prefixed_output cmd compile_output (Fpath.to_string file) lines

let link ?(ignore_output = false) file =
  let open Cmd in
  let cmd = odoc % "link" % p file % "-I" % "." in
  let cmd = if Fpath.to_string file = "stdlib.odoc" then cmd % "--open=\"\"" else cmd in
  Format.printf "%a" pp cmd;let lines = OS.Cmd.(run_out ~err:err_run_out cmd |> to_lines) |> get_ok in
  if not ignore_output then
    add_prefixed_output cmd link_output (Fpath.to_string file) lines

let html_generate ?(ignore_output = false) file =
  let open Cmd in
  let cmd =
    odoc % "html-generate" % p file % "-o" % "html" % "--theme-uri" % "odoc"
    % "--support-uri" % "odoc"
  in
  let lines = OS.Cmd.(run_out cmd ~err:err_run_out |> to_lines) |> get_ok in
  if not ignore_output then
    add_prefixed_output cmd generate_output (Fpath.to_string file) lines

let support_files () =
  let open Cmd in
  let cmd = odoc % "support-files" % "-o" % "html/odoc" in
  OS.Cmd.(run_out cmd |> to_lines) |> get_ok
]}

We'll now make some library lists. We have not only external dependency libraries, but
[odoc] itself is also separated into libraries too. These two sets of libraries will be
documented in different sections, so we'll keep them in separate lists.
Additionally we'll also construct a list containing the extra documentation pages. Finally let's create a list mapping the section to its parent, which matches
the hierarchy declared above.

{[
let dep_libraries_core =
  [
    "odoc-parser";
    "astring";
    "cmdliner";
    "fpath";
    "result";
    "tyxml";
    "fmt";
    "stdlib";
    "yojson";
    "biniou";
  ]

let extra_deps =
  [
    "base";
    "core_kernel";
    "bin_prot";
    "sexplib";
    "sexplib0";
    "base_quickcheck";
    "ppx_sexp_conv";
    "ppx_hash";
  ]

let dep_libraries =
  match Sys.getenv_opt "ODOC_BENCHMARK" with
  | Some "true" -> dep_libraries_core @ extra_deps
  | _ -> dep_libraries_core

let odoc_libraries =
  [
    "odoc_xref_test";
    "odoc_xref2";
    "odoc_odoc";
    "odoc_model_desc";
    "odoc_model";
    "odoc_manpage";
    "odoc_loader";
    "odoc_latex";
    "odoc_html";
    "odoc_document";
    "odoc_examples";
  ]

let all_libraries = dep_libraries @ odoc_libraries

let extra_docs =
  [
    "interface";
    "contributing";
    "driver";
    "parent_child_spec";
    "features";
    "dune_wrapping";
    "interface";
    "odoc_for_authors";
    "dune";
    "ocamldoc_differences";
  ]

let parents =
  let add_parent p l = List.map (fun lib -> (lib, p)) l in
  add_parent "deps" dep_libraries @ add_parent "odoc" odoc_libraries
]}

[odoc] operates on the compiler outputs. We need to find them for both the files compiled by Dune within this project and those in libraries we compile against.
The following uses [ocamlfind] to locate the library paths for our dependencies:

{[
let ocamlfind = Cmd.v "ocamlfind"

let lib_path lib =
  let cmd = Cmd.(ocamlfind % "query" % lib) in
  OS.Cmd.(run_out cmd |> to_lines >>|= List.hd)

let lib_paths =
  List.fold_right
    (fun lib acc ->
      acc >>= fun acc ->
      lib_path lib >>|= fun l -> (lib, l) :: acc)
    dep_libraries (Ok [])
  |> get_ok
]}

We need a function to find [odoc] inputs given a search path. [odoc]
operates on [.cmti], [.cmt] or [.cmi] files, in order of preference, and the following
function finds all matching files given a search path. Then it returns an [Fpath.Set.t]
that contains the [Fpath.t] values representing the absolute file path, without its extension.

{[
let find_units p =
  OS.Dir.fold_contents ~dotfiles:true
    (fun p acc ->
      if List.exists (fun ext -> Fpath.has_ext ext p) [ "cmt"; "cmti"; "cmi" ]
      then p :: acc
      else acc)
    [] (Fpath.v p)
  >>|= fun paths ->
  let l = List.map Fpath.rem_ext paths in
  let l =
    List.filter
      (fun f ->
        not @@ Astring.String.is_infix ~affix:"ocamldoc" (Fpath.to_string f))
      l
  in
  List.fold_right Fpath.Set.add l Fpath.Set.empty]}

Since the units returned by this function have their extension stripped, we need
function to find the best file to use with this basename.

{[
let best_file base =
  List.map (fun ext -> Fpath.add_ext ext base) [ "cmti"; "cmt"; "cmi" ]
  |> List.find (fun f -> Bos.OS.File.exists f |> get_ok)
]}

Many of the units will be 'hidden' -- that is, their name will be mangled by Dune
in order to namespace them. This is achieved by prefixing the namespace module and
a double underscore, so we can tell by the existence of a double underscore that
a module is intended to be hidden. The following predicate tests for that condition:

{[
let is_hidden path = Astring.String.is_infix ~affix:"__" (Fpath.to_string path)
]}

To build the documentation, we start with these files. With the following function, we'll call [odoc compile-deps] on the file to
find all other compilation units upon which it depends:

{[
type compile_deps = { digest : Digest.t; deps : (string * Digest.t) list }

let compile_deps f =
  let cmd = Cmd.(odoc % "compile-deps" % Fpath.to_string f) in
  OS.Cmd.(run_out cmd |> to_lines)
  >>|= List.filter_map (Astring.String.cut ~sep:" ")
  >>= fun l ->
  let basename = Fpath.(basename (f |> rem_ext)) |> String.capitalize_ascii in
  match List.partition (fun (n, _) -> basename = n) l with
  | [ (_, digest) ], deps -> Ok { digest; deps }
  | _ -> Error (`Msg "odd")
]}

Let's now put together a list of all possible modules. We'll keep track of
which library they're in, and whether that library is a part of [odoc] or a dependency
library.

{[
let odoc_all_unit_paths = find_units ".." |> get_ok

let odoc_units =
  List.map
    (fun lib ->
      Fpath.Set.fold
        (fun p acc ->
          if Astring.String.is_infix ~affix:lib (Fpath.to_string p) then
            ("odoc", lib, p) :: acc
          else acc)
        odoc_all_unit_paths [])
    odoc_libraries
]}

{[
let all_units =
  let lib_units =
    List.map
      (fun (lib, p) ->
        Fpath.Set.fold
          (fun p acc -> ("deps", lib, p) :: acc)
          (find_units p |> get_ok)
          [])
      lib_paths in
  odoc_units @ lib_units |> List.flatten
]}

Now we'll compile all of the parent [.mld] files. To ensure that the parents are compiled before the children, we start with [odoc.mld], then [deps.mld], and so on. The result of this file is a list of the resulting [odoc] files.

{[
let compile_mlds () =
  let mkpage x = "page-\"" ^ x ^ "\"" in
  let mkmod x = "module-" ^ String.capitalize_ascii x in
  let mkmld x = Fpath.(add_ext "mld" (v x)) in
  ignore
    (compile (mkmld "odoc")
       ("page-deps" :: List.map mkpage (odoc_libraries @ extra_docs)));
  ignore (compile (mkmld "deps") ~parent:"odoc" (List.map mkpage dep_libraries));
  let extra_odocs =
    List.map
      (fun p ->
        ignore (compile (mkmld p) ~parent:"odoc" []);
        "page-" ^ p ^ ".odoc")
      extra_docs
  in
  let odocs =
    List.map
      (fun library ->
        let parent = List.assoc library parents in
        let children =
          List.filter_map
            (fun (parent, lib, child) ->
              if lib = library then Some (Fpath.basename child |> mkmod)
              else None)
            all_units
        in
        ignore (compile (mkmld library) ~parent children);
        "page-" ^ library ^ ".odoc")
      all_libraries
  in
  List.map
    (fun f -> (Fpath.v f, false))
    ("page-odoc.odoc" :: "page-deps.odoc" :: odocs @ extra_odocs)
]}

Now we get to the compilation phase. For each unit, we query its dependencies, then recursively call to compile these dependencies. Once this is done we compile the unit itself. If the unit has already been compiled we don't do anything. Note that we aren't checking the hashes of the dependencies which a build system should do to ensure that the module being compiled is the correct one. Again we benefit from the fact that we're creating the docs for one leaf package and that there must be no module name clashes in its dependencies. The result of this function is a list of the resulting [odoc] files.

{[
let compile_all () =
  let mld_odocs = compile_mlds () in
  let rec rec_compile parent lib file =
    let output = Fpath.(base (set_ext "odoc" file)) in
    if OS.File.exists output |> get_ok then []
    else
      let deps = compile_deps file |> get_ok in
      let files =
        List.fold_left
          (fun acc (dep_name, digest) ->
            match
              List.find_opt
                (fun (_, _, f) ->
                  Fpath.basename f |> String.capitalize_ascii = dep_name)
                all_units
            with
            | None -> acc
            | Some (parent, lib, dep_path) ->
                let file = best_file dep_path in
                rec_compile parent lib file @ acc)
          [] deps.deps
      in
      let ignore_output = parent = "deps" in
      ignore (compile file ~parent:lib ~ignore_output []);
      (output, ignore_output) :: files
  in
  List.fold_left
    (fun acc (parent, lib, dep) -> acc @ rec_compile parent lib (best_file dep))
    [] all_units
  @ mld_odocs
]}

Linking is now straightforward. We only need to link non-hidden [odoc] files, as any hidden are almost certainly aliased inside the non-hidden ones (a result of namespacing usually, and these aliases will be expanded).

{[
let link_all odoc_files =
  let not_hidden (f, _) = not (is_hidden f) in
  List.map
    (fun (odoc_file, ignore_output) ->
      ignore (link ~ignore_output odoc_file);
      Fpath.set_ext "odocl" odoc_file)
    (List.filter not_hidden odoc_files)
]}

Now we simply run [odoc html-generate] over all of the resulting [odocl] files.

{[
let generate_all odocl_files =
  List.iter (fun f -> ignore(html_generate f)) odocl_files;
  support_files ()
]}

The following code actually executes all of the above, and we're done!

{[
let compiled = compile_all () in
let linked = link_all compiled in
generate_all linked
]}

Let's see if there was any output from the [odoc] invocations:

{[
# #print_length 655360;;
# !compile_output;;
- : string list = [""]
# !link_output;;
- : string list = [""]
# !generate_output;;
- : string list =
["";
 "'../src/odoc/bin/main.exe' 'html-generate' 'odoc_xref_test.odocl' '-o' 'html' '--theme-uri' 'odoc' '--support-uri' 'odoc'";
 "odoc_xref_test.odocl: Warning, resolved hidden path: Odoc_model__Lang.Signature.t";
 "'../src/odoc/bin/main.exe' 'html-generate' 'odoc_examples.odocl' '-o' 'html' '--theme-uri' 'odoc' '--support-uri' 'odoc'";
 "odoc_examples.odocl: Warning, resolved hidden path: Odoc_examples__Unexposed.t"]
]}
