NEW: Introducing jupyterbook-latex and added handling of tableofcontents for latex (PR #1167)

This PR brings improvements to `PDF` builds via `LaTeX` by incorporating the
[jupyterbook-latex](https://github.com/executablebooks/jupyterbook-latex

) extension.

This initial release:

1. harmonises the document structure for `html` and `latex` outputs, by implementing support for `part`/`chapter` structures in the `_toc.yml` for PDF via LaTeX,
2. brings support for the jupyter-book directive `tableofcontents` to PDF via LaTeX, and
3. adds support for commonly used tags on `code-cell` directives (such as `hide-cell`) to PDF via LaTeX

Co-authored-by: mmcky <mamckay@gmail.com>
Co-authored-by: mmcky <mmcky@users.noreply.github.com>

.github/workflows/pdf.yml

@@ -66,11 +66,12 @@ jobs:
         sudo apt-get install -y     \
           texlive-latex-recommended \
           texlive-latex-extra       \
-          texlive-fonts-recommended \
           texlive-fonts-extra       \
+          fonts-freefont-otf        \
           texlive-xetex             \
           latexmk                   \
           xindy
+
     - name: Build PDF from LaTeX (Docs)
       run: |
         jb build docs --builder pdflatex -n -W --keep-going

.github/workflows/tests.yml

@@ -57,7 +57,10 @@ jobs:
           texlive                 \
           texlive-xetex           \
           texlive-latex-extra     \
-          latexmk
+          texlive-fonts-extra     \
+          fonts-freefont-otf      \
+          latexmk                 \
+          xindy
 
     - name: Run pytest
       run: |

docs/advanced/pdf.md

@@ -112,7 +112,20 @@ For `Windows` users, please install [TeX Live](https://www.tug.org/texlive/windo
 
 ### Build
 
-To build a single PDF using LaTeX, use the following command:
+`jupyter-book` uses the [jupyterbook-latex](https://github.com/executablebooks/jupyterbook-latex) package
+which handles much of the customised LaTeX infrastructure. A feature list of this package can be found
+[here](https://github.com/executablebooks/jupyterbook-latex/blob/master/docs/intro.md#feature-list). It
+enables building `pdf` files with full support for the `file` and `part/chapter`
+[structures that are defined in the _toc.yml](https://jupyterbook.org/customize/toc.html).
+
+If you need to **turn off** this package, the following config is required:
+
+```yaml
+latex:
+  use_jupyterbook_latex: false
+```
+
+To build a PDF of your project using LaTeX, use the following command:
 
 ```bash
 jupyter-book build mybookname/ --builder pdflatex
@@ -135,6 +148,14 @@ jb build mybookname/ --builder latex
 
 **Individual PDF Files:**
 
+:::{warning}
+The current implementation of `--individualpages` does **not** make use of the improvements
+introduced by [jupyterbook-latex](https://github.com/executablebooks/jupyterbook-latex) and
+uses the default `latex` writer included with Sphinx.
+We are currently working on making improvements to how `--individualpages` are constructed.
+You can track progress [here](https://github.com/executablebooks/jupyterbook-latex/issues/41)
+:::
+
 To build PDF files for each page of the project,
 you can specify the option `--individualpages` for `--builder=pdflatex`.
 
@@ -166,27 +187,29 @@ easier to find.
 
 ### Using a different LaTeX engine
 
-Some users may want to switch to using a different LaTeX engine to build the
-`PDF` files. For example, if your project contains `Unicode` you will need to
-use `xelatex` to build the `PDF` file.
+The current default is to use `xelatex` to build `pdf` files.
+
+:::{warning}
+The `--individualpages` option currently uses `pdflatex` by default.
+:::
 
-To update the `LaTeX` engine to `xelatex` you can add the following to your `_config.yml`
+Some users may want to switch to using a different LaTeX engine such as `pdflatex`.
+To revert the `LaTeX` engine to `pdflatex` you can add the following to your `_config.yml`
 
 ```yaml
 latex:
-  latex_engine: xelatex
+  latex_engine: pdflatex
 ```
 
 :::{note}
-We will be making `xelatex` the default in the near future, so this can be used to
-specify other builders such as `pdflatex`, or `lualatex`.
-
-See the Sphinx documentation [for available builders](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine)
+The Sphinx documentation [for available builders](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine)
+contains a full list of supported `latex` builders.
 :::
 
 ### Other Sphinx LaTeX settings
 
-Other [LaTeX settings](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine) available to Sphinx can be passed through using the config section
+Other [LaTeX settings](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_engine) available
+to Sphinx can be passed through using the config section
 of `Sphinx` in the `_config.yml` file for your project.
 
 For example, if you would like to set the [latex_toplevel_sectioning](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-latex_toplevel_sectioning) option to use `part` instead of `chapter` you would use:

jupyter_book/__init__.py

@@ -3,10 +3,13 @@ from pathlib import Path
 
 from .toc import add_toc_to_sphinx, add_toctree
 from .directive.toc import TableofContents, SwapTableOfContents
+from sphinx.util import logging
 
 
 __version__ = "0.10.1"
 
+logger = logging.getLogger(__name__)
+
 
 def add_static_files(app, config):
     """Search the static files paths and initialize any CSS of JS files."""
@@ -30,6 +33,7 @@ def setup(app):
 
     # Path for `_toc.yml`
     app.add_config_value("globaltoc_path", "toc.yml", "env")
+    app.add_config_value("use_jupyterbook_latex", True, "env")
 
     # Add custom static files to the sphinx build
     app.connect("config-inited", add_static_files)
@@ -40,6 +44,7 @@ def setup(app):
     # Transforms
     app.add_post_transform(SwapTableOfContents)
 
+    # Extensions
     return {
         "version": __version__,
         "parallel_read_safe": True,

jupyter_book/config.py

@@ -159,6 +159,9 @@ def get_final_config(
         "latex_individualpages": cli_config.pop("latex_individualpages"),
     }
 
+    if sphinx_config.get("use_jupyterbook_latex"):
+        sphinx_config["extensions"].append("jupyterbook_latex")
+
     # finally merge in CLI configuration
     _recursive_update(sphinx_config, cli_config or {})
 
@@ -311,6 +314,7 @@ def yaml_to_sphinx(yaml: dict):
     if latex:
         for spx_key, yml_key in [
             ("latex_engine", "latex_engine"),
+            ("use_jupyterbook_latex", "use_jupyterbook_latex"),
         ]:
             if yml_key in latex:
                 sphinx_config[spx_key] = latex[yml_key]
@@ -325,6 +329,7 @@ def yaml_to_sphinx(yaml: dict):
     extra_extensions = yaml.get("sphinx", {}).get("extra_extensions")
     if extra_extensions:
         sphinx_config["extensions"] = get_default_sphinx_config()["extensions"]
+
         if not isinstance(extra_extensions, list):
             extra_extensions = [extra_extensions]
 

jupyter_book/default_config.yml

@@ -63,6 +63,7 @@ html:
 # LaTeX-specific settings
 latex:
   latex_engine              : pdflatex  # one of 'pdflatex', 'xelatex' (recommended for unicode), 'luatex', 'platex', 'uplatex'
+  use_jupyterbook_latex     : true # use jupyterbook-latex for pdf builds as default
 
 #######################################################################################
 # Launch button settings

jupyter_book/directive/toc.py

@@ -2,8 +2,15 @@ from docutils import nodes
 
 from sphinx.util import logging
 from sphinx.util.docutils import SphinxDirective
-from sphinx.transforms import SphinxTransform
+from sphinx.util.nodes import clean_astext
+from sphinx.transforms.post_transforms import SphinxPostTransform
+from sphinx import builders
+from pathlib import Path
+from typing import Dict
 
+import copy
+
+from sphinx import addnodes
 from docutils.nodes import compound
 
 logger = logging.getLogger(__name__)
@@ -22,29 +29,156 @@ class TableofContents(SphinxDirective):
         return [TableOfContentsNode()]
 
 
-class SwapTableOfContents(SphinxTransform):
-    default_priority = 700
+def process_toc_dict(
+    globaltoc: Dict[str, nodes.Element],
+    parent_file: str,
+    filtered_toc: Dict[str, nodes.Element],
+):
+    """Filters globaltoc to take children of the current page only
+    :param globaltoc: toc entries
+    :param parent_file: chapter filename
+    :param filtered_toc: filtered toc entries
+    """
+    for key, val in globaltoc.items():
+        if key == "file":
+            if val == parent_file:
+                del globaltoc["file"]
+                return globaltoc
+        if key == "sections":
+            for item in val:
+                if "file" in item and item["file"] == parent_file:
+                    del item["file"]
+                    filtered_toc = item
+                    break
+                elif "sections" in item:
+                    filtered_toc = process_toc_dict(item, parent_file, filtered_toc)
+            if filtered_toc:
+                return filtered_toc
+    return
+
+
+def has_toc_yaml(
+    self, subnode: nodes.Element, tocdict: Dict[str, nodes.Element], depth: int
+) -> None:
+    """constructs toc nodes from globaltoc dict
+    :param subnode: node to which toc constructed here is appended to
+    :param tocdict: dictionary of toc entries
+    :param depth: current toclevel depth
+    """
+    depth += 1
+    for key, val in tocdict.items():
+        if key in ["file", "url"]:
+            internal = False
+            if "title" in tocdict:
+                title = tocdict["title"]
+            else:
+                if val not in self.env.titles:
+                    continue
+                title = clean_astext(self.env.titles[val])
+            if "url" in tocdict:
+                if "http" in tocdict["url"]:
+                    internal = False
+                else:
+                    continue
+            else:
+                val = "%" + val
+                internal = True
+            reference = nodes.reference(
+                "",
+                "",
+                internal=internal,
+                refuri=val,
+                anchorname="",
+                *[nodes.Text(title)],
+            )
+            para = addnodes.compact_paragraph("", "", reference)
+            item = nodes.list_item("", para)
+            item["classes"].append("tableofcontents-l%d" % (depth))
+            subnode.append(item)
+        if key in ["sections"]:
+            sectionlist = nodes.bullet_list().deepcopy()
+            sectionheader = None
+            for item in val:
+                if "part" in item:
+                    sectionheader = handle_toc_header(item["part"])
+                    sectionlist.append(sectionheader)
+                    del item["part"]
+                    has_toc_yaml(self, sectionlist, item, depth)
+                else:
+                    has_toc_yaml(self, sectionlist, item, depth)
+            subnode.append(sectionlist)
+
+
+def handle_toc_header(val: str) -> nodes.Element:
+    """Constructs node for the headers in globaltoc
+    :param val: value of the node
+    """
+    para = addnodes.compact_paragraph("", "", nodes.Text(val))
+    item = nodes.list_item("", para)
+    item["classes"].append("fs-1-2")
+    return item
+
+
+class SwapTableOfContents(SphinxPostTransform):
+    default_priority = 900
+
+    def _get_parent_file(self, tocnode: nodes.Element) -> str:
+        """searches parent nodes to find the chapter name"""
+        if isinstance(tocnode.parent, nodes.document):
+            parent_file_path = Path(tocnode.parent.attributes["source"]).relative_to(
+                self.env.app.confdir
+            )
+            parent_file = str(parent_file_path).replace(parent_file_path.suffix, "")
+            return parent_file
+        else:
+            return self._get_parent_file(tocnode.parent)
 
     def apply(self):
-        # toctrees will have a compound just before them
-        toctrees = [
-            ii
-            for ii in self.document.traverse(compound)
-            if "toctree-wrapper" in ii.attributes["classes"]
-        ]
-
-        # replacing all TableOfContentsNode with tocnode
-        for index, tocnode in enumerate(self.document.traverse(TableOfContentsNode)):
-            if len(toctrees) == 0:
-                warn = f"Found tableofcontents directive in {self.env.docname}, but this file has no descendents."  # noqa: E501
-                logger.warning(warn)
-
-            # Replace the tableofcontents node with the page's toctree
-            wrappernode = compound(classes=["tableofcontents-wrapper"])
-            for itoc in toctrees:
-                toctree = itoc.children[0].deepcopy()
-                toctree.attributes["hidden"] = False
-                if index == len(self.document.traverse(TableOfContentsNode)):
-                    itoc.parent.remove(itoc)
-                wrappernode.append(toctree)
-            tocnode.replace_self(wrappernode)
+        if isinstance(self.env.app.builder, builders.latex.LaTeXBuilder):
+            # for the case of LaTeX builder
+            # if Latex Builder makes reference nodes instead of using toctree directive
+            parent_file = None
+            for index, tocnode in enumerate(
+                self.document.traverse(TableOfContentsNode)
+            ):
+                parent_file = self._get_parent_file(tocnode)
+
+                ret = []
+                wrappernode = nodes.compound(classes=["tableofcontents-wrapper"])
+
+                depth = 0
+
+                filtered_toc = process_toc_dict(
+                    copy.deepcopy(self.config.globaltoc), parent_file, filtered_toc=None
+                )
+
+                wncopy = wrappernode.deepcopy()
+                has_toc_yaml(self, wncopy, filtered_toc, depth)
+                ret.append(wncopy)
+                tocnode.replace_self(ret)
+        else:
+            # for other builders apart from LaTeX
+            # toctrees will have a compound just before them
+            toctrees = [
+                ii
+                for ii in self.document.traverse(compound)
+                if "toctree-wrapper" in ii.attributes["classes"]
+            ]
+
+            # replacing all TableOfContentsNode with tocnode
+            for index, tocnode in enumerate(
+                self.document.traverse(TableOfContentsNode)
+            ):
+                if len(toctrees) == 0:
+                    warn = f"Found tableofcontents directive in {self.env.docname}, but this file has no descendents."  # noqa: E501
+                    logger.warning(warn)
+
+                # Replace the tableofcontents node with the page's toctree
+                wrappernode = compound(classes=["tableofcontents-wrapper"])
+                for itoc in toctrees:
+                    toctree = itoc.children[0].deepcopy()
+                    toctree.attributes["hidden"] = False
+                    if index == len(self.document.traverse(TableOfContentsNode)):
+                        itoc.parent.remove(itoc)
+                    wrappernode.append(toctree)
+                tocnode.replace_self(wrappernode)

setup.py

@@ -17,6 +17,7 @@ doc_reqs = [
 # Test requirements
 test_reqs = [
     "coverage",
+    "texsoup",
     "pytest>=3.6,<4",
     "pytest-cov",
     "pytest-xdist",
@@ -82,6 +83,7 @@ setup(
         "sphinx-thebe>=0.0.6",
         "sphinx-panels~=0.5.2",
         "nested-lookup~=0.2.21",
+        "jupyterbook-latex~=0.2.0",
     ],
     extras_require=extras,
     entry_points={

tests/books/config/_config_jupyterbooklatex.yml

+sphinx:
+  extra_extensions:
+    - jupyterbook_latex

tests/books/toc/_toc_urllink.yml

@@ -2,7 +2,6 @@ file: index
 title: Toc
 sections:
 - file: content1
-  title: Content1
 - file: content2
   title: Content2
 - file: content3
@@ -14,5 +13,5 @@ sections:
   sections:
   - file: subfolder/asubpage
     title: Asubpage
-  - url: https://jupyterbook.org/file-types/markdown.html
+  - url: www.jupyterbook.org/file-types/markdown.html
     title: Markdown pages

tests/books/toc/index.md

 # Main index
 
+**Table of Contents**
 
 ```{tableofcontents}
 ```

tests/books/toc/subfolder/index.md

 # Subfolder index
+
+**Table of contents directive**
+
+```{tableofcontents}
+```

tests/test_config/test_config_sphinx_command.txt

@@ -6,7 +6,7 @@ execution_allow_errors = False
 execution_excludepatterns = []
 execution_in_temp = False
 execution_timeout = 30
-extensions = ['sphinx_togglebutton', 'sphinx_copybutton', 'myst_nb', 'jupyter_book', 'sphinx_thebe', 'sphinx_comments', 'sphinx.ext.intersphinx', 'sphinx_panels', 'sphinx_book_theme']
+extensions = ['sphinx_togglebutton', 'sphinx_copybutton', 'myst_nb', 'jupyter_book', 'sphinx_thebe', 'sphinx_comments', 'sphinx.ext.intersphinx', 'sphinx_panels', 'sphinx_book_theme', 'jupyterbook_latex']
 html_add_permalinks = '¶'
 html_baseurl = ''
 html_favicon = ''
@@ -26,3 +26,4 @@ numfig = True
 panels_add_bootstrap_css = False
 pygments_style = 'sphinx'
 suppress_warnings = ['myst.domains']
+use_jupyterbook_latex = True

tests/test_config/test_get_final_config_custom_myst_extensions.yml

@@ -27,6 +27,7 @@ final:
   - sphinx.ext.intersphinx
   - sphinx_panels
   - sphinx_book_theme
+  - jupyterbook_latex
   html_add_permalinks: ¶
   html_baseurl: ''
   html_favicon: ''
@@ -71,6 +72,7 @@ final:
   pygments_style: sphinx
   suppress_warnings:
   - myst.domains
+  use_jupyterbook_latex: true
 metadata:
   latex_doc_overrides:
     title: My Jupyter Book

tests/test_config/test_get_final_config_empty_.yml

@@ -24,6 +24,7 @@ final:
   - sphinx.ext.intersphinx
   - sphinx_panels
   - sphinx_book_theme
+  - jupyterbook_latex
   html_add_permalinks: ¶
   html_baseurl: ''
   html_favicon: ''
@@ -68,6 +69,7 @@ final:
   pygments_style: sphinx
   suppress_warnings:
   - myst.domains
+  use_jupyterbook_latex: true
 metadata:
   latex_doc_overrides:
     title: My Jupyter Book

tests/test_config/test_get_final_config_exclude_patterns_.yml

@@ -27,6 +27,7 @@ final:
   - sphinx.ext.intersphinx
   - sphinx_panels
   - sphinx_book_theme
+  - jupyterbook_latex
   html_add_permalinks: ¶
   html_baseurl: ''
   html_favicon: ''
@@ -71,6 +72,7 @@ final:
   pygments_style: sphinx
   suppress_warnings:
   - myst.domains
+  use_jupyterbook_latex: true
 metadata:
   latex_doc_overrides:
     title: My Jupyter Book

tests/test_config/test_get_final_config_execute_method_.yml

@@ -26,6 +26,7 @@ final:
   - sphinx.ext.intersphinx
   - sphinx_panels
   - sphinx_book_theme
+  - jupyterbook_latex
   html_add_permalinks: ¶
   html_baseurl: ''
   html_favicon: ''
@@ -70,6 +71,7 @@ final:
   pygments_style: sphinx
   suppress_warnings:
   - myst.domains
+  use_jupyterbook_latex: true
 metadata:
   latex_doc_overrides:
     title: My Jupyter Book

tests/test_config/test_get_final_config_extended_syntax_.yml

@@ -27,6 +27,7 @@ final:
   - sphinx.ext.intersphinx
   - sphinx_panels
   - sphinx_book_theme
+  - jupyterbook_latex
   html_add_permalinks: ¶
   html_baseurl: ''
   html_favicon: ''
@@ -68,6 +69,7 @@ final:
   pygments_style: sphinx
   suppress_warnings:
   - myst.domains
+  use_jupyterbook_latex: true
 metadata:
   latex_doc_overrides:
     title: My Jupyter Book

tests/test_config/test_get_final_config_html_extra_footer_.yml

@@ -26,6 +26,7 @@ final:
   - sphinx.ext.intersphinx
   - sphinx_panels
   - sphinx_book_theme
+  - jupyterbook_latex
   html_add_permalinks: ¶
   html_baseurl: ''
   html_favicon: ''
@@ -70,6 +71,7 @@ final:
   pygments_style: sphinx
   suppress_warnings:
   - myst.domains
+  use_jupyterbook_latex: true
 metadata:
   latex_doc_overrides:
     title: My Jupyter Book

tests/test_config/test_get_final_config_latex_doc_.yml

@@ -28,6 +28,7 @@ final:
   - sphinx.ext.intersphinx
   - sphinx_panels
   - sphinx_book_theme
+  - jupyterbook_latex
   html_add_permalinks: ¶
   html_baseurl: ''
   html_favicon: ''
@@ -72,6 +73,7 @@ final:
   pygments_style: sphinx
   suppress_warnings:
   - myst.domains
+  use_jupyterbook_latex: true
 metadata:
   latex_doc_overrides:
     targetname: book.tex