How To Setup the Development Environment

Warning

This guide assumes you are not using Windows. If you are using Windows, you may still be able to get some use from it, but feel free to open an issue if you get stuck!

This guide outlines how to setup the development environment(s) necessary to work on the various components located in the swyddfa/esbonio GitHub repository.

Dev Container

This repository provides a simple devcontainer definition that you can use.

Note that using this container is entirely optional and you can easily setup a development environment without it. The devcontainer is only used to provide a known starting point from which all scripts and automations in the repository can build against.

.devcontainer/devcontainer.json

// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/ubuntu
{
	"name": "Ubuntu",
	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
	"image": "ghcr.io/swyddfa/esbonio-devenv:latest",
	// "build": {
	// 	"dockerfile": "Dockerfile"
	// },
	"containerEnv": {
		"TZ": "UTC"
	},
	// Features to add to the dev container. More info: https://containers.dev/features.
	// "features": {},
	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],
	// Use 'postCreateCommand' to run commands after the container is created.
	// "postCreateCommand": "uname -a",
	// Configure tool-specific properties.
	"customizations": {
		"vscode": {
			"extensions": [
				"charliermarsh.ruff",
				"ms-python.python",
				"tamasfe.even-better-toml"
			]
		}
	}
	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
	// "remoteUser": "root"
}

Makefiles

Use of the Makefiles is optional, but recommended to those who are unfamiliar with the repository.

Many common tasks, from installing tools such as uv to packaging the VSCode extension have been automated in Makefiles located throughout the repository.

While they are written and tested with the environment provided by the devcontainer in mind, the Makefiles try, where possible, to work in “any” Unix-like environment. This is done by reusing tools already present on your $PATH and automatically installing any missing tools into $HOME/.local.

Running make tools in the root of the repository will list all tools required for this project, automatically installing them if necessary

$ make tools
...
/home/vscode/.local/bin/uv      uv 0.5.21
/home/vscode/.local/bin/python  Python 3.13.1
/home/vscode/.local/bin/hatch   Hatch, version 1.14.0
/home/vscode/.local/bin/pre-commit      pre-commit 4.0.1
/home/vscode/.local/bin/npm     10.8.2
/home/vscode/.local/bin/npx     10.8.2

Tip

The provided devcontainer is the result of taking a standard Ubuntu image and running make tools

Full details on how the tools are installed can be found in the .devcontainer/tools.mk file

.devcontainer/tools.mk

ARCH ?= $(shell arch)
BIN ?= $(HOME)/.local/bin

ifeq ($(strip $(ARCH)),)
$(error Unable to determine platform architecture)
endif

NODE_VERSION := 20.18.0
UV_VERSION := 0.5.21

UV ?= $(shell command -v uv)
UVX ?= $(shell command -v uvx)

ifeq ($(strip $(UV)),)

UV := $(BIN)/uv
UVX := $(BIN)/uvx

$(UV):
	curl -L --output /tmp/uv.tar.gz https://github.com/astral-sh/uv/releases/download/$(UV_VERSION)/uv-$(ARCH)-unknown-linux-gnu.tar.gz
	tar -xf /tmp/uv.tar.gz -C /tmp
	rm /tmp/uv.tar.gz

	test -d $(BIN) || mkdir -p $(BIN)

	mv /tmp/uv-$(ARCH)-unknown-linux-gnu/uv $@
	mv /tmp/uv-$(ARCH)-unknown-linux-gnu/uvx $(UVX)

	$@ --version
	$(UVX) --version

endif

# The versions of Python we support
PYXX_versions := 3.9 3.10 3.11 3.12 3.13

# Our default Python version
PY_VERSION := 3.13

# This effectively defines a function `PYXX` that takes a Python version number
# (e.g. 3.8) and expands it out into a common block of code that will ensure a
# verison of that interpreter is available to be used.
#
# This is perhaps a bit more complicated than I'd like, but it should mean that
# the project's makefiles are useful both inside and outside of a devcontainer.
#
# `PYXX` has the following behavior:
# - If possible, it will reuse the user's existing version of Python
#   i.e. $(shell command -v pythonX.X)
#
# - The user may force a specific interpreter to be used by setting the
#   variable when running make e.g. PYXX=/path/to/pythonX.X make ...
#
# - Otherwise, `make` will use `$(UV)` to install the given version of
#   Python under `$(BIN)`
#
# See: https://www.gnu.org/software/make/manual/html_node/Eval-Function.html
define PYXX =

PY$(subst .,,$1) ?= $$(shell command -v python$1)

ifeq ($$(strip $$(PY$(subst .,,$1))),)

PY$(subst .,,$1) := $$(BIN)/python$1

$$(PY$(subst .,,$1)): | $$(UV)
	$$(UV) python find $1 || $$(UV) python install $1
	ln -s $$$$($$(UV) python find $1) $$@

	$$@ --version

endif

endef

# Uncomment the following line to see what this expands into.
#$(foreach version,$(PYXX_versions),$(info $(call PYXX,$(version))))
$(foreach version,$(PYXX_versions),$(eval $(call PYXX,$(version))))


# Hatch is not only used for building packages, but bootstrapping any missing
# interpreters
HATCH ?= $(shell command -v hatch)

ifeq ($(strip $(HATCH)),)

HATCH := $(BIN)/hatch

$(HATCH): | $(UV)
	$(UV) tool install hatch
	$@ --version

endif

PRE_COMMIT ?= $(shell command -v pre-commit)

ifeq ($(strip $(PRE_COMMIT)),)
PRE_COMMIT := $(BIN)/pre-commit

$(PRE_COMMIT): | $(UV)
	$(UV) tool install pre-commit
	$@ --version

endif

PY_TOOLS := $(HATCH) $(PRE_COMMIT)

# Set a default `python` command if there is not one already
PY ?= $(shell command -v python)

ifeq ($(strip $(PY)),)
PY := $(BIN)/python

$(PY): | $(UV)
	$(UV) python install $(PY_VERSION)
	ln -s $$($(UV) python find $(PY_VERSION)) $@
	$@ --version
endif

# Node JS
NPM ?= $(shell command -v npm)
NPX ?= $(shell command -v npx)

ifeq ($(strip $(NPM)),)

NPM := $(BIN)/npm
NPX := $(BIN)/npx
NODE := $(BIN)/node
NODE_DIR := $(HOME)/.local/node

$(NPM):
	curl -L --output /tmp/node.tar.xz https://nodejs.org/dist/v$(NODE_VERSION)/node-v$(NODE_VERSION)-linux-x64.tar.xz
	tar -xJf /tmp/node.tar.xz -C /tmp
	rm /tmp/node.tar.xz

	[ -d $(NODE_DIR) ] || mkdir -p $(NODE_DIR)
	mv /tmp/node-v$(NODE_VERSION)-linux-x64/* $(NODE_DIR)

	[ -d $(BIN) ] || mkdir -p $(BIN)
	ln -s $(NODE_DIR)/bin/node $(NODE)
	ln -s $(NODE_DIR)/bin/npm $(NPM)
	ln -s $(NODE_DIR)/bin/npx $(NPX)

	$(NODE) --version
	PATH=$(BIN) $(NPM) --version
	PATH=$(BIN) $(NPX) --version

endif

# One command to bootstrap all tools and check their versions
.PHONY: tools
tools: $(UV) $(PY) $(PY_TOOLS) $(NPM) $(NPX)
	for prog in $^ ; do echo -n "$${prog}\t" ; PATH=$(BIN) $${prog} --version; done

Language Server

The language server is written in pure Python, so if you are familiar with Python development, you should not find anything too surprising here.

We use hatch both for packaging the language server as well as managing all the test environments.

To produce both source and wheel packages run the following command from the lib/esbonio directory

$ hatch build
───────────────────────────────────────── sdist ─────────────────────────────────────────
dist/esbonio-1.0.0b9.tar.gz
───────────────────────────────────────── wheel ─────────────────────────────────────────
dist/esbonio-1.0.0b9-py3-none-any.whl

Running tests is a little more involved, while running hatch test indeed looks promising

$ hatch test
========================================= test session starts =========================================
platform linux -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0
rootdir: /workspaces/develop/lib/esbonio
configfile: pyproject.toml
plugins: asyncio-0.25.2, rerunfailures-14.0, mock-3.14.0, lsp-1.0.0b2, xdist-3.6.1
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=function
collected 254 items

tests/server/feature/test_completion.py ...................                                     [  7%]
tests/server/features/test_directive_completion.py ...............................              [ 19%]
tests/server/features/test_logging.py .......                                                   [ 22%]
tests/server/features/test_role_completion.py ...........                                       [ 26%]
tests/server/features/test_sphinx_config.py ....s..s......ss.s.ss                               [ 35%]
tests/server/test_configuration.py .....s.s.s.s.s.s.s.s.s.s......ssssss......                   [ 51%]
tests/server/test_patterns.py ................................................................. [ 77%]
..........................................................                                      [100%]

=================================== 231 passed, 23 skipped in 1.24s ===================================

This is in fact only a small subset of the full test suite! To see the full set of test environments, run the following command

$ hatch env show --internal
                                                             Matrices
┏━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Name       ┃ Type    ┃ Envs                      ┃ Dependencies                    ┃ Environment variables ┃ Scripts     ┃
┡━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ hatch-test │ virtual │ hatch-test.py3.9          │ coverage-enable-subprocess==1.0 │ UV_PRERELEASE=allow   │ cov-combine │
│            │         │ hatch-test.py3.10         │ coverage[toml]~=7.4             │                       │ cov-report  │
│            │         │ hatch-test.py3.11         │ pytest-lsp>=1.0b0               │                       │ run         │
│            │         │ hatch-test.py3.12         │ pytest-mock~=3.12               │                       │ run-cov     │
│            │         │ hatch-test.py3.13         │ pytest-randomly~=3.15           │                       │             │
│            │         │ hatch-test.py3.9-sphinx6  │ pytest-rerunfailures~=14.0      │                       │             │
│            │         │ hatch-test.py3.9-sphinx7  │ pytest-xdist[psutil]~=3.5       │                       │             │
│            │         │ hatch-test.py3.10-sphinx6 │ pytest~=8.1                     │                       │             │
│            │         │ hatch-test.py3.10-sphinx7 │                                 │                       │             │
│            │         │ hatch-test.py3.11-sphinx6 │                                 │                       │             │
│            │         │ hatch-test.py3.11-sphinx7 │                                 │                       │             │
│            │         │ hatch-test.py3.12-sphinx6 │                                 │                       │             │
│            │         │ hatch-test.py3.12-sphinx7 │                                 │                       │             │
│            │         │ hatch-test.py3.13-sphinx6 │                                 │                       │             │
│            │         │ hatch-test.py3.13-sphinx7 │                                 │                       │             │
│            │         │ hatch-test.py3.10-sphinx8 │                                 │                       │             │
│            │         │ hatch-test.py3.11-sphinx8 │                                 │                       │             │
│            │         │ hatch-test.py3.12-sphinx8 │                                 │                       │             │
│            │         │ hatch-test.py3.13-sphinx8 │                                 │                       │             │
└────────────┴─────────┴───────────────────────────┴─────────────────────────────────┴───────────────────────┴─────────────┘

In addition to an environment for each version of Python we support, we also define an environment for each version of Sphinx we support. The following will run all the tests for a given Python version

$ hatch test --include py=3.x

See the upstream documentation for more details on the hatch test command.

Makefile Targets

For convenience the following make targets are available in the lib/esbonio directory

• make dist: Package esbonio as a *.whl file

• make test: Run the full test suite for your current Python version

VSCode Extension

The development environment for the VSCode is quite involved as not only does it depend on the language server and the TypeScript glue code, but it also requires two separate, standalone Python environments. The fully packaged version of the extension contains the following key folders

esbonio-0.96.1.vsix
└─ extension/
   ├─ bundled/
   │  ├─ env/ (3330 files) [50.93 MB]  <-- Bundled fallback Sphinx environment
   │  └─ libs/ (530 files) [4.39 MB]   <-- Bundled language server and dependencies
   ├─ dist/
   │  └─ node/ (1 file) [794.54 KB]    <-- Glue compiled JavaScript code that integrates the server into VSCode
   ├─ guides/
   └─ syntaxes/

All of which have to be setup on your machine

Tip

When working on the VSCode extension, the provided Makefile is especially useful.

Compiled TypeScript

To compile the necessary type script code, first you need to install the necessary dependencies. Run the following command in the code/ directory

$ npm ci

Then to compile

$ npm run compile

Alternatively, if you want to automatically re-compile each time you modify the source

$ npm run watch

Python Environments

To setup the necessary Python environments, run the following commands in the code/ directory

$ python -m pip install -t ./bundled/env --no-cache-dir --implementation py --no-deps --upgrade -r ./requirements-env.txt
$ python -m pip install -t ./bundled/libs --no-cache-dir --implementation py --no-deps --upgrade -r ./requirements-libs.txt

Install Esbonio

The previous pip install commands installed the necessary third-party dependencies however, we still need to install the esbonio server itself. Chances are you will want to use the version that is in the repository, in which case all we have to do is create a symlink to the right folder. Run the following command in the code/ folder, replacing /path/to/repo with the actual path to your clone of the repository.

$ ln -s /path/to/repo/lib/esbonio/esbonio bundled/libs/esbonio

With that complete you should have everything you need to run the VSCode Extension launch configuration in VSCode!

Install Workspace Extension (Optional)

If you are working on just the language server itself, you can install the VSCode Extension as a workspace local extension, which removes the need for a separate debug VSCode instance. To do this, create a symlink from the .vscode/extensions directory to the code/ directory

$ ln -s /path/to/repo/.vscode/extensions/esbonio /path/to/repo/code

You should then see the option to install the extension in the Recommended section of the VSCode extensions pane.

Call for testing!

You may have noticed we have not mentioned any automated tests… that’s because there isn’t any! 😱

After many attempts, I never figured out a way to write tests that I felt were useful enough to be worth the effort. If you know how to write tests for a VSCode extension I’d love to hear it!

Makefile Targets

The Makefile in the code/ directory provides the following high-level targets

make clean

Remove all non-source files (bundlded/, node_modules, etc.)

make compile

• Install all required development dependencies

• Bootstrap required Python environments (bundled/libs, bundlded/env)

• Create a symlink to lib/esbonio, so that the extension uses the in-repo version of the esbonio server

• Compile the TypeScript under code/src/ into JavaScript under dist/

make watch

Same as make compile, but automatically recompile TypeScript code when modified.

make install

Same as make compile, but also create a symlink from .vscode/extensions/esbonio to code/, allowing you to install the in-repo version of the extension as a workspace local extension

make dist

• Install all required development dependencies

• Bootstrap required Python environments (bundled/libs, bundlded/env)

• Install esbonio from a *.whl file into bundled/libs. By default, the *.whl file will be the latest release downloaded from PyPi. This however, can be changed by setting the WHL variable when invoking make

  $ WHL=./esbonio-1.0-py3-none-any.whl make dist
  

• Compile the TypeScript under code/src/ into JavaScript under dist/