Debugging Rust with Vim’s Termdebug

# Debugging is a first resort.

That’s the prime requirement in my search for an ideal debugging workflow. Running my programs through a debugger shouldn’t be a last resort when I’m at the end of my wits. It should be the first and only way I run the program while I’m working on it. Even when there’s not a bug, if only to step through my freshly fallen code to understand it more intimately. Even when there’s no breakpoints, just to reinforce the habit of Always Be Debugging.

For the first many years of my career, I treated debugging as a last resort. Once I came to adopt this creed, I still didn’t have the tools that suited it. I’ve tried GDB, LLDB (directly and via DAP), nvim-dap with and without nvim-dap-ui, and various IDE debuggers: VSCode with CodeLLDB, RustRover, and Zed. In other words, several different debugger front-ends and back-ends. Each one offered something different and I found sources of inspiration in each one. Still, none of them lived the creed. Read on for an outline of the gripes I have with the debuggers I’ve tried, and a list of my personal requirements for a more ideal debugging setup, followed by a solution discovered in an unlikely place.

# Part 1: What I want, and found wanting, in debuggers

# Part 1: What I found wanting, and want, in debuggers

# Launch friction

Problem statement: launching the program without a debugger is easy; launching it with a debugger is harder. That’s problem one. Examples range from cargo run being easier than cargo build && gdb target/debug/foo, to having to create a launch.json file in VSCode before, to having to click through multiple menus.

In my case, with Rust, that means launching the program with a debugger attached needs to be as easy, or easier, than typing cargo run with some args (foreshadowing…).

# Setting breakpoints out-of-context

First, I tried GDB directly, but encountered so much friction while setting breakpoints that it was unlikely to become habit. It was not a fluid workflow, writing out file paths and copying over line numbers from vim to produce a command like b src/module/lib.rs:172. Breakpoint creation really benefits from being in-context, in-editor. At the same time, GDB has certain breakpoint creation that are more powerful and dynamic than a file+line, and worth having in the back pocket.

# Less-than-pretty printing

Stock GDB also wasn’t great at pretty-printing Rust values, and I hadn’t yet discovered the rust-gdb wrapper that’s bundled with rustup installs, which goes a long way towards solving this issue, but it’s still worth adding as a requirement.

# Debugging tests is hard

I wasn’t far down the GDB road when I realized debugging tests is quite hard. GDB naturally wants to be handed an executable binary with debug symbols. For binary crates, the binary’s location is predictable, but what about tests? Unit tests in a workspace are compiled into a separate binary for each crate. Integration tests are compiled into a separate binary per integration test file. So, one binary per crate and one binary per integration test, stored in target/debug/deps, with unpredictable names.

My main project has 60 test executables to sift through. Is it target/debug/deps/transform-92b4dde7178df872 or target/debug/deps/stats-04a1c2dc2afeb2a7 or one of the other 58 test executables?

# Multiple binaries is hard

This is a lesser cousin of the previous problem. Crates can contain multiple binaries and it should be made easy to debug whichever one I happen to be looking at in the moment.

# Pretty-print mismatch

Next, I walked away from GDB and tried nvim-dap and nvim-dap-ui for quite a while, where breakpoint creation was effortless, but I had issues where the debugger’s (codelldb’s) pretty-printers for complex values would fall out of sync with the rustc version I was using. It’s frustrating to do all the required work of coaxing the program into the desired state in order to debug something, only for the debugger to tell you “The value of x is: An unexpected error has occurred.”

I’ve seen others experience this problem with VS Code’s debugger as well.

# Thread hopping

I also had issues when debugging multithreaded programs without a way (that I could find) to pin to the current thread. Debugging multithreaded programs without this feature quickly becomes incoherent as the debugger steps around to seemingly random lines.

# Inspecting unbound return values

Another issue I ran into while using nvim-dap and other DAP-based debuggers is the perplexing inability to inspect return values. Implicitly returned closures are commonplace in Rust, but this problem applies to explicit returns and regular functions as well. Often I’ll be paused within a function, wanting to see what the function returns, yet the return value is not bound to a variable. For example:

fn have_some_pi(n: f64) -> f64 {
    PI * n
}

The have_some_pi function has an implicit return value which I might want to inspect while debugging. Since no variable is bound to its value, the value can’t be inspected. Sometimes the expression (PI * n) can be evaluated by the debugger, but my experience has been that expression evaluation in Rust almost never works.

For some debugging hotspots, I’ve chosen to rewrite code in this form, simply to make debugging the return value possible.

1
fn have_some_pi(n: f64) -> f64 {
2
  let ret = PI * n;
3
  ret
4
}

Now a breakpoint can be set on line 3 in order to inspect the value, and it’s usually an easy edit to make. I’m not against writing code with debuggability in mind, but implicit returns are an idiom in Rust. It feels so unnecessary to reduce the idiomatical adherence of the code due to a DAP gap, especially when the debugger on the other side of DAP supports logging return values. Just don’t make me do this. Clippy will yell at me.

# Dynamic command-line args

My next requirement is around command-line options. I work on programs with a multitude of them, and often want to debug a very specific combination of options.

VSCode’s approach to command-line arguments entails entering them into a version-controlled JSON file.

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "codelldb",
      "request": "launch",
      "name": "Debug executable 'foo'",
      "cwd": "${workspaceFolder}",
      "program": "${workspaceFolder}/target/debug/foo",
      "args": [
        "--how-do-you",
        "stand",
        "entering",
        "--command-line",
        "arguments",
        "this",
        "way",
        "-?"
      ]
    }
  ]
}

I can’t argue with the utility of putting a debugging spec file in source control so all team members can benefit. It’s even possible to use launch.json in vim with nvim-dap. That’s all fine and good, and maybe I work on weird programs, but the kind of programs I work on need different args nearly every time I debug them. Take a command like foo -j --verbose --filter "core_attest" --id 8236, familiar and functional and perfect, and suffer coercing it into JSON for essentially no reason. It’s the clerical tedium of my nightmares.

Trail not the commas
Quote ever the strings
Endure first the trial
Debug then the things

No thanks. I wouldn’t even ask an LLM to do that.

# Per-project setup

Many IDE’s debuggers seem to have a “bookkeeping before debugging” mantra. VS Code’s required creation of launch.json is an example of this.

Even worse, some debuggers require new projects to appease a GUI configuration wizard before debugging. Some debuggers require this before every debug run. I would like to clone a new Rust repo and start debugging it with a few keystrokes only. Despite the earlier statements about GDB, this is one area where it excels. If you have a binary with debug symbols, you can debug it.

# Intermission

That’s it for the requirements I accumulated from personal experience with a variety of debuggers, many of which went unmentioned. In the next part, I’ll cover the approach I settled on and how it meets the requirements.

# Part 2: Creating a solution

Now that there’s a list of requirements, it’s tempting to analyze them before choosing a solution. In reality, these requirements are gradually-accumulated brain gunk during a random walk of the debuggers on offer. In other words, the requirements weren’t wholly formed, and couldn’t be written down, until the random walk was over and I had a solution in hand. Still, some analysis is informative.

Here are the requirements broken down by what irritant caused each requirement to arise.

That’s four complaints about GDB, four about DAP-based debuggers, one that applies to the editor or IDE in use, and then requirement 0 applies to All, the whole debugging stack. It’s the super-requirement sum of other requirements, and is basically a restatement of my debugging goals.

The GDB irritants are solvable. Of the DAP irritants, some are solvable and some are not, or perhaps they are solvable and their solutions were undiscoverable by me. Regardless, let’s look at the solution.

# Solution description

The solution I settled on is presented as a neovim plugin, rust-termdebug.nvim, but the plugin is not the star of the show. The stars are GDB, vim’s termdebug feature, and indirectly, cargo for being omnipresent.

Termdebug might need an introduction. It’s a vim feature that embeds a GDB instance into a window and communicates with it via GDB/MI, a text-based protocol for interacting with GDB.

Termdebug solves the biggest gripe I had with GDB, since it allows setting breakpoints in-context with the code I’m editing.

The rust-termdebug.nvim plugin has these purposes:

• integrate with cargo to locate debuggable code
• prioritize currently edited code
• create keymaps for debugging actions
• smooth some termdebug behaviors I deemed to be rough edges

# Solutions for each requirement

Here’s how this setup solves each requirement.

# Requirement 0

launching via debugger should be as easy, or easier, than any other way.

The debugger is launched with \ds (debug start) to launch GDB inside termdebug, then r in the GDB window. Launching a program this way is faster than moving to a terminal and typing cargo run. This lack of friction creates a great incentive to Always Be Debugging.

Note: the \ is vim’s default “leader” key that prefixes normal mode commands. Many, including myself, remap it to the spacebar, but I write it here as \ to respect the default.

# Requirement 1

can set breakpoints within the context of the code I’m working on.

Termdebug provides :Break which sets a breakpoint on the current line. I map this to \b.

# Requirement 2

has excellent formatters for complex data types in rust

rust-gdb improves the pretty printing of Rust values dramatically. Generally, the closer to the standard library the types are, the prettier they’ll print. For frequently used types that still don’t print well, custom GDB formatters are an option.

# Requirement 3

must be able to debug tests easily

The debugger can be launched with \dt (debug tests). The plugin consults cargo to learn the location of the test executables.

(The plugin also provides \de to debug examples/)

# Requirement 4

multiple binaries in a workspace should all be debuggable

If the debugger is launched (\ds) in a workspace with multiple binaries, they’ll be listed and if one of them is currently being edited, it will be at the top of the list.

# Requirement 5

debugger must be sync’d to rust version.

Installing Rust with rustup will also install rust-gdb of a compatible version. It’s also available in some non-rustup packaging systems, like rust-gdb in Fedora (thanks cuviper!).

# Requirement 6

can pin the debugger to the current thread.

The plugin maps \dp (debug pin) to the GDB command set scheduler-locking on.

# Requirement 7

can print return values.

The GDB command finish (or fin) will return from the current function and print the return value.

# Requirement 8

command-line args are entered like command-line args.

After entering \ds or \dt to start the debugger, command-line arguments are entered in the GDB window in the familiar way, following the run (or r) command. The example arguments from earlier would be entered like this:

\ds   # to start the debugger

# move to GDB window

run -j --verbose --filter "core_attest" --id 8236

# Requirement 9

minimal-to-no setup time for debugging new rust projects

Thanks to the near-universality of cargo, this arrangement of tools should make the majority of cargo projects debuggable without any initial setup. However, there are Rust projects that don’t use cargo, and there are cargo projects that don’t produce locally-executable debug-symbol-laden binaries, such as cross-target compilation for multiple CPU architectures or for WebAssembly. Those would require additional thought to integrate into the plugin.

# Trade-offs

There are some caveats, the main one being that this only works with Rust. GDB works with several languages, but this setup is deeply tied to Rust and especially cargo.

The plugin currently doesn’t offer a great way to attach to a running process. It’s doable today with \ds and GDB’s attach command, but just starting the debugger automatically runs a cargo build, unnecessarily in the case of attaching. Looking up the PID could be much easier with a smartly sorted and filterable list. This is a TODO.

I don’t work often on embedded systems or other cross-compiled targets. Some WebAssembly, sometimes. What I’m getting to is that debugging cross-compiled binaries isn’t something I’ve tried yet, though I know GDB supports it.

Another caveat is that naturally this setup supports only GDB. If LLDB or some other debugger is your preference, it won’t work. Theoretically LLDB could work since it supports gdb-mi, but there are still some gaps that prevent it from working inside termdebug.

The last caveat is that I’ve used the word “vim” pervasively here, but the plugin is written for neovim in Lua, and isn’t compatible with vim at all. Since termdebug is a much more important piece of the puzzle than the Lua plugin, and termdebug is a part of vim and neovim only inherits it, I felt compelled to say “vim” to give due credit for termdebug. I would have preferred to write the plugin for vim and thereby support both editors, I just don’t know vimscript well enough to do that. If someone (or something) ports it to vimscript, I’d be thrilled.