Code Navigation for Common Lisp

[avodonosov]

Select Topic Area

Question

Body

Could you enable Code Navigation (at least Jump to Definition) for Common Lisp?
The tree-sitter library, which is used by the "search based" code navigation, has a plugin for Common Lisp, according to the official documentation.

Here is the plugin: https://github.com/theHamsta/tree-sitter-commonlisp

Isn't that enough to just enable the support?

[avodonosov]

It seems github uses the tree-sitter tags for the code navigation [1], [2].
And for the command to work, the language parser should provide a queries/tags.scm file.
Currently tree-sitter-commonlisp does not have it. I opened a pull request [3] with tags query for function definitions.

1 - https://tree-sitter.github.io/tree-sitter/code-navigation-systems
2 - https://queue.acm.org/detail.cfm?id=3487022
3 - tree-sitter-grammars/tree-sitter-commonlisp#13

[avodonosov]

@tclem, @patrickt, can you advise, what blocks Jump to Definition for Common Lisp? And why not allow users to work on this?

GitLab, for example, allows to specify a custom script that produces a Language Server Index Format file: https://docs.gitlab.com/ee/user/project/code_intelligence.html
So users can themselves implement support for their languages.

Can't GitHub allow a similar thing? Maybe allow to specify a script that GitHub will call for changed files to produce tree-sitter tags. It's easy to implement for users (in the above comments I've tried). Only an extension point is needed where user can hook their scripts.

At the moment I go as far as generating hyperlinked colorized version of source code and publishing it at github pages: https://github.com/avodonosov/browsable-colorize

[patrickt]

@avodonosov Hi there! I’m happy to report that there isn’t anything stopping you from working on a Common Lisp backend for code navigation. The documentation is here, and you can check the tree-sitter-elixir repository for examples of another language that contributed jump-to-definition externally.

[avodonosov]

@patrickt, thank you for the response.

We have a tree-sitter grammar for Common Lisp, which includes queries/tags.scm and thus supports tree-sitter tags command.

How to try it for GitHub code navigation?

And of course, it is desirable to be able to iterate on it quickly - observe how it works with GitHub, implement improvements and try again.

While simple things like global function definitions and references are easy for basic cases, there are many nuances or undocumented behaviors (quoting, local vars, macros, what if same symbol names a class and a function, etc) that almost certainly will call for improvements.

[avodonosov]

@patrickt , so how about trying https://github.com/theHamsta/tree-sitter-commonlisp for Common Lisp code navigation?

[patrickt]

@avodonosov Great! I wasn’t aware that there was a tagger already in place for Common Lisp. I’ll see what I can do about merging this in to our tagging/search backend. Thanks for letting me know.

[avodonosov]

Updated the link in my last comment to point the original tree-sitter-commonlisp repo, now that I have write access to it. (Instead of my fork, that I referenced in that last comment originally)

[avodonosov]

@patrickt , if you commit the Common Lisp tagger, how is it done? By copying the query? Or by adding a dependency in some module manager? If so, which module manager, npm, cargo? How the version is specified - will it accept new patch version, will it accept new minor versions (in ~maj.min.path, or ^maj.min.path in npm notation)?

[avodonosov]

I'd like to note a degradation of Common Lisp code search at github.

Around the time of this discussion beginning, it was possible to click anywhere in Common Lisp code at github, and github would select a symbol by closest whitespace or punctuation around the click point. And search results for the selected symbol where presented on the right of the screen.

Here is a screenshot taken at June 13: https://user-images.githubusercontent.com/2444/245358984-0b2a1e09-d740-4dba-a703-4ee79d8a1108.png

So this search provided a kind of "jump to definition" / "find usages" for Common Lisp code.

This didn't work for the use case that motivated me most - exploring library API by quickly navigating from a symbol in a package export list to the symbol definition, where we can look at the function parameters, docstring, comments. Because most often people use so called uninterned symbols in package export list - symbols starting with #:, like #:some-function. Example: https://github.com/edicl/drakma/blob/007e0e59e5f8b155f749f81b02d68245b33fc4da/packages.lisp#L49

The #: confuses gihtub code search - symbol is selected together with #: and the search fails to find some-function when searching for #:some-function.

However, for all other places the search worked and was somewhat useful.

Several weeks ago even this feature has broken. Now if we click in the Common Lisp code, symbols are almost never selected and searched. Click anywhere in the function body here, nothing will be selected: https://github.com/edicl/drakma/blob/007e0e59e5f8b155f749f81b02d68245b33fc4da/request.lisp#L81

At the same time, there is some Common Lisp specific functionality in the code presentation. Some standard Lisp words are highlighted (cond, setf, setq, defun, and, when, ...). And if you click name of the function being defined by a defun form, like (defun read-body ...) at line 152, search for the name is presented at the right of the screen:

This only works for defun, but not for flet, labels, etc.

So, there was some Lisp-specific change, which degraded the functionality. "Jump to definition" is completely lost, only some form of "find usages" remains.

The tree-sitter grammar at https://github.com/theHamsta/tree-sitter-commonlisp tags every call position with @reference.call, tags definitions with @definition.function, so if github relies on such tags, it would allow for "Jump to definition" and "Find usages".

However, it remains unclear how to enable "Jump to definition" for list of exported symbols in package definition. Tree sitter only defines three types of reference tags: @reference.call, @reference.class, @reference.implementation, and none of them is suitable for symbols in export lists.

If github just uses some regular expression to select a symbol around a click point, it would be good to document it, allow to override it for a repo (e.g. some config file in the .github/ folder). And allow to manually correct the selection in the UI - current UI does not allow this.

cc @patrickt, @tclem.

[avodonosov]

Still broken. By some intentional change, as bescribed in the previous comment.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[avodonosov]

Still broken, and I haven't found a solution

[look]

I am working on documentation for how language communities can help us add code navigation. I'll update this discussion when it's ready.