Bump tree-sitter version to 0.25.2

[gmlarumbe]

Separate PR from #6076 to update tree-sitter to a newer version after adding tree-sitter-systemverilog CI jobs.

[gmlarumbe]

For some reason pip cannot find the latest tree-sitter version (https://pypi.org/project/tree-sitter/):

ERROR: Could not find a version that satisfies the requirement tree_sitter==0.23.0 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.2.2, 0.19.0, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.20.4)
ERROR: No matching distribution found for tree_sitter==0.23.0

Might it have to do with python version?

On the other hand, tree-sitter-verilog seems unmaintained. There is an open PR from some months ago to update it to version 0.21: tree-sitter/tree-sitter-verilog#76

[gmlarumbe]

@tgorochowik ,

I have made some changes to tree-sitter-systemverilog that should fix almost all failing tests. However this requires version 0.25.2 of tree-sitter which is currently not available via pip.

Would it be possible to use this latest tree-sitter version only for tree-sitter-systemverilog tests? The following snippet is what I use for CI:

git clone https://github.com/tree-sitter/tree-sitter.git
cd tree-sitter
git fetch --tags
latestTag=$(git describe --tags "$(git rev-list --tags --max-count=1)")
git checkout $latestTag
make all && sudo make install

Thanks!

[tgorochowik]

@gmlarumbe sounds great to me! Please try to adjust the CI so it works the way you propose and we can merge it if green, thanks!

[gmlarumbe]

@tgorochowik ,

All tests are failing for tree-sitter-systemverilog due to this version mismatch issue (see #7024):

Incompatible Language version 15. Must be between 13 and 14

[tests/generic/member/class_member_test_0.sv](https://chipsalliance.github.io/sv-tests-results/tests/generic/member/class_member_test_0.sv.html)::: error: unknown error: Parsing failed

I tried updating the conf/requirements.txt but required tree-sitter version is not available with pip yet.

Do you have any suggestion about how to add this latest version as a dependency? Would this latest version also be used by tree-sitter-verilog?

Thanks!

[tgorochowik]

@gmlarumbe what is the exact version that you need?

if the release on pypi is not sufficient, maybe we could switch to using git directly? i.e https://github.com/tree-sitter/py-tree-sitter/tree/v0.24.0 ?

this would be

git+https://github.com/tree-sitter/py-tree-sitter.git@v0.24.0

in the requirements file.

If that would help, we could also use a specific commit instead of the version tag (if you need something from the dev branch for example)

[gmlarumbe]

Hi @tgorochowik ,

The required version would be at least v0.25.2. This version still does not have a release for the Python bindings that are used in the runner.

In my opinion the simplest fix would be to change the runner to use the tree-sitter-cli instead of the Python bindings. This would require installation of tree-sitter-cli (https://github.com/tree-sitter/tree-sitter/tree/master/cli). There are pre-built binaries (https://github.com/tree-sitter/tree-sitter/releases/tag/v0.25.3).

What would be the way to include this pre-built binary of tree-sitter-cli in the conf/requirements.txt file?

Thanks again!

[zoomer-k]

python binding is updating now
tree-sitter/py-tree-sitter#333

not merged yet

[tgorochowik]

@gmlarumbe We already use npm in the build process, I guess we could install tree-sitter-cli through npm before building (instead of using pip).

Alternatively, I think it should be okay to use pip with git pointing to the current last commit from https://github.com/tree-sitter/py-tree-sitter/tree/0.25 (so this is kind of what @zoomer-k suggest - thanks for this btw) - this should probably be simpler as there are fewer changes

[tgorochowik]

I pushed a commit that should switch the package to a dev release, let's see if this helps. If not, feel free to go the npm way

[gmlarumbe]

I guess this would be the simplest solution. However it seems to be incompatible with current runner of tree-sitter-verilog. It throws the following error:

09:27:37 | AttributeError: type object 'tree_sitter.Language' has no attribute 'build_library'

Due to the latest line of its runner in runners.mk:

$(INSTALL_DIR)/lib/tree-sitter-verilog.so:
	mkdir -p $(INSTALL_DIR)/lib
	cd $(RDIR)/tree-sitter-verilog && npm install
	/usr/bin/env python3 -c "from tree_sitter import Language; Language.build_library(\"$@\", [\"$(abspath $(RDIR)/tree-sitter-verilog)\"])"

Probably the runner of tree-sitter-verilog should be updated. Related GitHub issue: tree-sitter/py-tree-sitter#145 (comment)

On the other hand the Code Quality Checks fail with the following error:

ERROR: Package 'tree-sitter' requires a different Python: 3.8.18 not in '>=3.10'

Any idea of what would be the best option?

Thanks again for taking the time to find a solution.

[tgorochowik]

@gmlarumbe would you be willing to update tree-sitter-verilog to use the pre-compiled wheel as suggested in the linked issue?

Alternatively we could try to come up with a way to isolate the runners a bit more and install dedicated dependencies (i.e. separate requirements, maybe venvs?) but that could be a bit more work I imagine.

[gmlarumbe]

I force-pushed some changes that fix the compilation of the shared object for tree-sitter-verilog.

To fix the Code Quality Checks would it be possible to add Python 3.10 to the conf.environment.yml file and coexist with current Python 3.8?

[tgorochowik]

do we even need python 3.8? would it be possible to just switch to 3.10 completely?

[tgorochowik]

I am not sure if the conda environment is still even used though? feel free to modify it if it still is used

[tgorochowik]

Thank you for your hard work on this @gmlarumbe !

Are the results satisfactory?

[gmlarumbe]

Hi @tgorochowik ,

I would say they are not yet since it seems almost all tests are failing both for tree-sitter-verilog and tree-sitter-systemverilog.

Would it be possible to check the SV test logs of this integration CI?

Thanks!

[tgorochowik]

We don't put the output of the tools in the CI logs as they are (at least for some tools and tests) huge, make the CI much slower and the logs much less readable.

That being said, if it helps debugging, feel free to just add some temporary print statements for example here: https://github.com/chipsalliance/sv-tests/blob/master/tools/runners/tree_sitter_systemverilog.py#L100 (I assume self.log is what you need?) - check what is needed, and then we can remove it before merging.

Would this help?

[gmlarumbe]

Hi @tgorochowik ,

I tried adding the GitHub action to sv-tests-ci.yml:

      - name: Checkout code
        uses: actions/checkout@v2
      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '22'
      - name: Check PATH
        run: |
          echo $PATH
      - name: Update apt repos
        run: |
          apt-get update -qq

However the environment cannot find the node executable:

Environment details
  13:44:48 | node: 
  13:44:48 | npm: 10.9.2
  13:44:48 | yarn: 1.22.15

And therefore still gives this error:

13:46:29 | Error when generating parser
13:46:29 | 
13:46:29 | Caused by:
13:46:29 |     Failed to load grammar.js -- Failed to run `node` -- No such file or directory (os error 2)
13:46:29 | make: *** [tools/runners.mk:77: tree-sitter-systemverilog] Error 1

This is the $PATH after running the GitHub action:

Run echo $PATH
13:44:56 | /root/_tool/node/22.14.0/x64/bin/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Any ideas?

[kbieganski]

The path set by setup-node was incorrect, it was missing a separator for some reason. It should've been:

/root/_tool/node/22.14.0/x64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

I switched to nvm. nvm puts stuff in .profile/.bash_profile etc. but GitHub Actions doesn't source that. So I sourced it manually. Now node finally works and the tree-sitter parsers build.
But now they both break at the Run step. Not sure why tree-sitter-verilog is affected.

[tgorochowik]

@gmlarumbe could you please check if everything works as expected after the changes from @kbieganski ?

[gmlarumbe]

Hi @tgorochowik ,

Sorry for the delay in the answer. I have been a bit busy lately.

Regarding the error I do not really think pushed changes have to do anything with it. Let me know if I need to do something else.

Thanks!