Bringing jscodeshift up to date

[ElonVolo]

I'd like to list some topics that need to be visited in order to bring jscodeshift out of stagnation.

1. The biggest issue is with recast. This library hasn't really had a lot of maintenance for the last couple of years, and there's something like 150+ issues and 40+ pull requests waiting to be merged. It seems like 80% of the issues that are logged against jscodeshift are actually recast issues. In order to fix the jscodeshift's outstanding issues, either recast itself needs to fix them or jscodeshift will need to adopt/create its own fork of recast to solve them. For the past year and a half or so putout's main developer has been maintaining a fork of recast and adding a lot of fixes to it. It might be worthwhile to look at switching to @putout/recast as opposed to the recast upstream. I've also been working on a fork of @putout/recast for evcodeshift that adds a few other things to make evcodeshift transforms more debuggable in vscode.

2. What can be said about recast can probably also be said to a lesser degree about ast-types.

3. How are existing outstanding issues and pull requests to jscodeshift going to be dealt with? Some of the issues/pull requests are from so long ago that I don't know if they're even relevant any more. Also, too many currently open pull requests is can scare away people who want to contribute. At this point, would it make sense to close most of them? If any can be easily and safely merged as is (after a bit of testing) should we go ahead and do so?

4. In a similar vein, a number of issues really aren't labelled. Is there any issue with retroactively labeling them so we can more easily search for needles of defects in a haystack of feature requests?

5. jscodeshift out-of-the-box uses the "babel5compat" setting, which keeps babel parser changes at pre-babel 6 AST's, which ostensibly seems to be done to avoid breaking old existing codemods. The "babylon" configuration, which targets newer post-Babel 6 features that more and more people seem to be complaining they don't have access to, has to be explicitly specified. At this point, should jscodeshift be switched over the babylon as a default?

6. In a similar vein, would any Facebookers being able to tell me what parser configuration is Facebook internally using for their jscodeshift transforms? babel5compat (the current default), babylon, ts, etc. I'm guessing bad things would happen if a new version of jscodeshift broke Facebook legacy codemods, so I'd like to know what I'm up against. It might be possible to hack together some kind of compatibility auto-detect functionality that scans codemod code to guess which parser to use, but I'd like to see if this even needs to be written before attempting to write it.

7. There's a number of issues in the project that aren't labelled, which makes searching explicitly for defects (as opposed to feature requests) a little more challenging. I assume there should be no issue with labelling them to make them more searchable?

[Daniel15]

would any Facebookers being able to tell me what parser configuration is Facebook internally using for their jscodeshift transforms?

Usually either Flow or Hermes. I'm not 100% sure if adapters for these parsers are in this repo, but both parsers are available on astexplorer.net.

How are existing outstanding issues and pull requests to jscodeshift going to be dealt with?

Maybe we can close the very old ones if they don't sound relevant any more?

I assume there should be no issue with labelling them to make them more searchable?

This sounds fine to me!

[MichaelDeBoey]

@Daniel15 @ElonVolo @trivikr Is there anything I can do to help out to get this project up to date again?

[trivikr]

Is there anything I can do to help out to get this project up to date again?

@ElonVolo is driving getting jscodeshift up to date and will respond to your question.

If I was at their place, I would have created issues for each suggestion (if they're not already present) and use GitHub's Project Tables to track a new project (naming it phoenix or jscodeshift returns)

I would have also made releases automated, suggestion in #494, to make the releases more frequent.

[ElonVolo]

Okay, now time for another one of my essay length posts, from most important to least important.

1. I've been on enough rewrite projects named Phoenix to know you should never, ever name your project Phoenix. Can we please avoid the name "Phoenix"? And yes, I'm consider the word to be cursed. :)

2. The biggest challenge facing jscodeshift (other than the documentation that everyone keeps complaining about) is 50-80% of its logged bugs are actually bugs in the recast and ast-types libraries, both of which have a substantial number of bugs and outstanding pull requests, and both of which tend to be updated infrequently. This means there are whole classes of bugs that jscodeshift as a project can't do anything about. At some point, unless something changes with those projects, it's probably going to be necessary to fork both those libraries, put in PR's to upstream back to them, and hope that they'll someday be merged.

3. @trivikr, I agree that all these updates and bug fixes we make are useless to people if we don't regularly publish them to npm. I'm not a Facebook employee (I'm actually cramming for SwiftUI screening tests this weekend so I can hopefully rejoin the ranks of the employed), so I probably don't have access to Facebook's credentials or have the authority to publish stuff to npm or actually know what Facebook's policy about publishing to npm is. So you'll have to refer to @Daniel15 about this one. It would be great if we had something that would let us publish at a regular cadence.

4. The github project tables sound look cool. What advantages do they have over our existing setup? Is their being in beta going to be any issue? (no pun intended) Is there some sort of migration path that would take the users to the new issues table if they click on the "Issues tab"?

5. One of my main priorities is to define concrete list of existing defects that might need to be fixed (if they really are reproducible defects). While a whole bunch of feature requests in the issues section that we'll not be able to implement, but at least we should try to fix existing defects (that aren't recast/ast-type bugs).

So I went through all the issues and closed ones that were ancient or irrelevant because they were fixed and never closed. Everything that remained I made sure had a label (with 1-2 exceptions). I didn't verify that the bugs were reproducable, I just made the distinction that it was reported as a defect.

6. Which is where @MichaelDeBoey's question comes in. We need someone (myself + others) to go through the list of 40+ issues marked as bugs and figure out whether they've been fixed in the most recent version or if they're still outstanding and mark it as such in the comments. If it's an easy fix, by all means fix it, but the most important thing is to make sure that our list of bugs is still relevant.

[List Of Bugs](https://github.com/facebook/jscodeshift/labels/bug](https://github.com/facebook/jscodeshift/labels/bug)

7. I feel we need enhancements to the CONTRIBUTING.md.

Refined bug reports

• The complete code that the person was trying to transform
• The complete code of the transform
• The full CLI command that was typed to do the transform
• The full output that the user was expected
• The actual output the user got, or alternatively the error message that was printed when the transform failed.

jscodeshift issues in large OSS frameworks

We've also had bug reports submitted from people developing projects using large open source frameworks that has jscodeshift baked into them (usually for migration purposes). It was a considerable amount of work learning the open source framework enough to figure out where jscodeshift was supposedly breaking (sometimes it was a bug in the framework), and in some of those cases the custom code that the user was trying to transform wasn't event present (I had to go the OSS frameworks "examples" directory and pick something similar to try to reproduce the error).

I'd like to propose adding something like "if you're having problems with jscodeshift where it's used in framework your project makes use of, please consider pushing up your entire project (or some subset thereof you're allowed to) to a new branch, make it ready-to-go for one of our developers to test, and post the link to it along with any special instructions in your bug report".

[Daniel15]

For bug reports, it's useful for users to link to a basic repro case on https://astexplorer.net/. Doesn't work in all scenarios, but it's great for things that can be reduced down to a few lines of repro code.

so I probably don't have access to Facebook's credentials or have the authority to publish stuff to npm or actually know what Facebook's policy about publishing to npm is. So you'll have to refer to @Daniel15 about this one.

As part of improving the maintenance of jscodeshift, I've volunteered to do that sort of stuff (tagging new jscodeshift versions and publishing them to npm). 😃

It would be great if we had something that would let us publish at a regular cadence.

Do you mean for beta versions, or do we actually want to push stable versions automatically?

[MichaelDeBoey]

Another ast-types related issue: #520

It seems to me that an ownership-transfer or fork would be a good idea imo.

[0xdevalias]

Since the last update on this issue was in August 2022, I just wanted to check in and see what the latest state of things is on this?

If jscodeshift is currently tightly tied to a 'rarely maintained' recast / ast-types; without much ability to fix bugs caused by those tools, should we also consider jscodeshift to be in a 'rarely maintained' state? And if so, are there any suggestions as to what the best 'modern alternative' would be?

[Daniel15]

If jscodeshift is currently tightly tied to a 'rarely maintained' recast / ast-types; without much ability to fix bugs caused by those tools, should we also consider jscodeshift to be in a 'rarely maintained' state?

We could potentially fork recast and ast-types. I think the thing is that we'd need some volunteers to maintain the forks. There's a lot of users of jscodeshift, but so far, nobody has volunteered to maintain a fork of those projects (or replace them with other projects that are more well-maintained).

are there any suggestions as to what the best 'modern alternative' would be?

jscodeshift is still a good tool, and I still use it myself. I haven't kept up-to-date with the open-source ecosystem so I'm not sure what the alternatives are, if any.

Internally at Meta, we're moving towards a new codemod system built on top of Flow. My understanding is that some parts of it are present in the open-source Flow release, but some of the infra around it hasn't been open-sourced yet. It also doesn't handle TypeScript-specific features that aren't available in Flow.

Because of this, there's no longer anyone at Meta officially maintaining jscodeshift. The only person actively doing anything on it is me, and I'm just handling the npm releases, not actually writing any code for the project. I'm going to find out if we can un-Metafy this repo by moving jscodeshift out of the facebook org and into a community one - I already have https://github.com/codemods for that purpose. If not, I think we'll have to fork this repo and archive the original (https://github.com/facebookarchive).

[0xdevalias]

We could potentially fork recast and ast-types. I think the thing is that we'd need some volunteers to maintain the forks. There's a lot of users of jscodeshift, but so far, nobody has volunteered to maintain a fork of those projects (or replace them with other projects that are more well-maintained).

@Daniel15 nods, makes sense.

It might be worth doing some high level analysis to understand what features recast / ast-types bring to each of the tools; as that might make it easier to know what the 'easiest' solution is.

(eg. if recast is fine, but ast-types has lots of bugs; then maybe porting recast to use something else as it's 'base layer' would make sense. Whereas if recast also has lots of bugs/etc, then maybe there is a better dependency that jscodeshift can use in place of it. Etc.)

At the very least, that analysis would 'lower the barrier of entry' for potentially interested contributors to know where could be a good place to start, etc.

This is by no means an authorative deepdive, but came across this old blog post:

• https://medium.com/@cpojer/effective-javascript-codemods-5a6686bb46fb
  • jscodeshift is a toolbox. It brings together babel’s parser (currently called babylon), Ben Newman’s ast-types for node creation and recast for pretty-printing — this is important because we care about printing code with certain formatting rules applied. It wraps these tools up nicely and provides a unified API to find, filter, map and replace AST nodes. It also comes with a runner/worker feature that can apply transforms to thousands of files in parallel.

---

jscodeshift is still a good tool, and I still use it myself. I haven't kept up-to-date with the open-source ecosystem so I'm not sure what the alternatives are, if any.

@Daniel15 nods, makes sense

I'm by no means across the ecosystem in this space either, but from a quick skim through some google results + related link following:

• https://www.libhunt.com/r/jscodeshift

  • Jscodeshift Alternatives

  • Doesn't seem to have anything that jumps out at me
• https://www.libhunt.com/r/codemod

  • Codemod Alternatives

  • Doesn't seem to have anything that jumps out at me
  • https://github.com/facebookarchive/codemod (deprecated, archived)
• https://github.com/codemodsquad/astx

  • Super powerful structural search and replace for JavaScript and TypeScript to automate your refactoring

• https://github.com/ast-grep/ast-grep

  • A CLI tool for code structural search, lint and rewriting. Written in Rust

  • https://ast-grep.github.io/
  • https://ast-grep.github.io/guide/rewrite-code.html
  • https://ast-grep.github.io/playground.html
• https://github.com/hypermod-io/hypermod-community

  • Community-owned global registry and documentation hub for codemods. Write & run codemods, share them with your friends!

  • Seems to largely be based on jscodeshift
  • https://www.codeshiftcommunity.com/
  • https://www.codeshiftcommunity.com/docs/
• https://nx.dev/nx-api/nx/documents/migrate
  • https://nx.dev/recipes/tips-n-tricks/advanced-update
  • https://nx.dev/core-features/automate-updating-dependencies
  • https://nx.dev/extending-nx/recipes/migration-generators
  • https://nx.dev/extending-nx/recipes/modifying-files#ast-manipulation
    • The way this page is written sounds like they may not have a built in 'library of choice' for AST based migrations, and may leave it up to the person implementing the migration
    • Eg. in their example, they are using typescript and tsquery
  • https://nx.dev/extending-nx/recipes/composing-generators#using-jscodeshift-codemods
    • jscodeshift is the only thing mentioned explicitly here
• https://nextjs.org/docs/app/building-your-application/upgrading/codemods

  • Next.js provides Codemod transformations to help upgrade your Next.js codebase when an API is updated or deprecated.

  • https://github.com/vercel/next.js/blob/canary/packages/next-codemod/package.json#L16
    • seems to use jscodeshift
• Use SWC for codemods? swc-project/swc#1643

  • Use SWC for codemods?

  • If you are using swc from rust, writing a human-readable code generator is not a hard task. But if you are using swc from js, ideal codemod is currently impossible because the code generator is writter for a compiler, although retaining original source location may worth

    • Retain lines swc-project/swc#1131

      • Retain lines

      • Babel has a feature where it can retain the same line numbers from the original source file. Is such a feature possible in swc, without degrading the performance?

      • Closing this for now, this could be a possible enhancement, but not something we are looking for immediately. We may revisit this later if things are changing

---

Internally at Meta, we're moving towards a new codemod system built on top of Flow.

Because of this, there's no longer anyone at Meta officially maintaining jscodeshift. The only person actively doing anything on it is me, and I'm just handling the npm releases, not actually writing any code for the project. I'm going to find out if we can un-Metafy this repo by moving jscodeshift out of the facebook org and into a community one

If not, I think we'll have to fork this repo and archive the original (@facebookarchive)

@Daniel15 Interesting. That's definitely useful info to know; thanks for sharing!

Definitely keen to hear what the 'word' is on un-Meta-fying this project, once you find out :)

[ElonVolo]

Here is The State Of The 'Shift.

1. evcodeshift jscodeshift fork

I've been maintaining (ok, somewhat maintaining) an unmeta-fied fork of jscodeshift called evcodeshift. In each release (which I really should be making more often) I bump everything up to the latest version of dependencies as well as use a fork of recast, @putout/recast, which for a while CodeRaiser was maintaining. This fork fixed some errors I was having in my codemods and their unit tests, which was one of my main reasons for forking jscodeshift.

Another reason I forked jscodeshift was Meta is not allowing new command line arguments and features be added to added to jscodeshift (which Meta of course has the right to do as they are the owner), but since I need things like the --gitignore flag, that's another reaso why I've maintained a fork.

Check it out when you get a chance.

https://github.com/ElonVolo/evcodeshift

2. evcodeshift future/jscodeshift rewrite

While jscodeshift has some of the most useful features out there, and I want to give the previous authors of jscodeshift much credit for before saying anything unpositive, jscodeshift also has some of the most confusing, and unmaintainable code that I've ever seen on any project I've ever worked on. Much of the code is written in a hyper-succinct style that takes a great while to detangle in one's head followed by a great many more minutes is spent wondering "why the heck did they do that there?" until you finally figure out "the why of it". This has made the jscodeshift project far more difficult to regularly maintain for everyone. There is also the matter that jscodeshift in its current state is (as far as I'm aware) permanently incompatible with ESM.

That's why I've been taking some of my post-layoff time between job searches to do a rewrite/port of jscodeshift to a certain strongly typed Javasscript-subset language that will be maintainable for the next several decades. My goals is to make sure there is full backwards compatibility between the new version of evcodeshift and everyone's old jscodeshift scripts they wrote 5 years ago. I'm about a 1/3 of the way through the detangling, defining, and demining of the original jscodeshift codebase.

3. ast-types and recast

Most of what people call "jscodeshift" is actually ast-types and recast, as jscodeshift itself is just a thin wrapper containing a babel front end plus some paths stuff and searching stuff. Probably 80% of all issues logged in jscodeshift are actually recast/ast-types issues. And yes, it has been an issue that these libraries are not only not getting updated, but that useful bug fixes in already-submitted good-looking PR's have been languishing unmerged for months.

Since these are projects completely separate from jscodeshift, they have been traditionally out of scope for jscodeshift issues. However there have been attempts to fork at least some of these, by both myself and CodeRaiser.

CodeRaiser found maintaining a fork of recast and keeping it in sync with upstream changes was such a PITA that he eventually stopped and invented something completely different that replaced recast on his project.

I found that both ast-types and recast, like jscodeshift, have some of the most confusing and unmaintainable code I've ever seen on a project. For example, one of recast's unit tests was using the actual transpiled code of the recast library itself as the javascript content for a unit test to parse, and it took me hours to find out why some compilation flag or line ending change was breaking an unrelated unit test for no seemingly darned good reason.

So to answer the qusetion about an ast-types and recast fork, they would need refactored to such an extent that they'll for all intents are purposes they'll be rewrites. And when we get to rewriting this much code it's pretty much a full-time job, which is why I totally get that Ben has a limited amout of time to work on recast/ast-types stuff. And maybe at some point we'll take that on, but for right now the "portwrite" of jscodeshift has priority. As a side of effect of rewriting jscodeshift in a strongly typed language and adding some form of dependency injection to it, alternatives to ast-types and recast can also be an option in the future.

4. The impact of AI on codemodding

At some point, AI is probably going to replace jscodeshift and you've be able to use more natural language to describe the codemods you want and the AI tool will crawl through your entire codebase making those changes flawlessly. So any codemodding solution that exists you choose probably going to have a 5-10 year lifespan before it's considered obsolete.

[trivikr]

4. The impact of AI on codemodding

At some point, AI is probably going to replace jscodeshift

The Amazon Q Code Transformation (preview) announcement today reminded me of this comment.

[Tobbe]

@ElonVolo Thanks for taking this on! Sounds like a lot of work, but you've made great progress! Please keep going 🙏 Your work is appreciated 🙌