organizing to release JQ 1.6.2 and JQ 1.7: Your suggestions/comments would be appreciated

[liquidaty]

This is an offshoot of #2305. Please read that for background.
cc: @stedolan @nicowilliams @wtlangford

I am posting this issue to organize a plan to create a new JQ fork with two primary objectives:

1. Merge existing bug fixes into a new 1.6.2 release
2. Merge existing new features into a new 1.7 release

The plan to create a new for is done out of necessity, not desire-- again, see #2305 for details. This is NOT an attempt to hijack or otherwise replace this repo, and the ultimate goal will be to either merge back into this repo, or to otherwise provide a successor with the blessing of this repo's owners/maintainers. If at any point in this process those maintainers would like to merge this effort back into this repo, that would be welcome

As a start, I would propose that we:

1. agree on a decision-making framework
2. identify a core set of roles, and folks who can fill said roles, to get this off the ground
3. agree on a preliminary target timeline
4. identify existing PRs for bug fixes and enhancements to target for 1.6.2 and 1.7, respectively

Request for comment

This entire issue post is a request for comment to see if we can get this off the ground. I will start with the below proposals. These are certainly incomplete and in need of further refinement / fleshing out, so please offer your suggestions / comments

Decision framework

I'm just going to throw out the simplest way to start: on any decisions that are not obvious, the core team votes (on the other hand, how do we decide the core team? kind of chicken & egg...). Maybe we start with the core team being, whoever are the first few volunteers who are willing commit meaningful time/effort/value to this?

As for decision guidelines, I would propose that a core value is impact-driven prioritization. Surely it is impossible to agree on exactly what outcomes have what impact, but we can also surely agree that a fatal bug fix is more important than a cosmetic one, and an enhancement with 100 likes and dozens of comments is probably more impactful than one with no likes and no comments.

Core set of necessary roles

• Administrator(s). owns the repo and handle stuff like collaborators, permissions etc.
• Merge management: approves merges; makes sure it is correct and agreed-on, and necessary parts are covered (for example, perhaps each merge includes someone who has reviewed code, some test that verifies the change and is incorporated into the CI/CD tests, and verification from original PR author)
• CI/CD testing: this will be critical
• C code reviewer: since for now we are only focused on existing PRs
• Cross compilation: for PRs that work in one platform but not others. Alternatively, the onus could be put on the PR author to fix any cross-compilation issues, and on CI/CD to verify across all platforms

Folks who can fill these roles?

Please volunteer. I'm happy to pitch in for any role, though surely others are more skilled than I at CI/CD or code review (and probably all the others too). @pkoppstein @wader @tst2005 @chickenandpork (apologies in advance for anyone, which there are sure to be, whom I should have mentioned and missed here-- pls do not take personally but rather blame my sleepiness): any takers?

Preliminary target timelines

Suggest May 1, 2023 for version 1.6.2. If there are more bug-fix PRs than can fit in that time, can always plan for 1.6.3 some time thereafter.

Suggest May 15, 2023 for version 1.7. Similarly, If there are more feature-enhancement PRs than can fit in that time, can always plan for 1.7.1 some time thereafter.

identify existing PRs for bug fixes and enhancements to target for 1.6.2 and 1.7, respectively

Note: I have not spent much time reviewing PRs, so I've just started by exporting the PRs to jq_prs-20230307.json.txt and spitting out in markdown below, and then mostly arbitrarily choosing a few to get started

number	title	is_bug	version
#2548	allow, and strip, comments (#, // or /*)	N	1.7
#2547	add utf8tolower, utf8toupper	N	1.7
#2546	Fix segmentation fault when using jq in threads	Y	1.6.2
#2538	fix: tarball fails distcheck: add files missing from manifest	y	1.6.2
#2535	Remove deprecated --nul-output documentation	Y	1.6.2
#2523	Added truncatestream builtin function	N	1.7?
#2519	C99 compatibility enhancements for the configure script		1.7?
#2505	Build and test using GitHub actions workflow		1.6.2?
#2504	doc: Added information on --indent to --help
#2492	Show missing backslash in manual
#2487	Add !x as syntactic sugar for x | not
#2484	Tidy up nomem_handler implementations to fix SEGVs and ensure more consistency between TLS and non-TLS implementations.
#2482	Fix double-installed doc
#2480	Use right headers and flags in configure
#2474	Document how sort_by() can sort by multiple keys, including an example.
#2473	Clarify in the manual how map_values() behaves differently from map().
#2470	Add examples for input & inputs to manual
#2468	Use autotools standard ${docdir}
#2462	docs: clarify split behavior
#2451	Bump lxml from 4.6.3 to 4.9.1 in /docs
#2434	Fixed typo
#2432	[docs] Use yaml.safe_load() in build_*.py scripts
#2398	Clarify use of --argjson option
#2391	Add fixes for manual
#2357	Validate module metadata to be an object
#2355	Link to the Onigurama docs
#2336	Fix typo
#2314	Support binary strings, preserve UTF-8 and UTF-16 errors
#2311	Replace bareword error
#2306	Added usage under Powershell to the manual
#2296	Use a regex to parse ISO8601 dates
#2292	Disable some tests when building without regex support
#2285	Add in-place modification (expression) shorthand in objects
#2256	Fix --indent 0 implicitly enabling compact-output
#2255	Added a first fuzzer for integration with OSS-Fuzz.
#2254	Fix number lexer to avoid conflict with object indexing
#2253	Make 0 divided by 0 results in NaN consistently
#2252	Fold modulo operator on constant values and raise zero remainder error quickly
#2249	Enhance @sh to output objects as bash associative array initialisers
#2241	Added base/1 and unbase/1
#2235	fix handling of -0
#2233	Fix typo
#2216	Add an example usage for the inputs builtin
#2212	Manual error -- key should be "foo" without spaces
#2205	add nix and guix to website and fix typo in manual
#2202	Fix strflocaltime on daylight saving time values
#2188	Add Windows installation via scoop
#2183	Build out of tree
#2157	Fix uri format to follow RFC 3986
#2142	Fix string multiplication with a value between 0.0 and 1.0	Y	1.6.2
#2137	allow MS Windows \r\n style line endings and it's (broken) "UTF-8 BOM"	Y	1.6.2
#2133	Fix deletion using assigning empty against arrays
#2112	Add debug(exp) builtin
#2108	Remove undefined behavior caught by LLVM 10 UBSAN.	Y	1.6.2
#2102	typos in manual.yml
#2099	alloc: properly allocate memory handler per thread
#2098	Adds "tobool" builtin function for converting strings to booleans	N	1.7?
#2094	manual: mention 0-based indexes in array slicing
#2093	Fix documentation for default search paths
#2092	spelling: text
#2081	use homebrew addons
#2079	Correct object construction with variables example
#2067	Fedora link returns 404
#2060	Added lpad and rpad
#2059	#618 alternative to dupn
#2010	Fix to incorrect --stream paths
#2004	Rewrite Dockerfile to use Alpine Linux, and produce a minimal image
#2000	Fix assertion when using multiple data imports
#1997	Use decnum for addition
#1993	Better dumping
#1989	Update manual.yml
#1977	Override field color with JQ_COLORS
#1962	[doc] replace find with select function as the example of jq-coded functions
#1961	implement scan/2 function
#1907	Unify/simplify the MultiByteToWideChar() code and add wrappers for open()/stat()
#1900	Optimise size of docker file
#1869	Make last(empty) empty
#1866	Should work on the entire object instead of each key of it.
#1862	Added jv_is_integer_large() to the library.
#1849	add nested array indexing example
#1843	dlopen(), random, I/O, eval, and co-expressions (co-routines)
#1798	Add fromstream_with_dups() builtin (fix #1795)
#1791	Set field color using JQ_COLORS
#1789	Provide a jvp_dump_raw_string function to implement the combination o…
#1767	Patches for AIX	Y	1.6.2
#1748	Make reverse more DWIM-y
#1743	Save literal value of the parsed number to preserve it for the output
#1738	Add new builtin fetch
#1736	Fix truncate_stream example
#1733	fixed word 'outputting' in manual.yml v1.5
#1726	Rename gen_* functions to jq_gen_*
#1703	wip: feat: terminate jq immediately after the outgoing pipe closed
#1701	Added is{scalar,value,..}. Fixes leaf_paths bug. Leaf paths	Y	1.6.2
#1673	Add support for CRLF line endings in filters	N	1.7
#1672	Add scoop installation instructions
#1671	"writing output failed" with non ASCII input
#1654	Make INDEX/2 more efficient
#1643	Add mul builtin
#1631	Fix Dockerfile
#1570	Fixes issue #1569	Y
#1558	Fix a try-catch example	Y
#1549	Use an unreachable commit ID instead of a tag
#1537	Update manual.yml
#1527	Update the man page to note that the sort functions are stable
#1523	Update maruku and rake dependencies to latest versions
#1522	Add ruby config dotfiles for docs
#1517	Fix build with linux on travis-ci	Y	1.6.2
#1460	Change tutorial to highlight filter part of query versus output part of query
#1458	fix portability problems, found in NetBSD-7.1/amd64	Y	1.6.2
#1454	Example of object creation attribute shortcut
#1440	Revisions to any/2, all/2, IN/1, IN/2 as per #1438
#1327	jv: Add some support for 64 bit ints in a very conservative way (ALTERNATIVE)
#1246	jv: Add some support for 64 bit ints in a very conservative way
#1228	Define format in jq code
#1215	Add support for seccomp
#1201	Add snap packaging support
#1183	More info. Directly install the release binary.
#1169	Fix some typos
#1159	C++ compatibility, C and C++ examples
#1127	Ignore jq.exe, fixed gcc compiler warnings for msys2 (windows).
#1103	Enhance the "add" operator so that, when applied to objects, if two f…
#1102	Allow tonumber to work on strings that contain commas as thousands separators
#1062	project/1, query/1 and unify/1 added to builtin.jq, with tests and documentation
#1060	Mention select() in if-then-else docs
#1058	add debian/ directory
#1032	Dump block
#673	Working module/package system
#458	Move tail call optimisations to compile.c.

[wader]

Great start. Will start going thru issues:

• doc: Added information on --indent to --help #2504

Add a PR comment. Documentation fix.

• Add examples for input & inputs to manual #2470 Add examples for input & inputs to manual

Has PR #2395, also related to #2216. Documentation fix.

• Remove deprecated --nul-output documentation #2535

Think the conclusion for master was --nul-output works, there is code for -0 but is treated as a number instead of flag (and --null-input has never been?). Mostly documentation fix, maybe -0 code should be cleaned up?

• docs: clarify split behavior #2462

Documentation fix.

• Clarify in the manual how map_values() behaves differently from map(). #2473

Documentation fix.

BTW also accidentally noticed that github markdown includes the issue/PR title if you put them in a list - #<number>!

Will continue later when i more time.

[pkoppstein]

@liquidaty - Thanks for getting the ball rolling so nicely!

Your post raises a number of issues, but in this response, I'd like to
focus on just a few that arise because of various complexities.

First, on governance - I suggest that a "core" group be formed
(defined by authority to make changes on github), and that at least
initially it proceed on the basis of unanimity.

Second, because there are so many issues, many of which are actually
quite tricky in one way or another, I think we need to prioritize
carefully and be a bit wary about pre-mature commitments to time lines. I'd
suggest focusing on generic "milestones" in the near term, perhaps along these lines:

Version 1.6.1 - the state of things at the time the fork is created.

Version 1.6.2 - minor documentation updates and fixes for bugs or
performance issues provided the solutions are known
to be quite straightforward.

Version 1.6.3 - incorporate changes of the "low hanging fruit"
variety, avoiding new features but possibly addressing
at most one important issue, e.g. an issue upon which
the resolution of others depends (e.g. the
'builtins.jq' issue (*)).

Version 1.6.4 - additional changes that adhere to the principle of
backward compatibility.

My guess is that there will be other releases before 1.7.

(*) I am not quite sure what the current status of the "builtins.jq" issue
is exactly, but the problem has to do with the run-time costs of
adding additional defs to builtins.jq. The general point of course
is that we have to be careful not to introduce unexpected performance
penalties.

[liquidaty]

@wader thanks for help reviewing. @pkoppstein I like all those ideas.

Does the JQ repo currently have any built-in benchmarking tests? I see some mentions in #1589 and #2386 but can't tell if anything made it into the repo. If not, maybe maintaining some basic benchmarks should be one of the "roles"-- though maybe it's more of a one-time setup

[pkoppstein]

@liquidaty asked about

built-in benchmarking tests?

Obviously the various tests in the tests/ directory can be used for benchmarking.
See also:

• https://github.com/stedolan/jq/wiki/Docs-and-tests#tests
• https://github.com/stedolan/jq/wiki/X---Experimental-Benchmarks
• https://github.com/01mf02/jaq/blob/main/bench.sh

[mfeit-internet2]

#1862 and #1752 are of interest to me because they cause problems in a production system and I've been running a patched version for years.

#1862 should be pulled into a 1.6.x and #1752 should go into 1.7.

I'll be happy to review, beta test, etc. anything related to either.

[hexagonrecursion]

1. Where is the fork? Does it already exist? I have looked at several items in https://github.com/stedolan/jq/forks?include=active&page=1&period=2y&sort_by=last_updated but nothing looks like it is actively maintained.
2. I agree with @pkoppstein: pre-mature commitments to time lines might do more harm than good. I also agree with their milestone criterias

[liquidaty]

The fork has not been created. From my perspective, the main thing that needs to happen first, is to establish the "core" group. This would be the group that has authority to make changes on github, and would initially make decisions based on unanimity.

I can appreciate that pre-mature commitments can be easily overdone, but there needs to be some level of commitment to make this work as a group, otherwise it might as well be a one-person project which defeats the entire purpose. Said another way, as part of determining that core team, we would need to have an understanding of some minimal level of commitment (e.g. obviously, unanimous decision-making can't happen if decision-makers are not available) and some very basic initial milestones or general objectives as to what we are trying to do in what time frame.

I would propose the following issues to be first addressed with the core team. And this is going to be a bit of a chicken and egg process given that we don't have a core team yet, but we have to start somewhere.

1. what issues / pull request comprise the total universe of potential issues to include for each of the various contemplated upcoming versions
2. how a change request should be initiated for review / approval; what are the necessary prerequisites
3. what the QC process will be
4. how often, at a minimum, each core member can make themselves available to review anything that awaits their feedback on a decision. we don't want anyone to feel like this requires an unreasonable amount of work on their part, and we also don't want to be stuck for 6 months waiting for a core member to opine on whether a pull request should be accepted
5. what differences we have amongst the team that we should consider so that we are taking advantage of our individual strengths to optimize effectiveness while minimizing burden

This is just a very rough initial proposal so please comment as you see fit. More to the point, assuming we can come to some reasonable agreement about how all this will come together:

More directly to the point: who is up for being part of (or at least considering being part of) the core team? and/or just being part of the team, but non-core? @pkoppstein, @wader, @chickenandpork, @tst2005, @hexagonrecursion, @mfeit-internet2 (or anyone else I missed)?

[nicowilliams]

The biggest problem I have maintaining jq, besides my own lack of time, is the inability to add maintainers w/o @stedolan's help.

Therefore, @owenthereal and I set up https://github.com/jqlang/jqlang a while back in preparation for forking some day. I've invited @stedolan to be the owner of that org. I will invite him again.

What I need right now in order to make that fork a reality is: candidates to be maintainers. To pick from the candidates I guess I'd want a small set of PRs or something to look at.

Mind you, you guys are pretty fed up -as you should be- with me and @stedolan, so if you want to fork using a different org and pick your own maintainers, that's fine. jqlang seems like a fine org name though.

Thoughts?

[mrienstra]

Stoked to hear from you @nicowilliams, your input is very welcome!

My voice shouldn't count for much -- I'm just a rando who got excited about the idea of helping this project get unstuck -- but I'm in favor of https://github.com/jqlang/jq as the new fork. Though it's worth mentioning the existence of https://github.com/jqmaintainers/jq, created by @tst2005.

Again, no particular reason to listen to me, but personally I think @wader & @itchyny should be fast-tracked in as a maintainers. I haven't looked closely at other people's contributions recently -- and again I'm not familiar enough with jq to be any kind of gatekeeper -- but just for convenience, here are some links, in alphabetical order (sorry if I missed anyone who has been active lately, I kinda threw this together):

adoxa	PRs	branches	commits	comments
agordon	PRs	branches	commits	comments
carlsmedstad	PRs	branches	commits	comments
chickenandpork	PRs	branches	commits	comments
dgryski	PRs	branches	commits	comments
DRMacIver	PRs	branches	commits	comments
dtolnay	PRs	branches	commits	comments
emanuele6	PRs	branches	commits	comments
hexagonrecursion	PRs	branches	commits	comments
itchyny	PRs	branches	commits	comments
jankatins	PRs	branches	commits	comments
leonid-s-usov	PRs	branches	commits	comments
liquidaty	PRs	branches	commits	comments
mfeit-internet2	PRs	branches	commits	comments
muhmuhten	PRs	branches	commits	comments
orzen	PRs	branches	commits	comments
pkoppstein	PRs	branches	commits	comments
stagrlee	PRs	branches	commits	comments
stroan	PRs	branches	commits	comments
tst2005	PRs	branches	commits	comments
wader	PRs	branches	commits	comments
wtlangford	PRs	branches	commits	comments

To recreate, starting w/ a list of @ mentions, use this regex: ^@(.+)$ ➝ | [$1](https://github.com/$1) | [PRs](https://github.com/stedolan/jq/pulls?q=is%3Apr+author%3A$1) | [branches](https://github.com/$1/jq/branches) | [commits](https://github.com/stedolan/jq/commits?author=$1) | [comments](https://github.com/stedolan/jq/issues?q=commenter%3A$1) |

[nicowilliams]

Thank you for that list @mrienstra! I'll review it tomorrow and invite some of them.

[nicowilliams]

Also, is there a PR for making the Windows build work again? And one for GitHub Actions?

[pkoppstein]

@nicowilliams wrote:

is there a PR for making the Windows build work again? And one for GitHub Actions?

This one seems relevant:

#2250 AppVeyor continuous integration tests all fail with GPG errors

[wader]

@nicowilliams Hey, nice to hear from you, hope all i well. My vote is for @itchyny also if he is up for it, and based on history maybe it's good to have more then one new maintainer :) i'm up for helping out either as maintainer or contributor. I'm not very familiar with the jq source code but i have done lots of C previously and feel motivated working on it.

[mrienstra]

Quick aside, I appreciate the process @liquidaty is championing in this thread, and I don't want to derail that train of thought. 🚂❤️

[owenthereal]

Therefore, @owenthereal and I set up https://github.com/jqlang/jqlang a while back in preparation for forking some day.

👋 Hello everyone!

I'm absolutely thrilled to see the move to https://github.com/jqlang finally happening 🚀. I can't wait to collaborate with all of you in maintaining and further developing jq!

A little about me: I'm the brains behind https://github.com/owenthereal/jqplay, the interactive playground that's linked directly from the jq documentation page 📚. Excited to bring my experience and passion for jq into this new venture with you all! 💻🔧🌟

[liquidaty]

I don't have any opinion on where the future fork is, and I'm happy to be a core member, or just a helping member, or not a member at all (in fact, this last would be my preference if the objective can be met without me!)-- really, I just want, for my selfish reasons, for the good work that many people have done on jq, as well as at least one of the updates that I've submitted and that is necessary for my purposes, to be available to everyone in a current and centralized single repo/fork so we don't have to all be using our separate forks.

So if the desired location of the fork is jqlang, that's fine with me.

That said, my prior question, which IMHO addresses the number 1 obstacle at this time ("who is up for being part of (or at least considering being part of) the core team? and/or just being part of the team, but non-core?") is still in want of a direct answer from anyone. Though, perhaps the latest comments and enthusiasm are an implicit volunteering to at least be a non-core contributor?

In any case, whatever team wants to maintain this, wherever the fork is housed, will still need to deal with figuring out what are the goals, what is the process and what is the team. On that last question, I would suggest shifting the focus from "who is sufficiently skilled to be a maintainer" to "what do we need to do" followed by "who can help and who is willing to help do those things".

So to try to put some specificity around "what we need to do" next, here is something to start with:

1. Agree on a set of releases. Let's start with those that @pkoppstein proposed on 3/7
2. For each release, agree broadly on a universe of issues/PRs applicable to each
3. Agree on a level of commitment. I would suggest we expect each person to spend an hour or two about once every 2 months. That might sound like a small number, but I'm trying to start with something sustainable assuming we are all busy professionals with personal lives. I would further suggest that we ask for the review team members to commit to this role in 1-year increments. This will make it easier for people to volunteer to help, because the commitment expectation is fixed and reasonably low. At the end of that time, each review member can decide to stay, or should make at least an effort to find a replacement
4. Agree on a review process. I would suggest:
   • Each PR must have an associated automated test. Someone on the core team will need to review the PR to make sure this condition is met
   • Each PR must pass all automated tests. We should have a git hook for this.
   • Agree on what platforms will be supported; all PRs must pass all automated tests on all platforms. Suggest Linux, Win, MacOS, Emscripten. Intel only (can add ARM/M1/M2 when github runners are available for them)
   • Exceptions can be allowed by "group decision" (whatever that is defined to be: unanimous, etc)
5. Agree on a review cadence. I would suggest starting with something that is easy for everyone to commit to, such as 1 PR every half month. If that is too slow, it's easy to speed it up later. If we start too fast, we could easily end up back to where we are today.

• That means that as a group we are reviewing something every 2 weeks
• Assuming we need at least 3 people to OK any PR, we need at least 12 people on the review team

6. Agree on what "group decision" means, and what happens if it breaks down

• does everyone need to review all the code changes and all the QC tests? Or do we specialize? Or rotate? Suggest unanimous but with specialization for actual review. i.e. one person reviews QC tests. One person reviews the actual code. Exceptions for large PRs that have a lot to review. Rotate duties. Votes can be "for", "against", or "abstain". If anyone votes "against", that is binding unless 4 others vote "for" and no others vote "against".
• unanimous, majority, depends?
• what is the basic timeline on a per-PR basis? e.g. at start of the month, 2 PRs are chosen, one voted on by 15th, the other by end of month?
• what if someone whose vote a decision requires is unresponsive for 2 weeks? 2 months? 1 year? what if someone's contributions don't seem to be helping? For whatever reason, I would suggest that someone can be forcibly removed from the team if 5 other members agree that doing so is in the best interests of the jq community

7. Get the fork ready. This includes making sure the automated QC tests are working for all target platforms. I am not sure this is already properly working today; if not, we will need some help fixing this as a first order of business for the new fork

So: how do those next steps sound, and if within reason (assuming some level of flexibility to refine), who would be willing to help, and which specific items above would you be willing to help with?

Again, I'm not trying to insert myself if I'm not needed. I'm just trying to be a catalyst to us being able to have a current and single jq repo, and offering to help if I can, but more than happy to leave this in more capable hands than mine if that goal can be met without my participation.

[liquidaty]

Hi everyone, thanks for your comments and feedback.

Thus far it doesn't look like we have sufficient (imho) momentum for a "core" team, so I will plan to create a new fork this weekend for the benefit of anyone who would like to use it. If the preferred location for that fork is jqlang and the owners want to make me an admin there, I'm happy to discuss that possibility, otherwise I'll choose a different location and notify this thread.

The first order of business will be to get CI/CD running smoothly, for win, linux and MacOS, after which time I will start looking at releases PRs / etc. If anyone would like to join me or has other comments before then, feel free to post here and I will check back in around the end of the week.

[nicowilliams]

@mrienstra: thanks for that list! I've invited @muhmuhten (accepted), @itchyny, @leonid-s-usov, @dtolnay (though I suspect he's quite busy with Rust), @wtlangford (though he is not interested in continuing as maintainer), @stedolan (as owner, of course), and I'll look more later.

[nicowilliams]

@liquidaty I don't think we need much governance unless there's disagreement.

My preference is that maintainers take any obvious PRs and that they seek consensus among themselves in case of controversy.

[nicowilliams]

Also, the way we did things when we had multiple active maintainers is that the maintainers reviewed each others' PRs and we needed each others' approval to push those, but as for PRs by non-maintainers we took the trivial ones w/o debate and discussed the ones that needed to be discussed.

[nicowilliams]

A bit of wisdom for new maintainers:

• don't add any more non-function-like language elements like try/catch -- try/catch should have been a function try(exp; catch) and try(exp; catch; finally)

• ditto break

• don't add any more polymorphism -- polymorphism in a dynamically-typed language creates problems

• do not add new types unless those are essentially sub-types of JSON types

  • for example, a "binary data" type should be indistinguishable from an array of small integers 0-255

  (this was something I discussed with @stedolan and he was quite adamant about this, and now so am I).

[chickenandpork]

I’m still interested in helping; I’m an SRE who enjoys CI/CD challenges.

I have seen statements of interest, but it seems like there’s a plan now :)

[pkoppstein]

@chickenandpork wrote:

I’m still interested in helping

Great. In case you didn't notice, @nicowilliams asked in this thread:

is there a PR for making the Windows build work again?

If you could offer a PR addressing the issue(*), it might even be incorporated into the stedolan version!

Thanks.

(*) Presumably #2250 (AppVeyor continuous integration tests all fail with GPG errors)

[liquidaty]

Seems like as a start it might be easier to remove the dependency on appveyor, since it adds another layer of applications, logins, authentications, accounts, etc. Is there anything that appveyor provides now, that isn't as easily provided by github ci/cd?

[itchyny]

I would like to join the communication space of maintainers if exists (Slack or Discord?).

[pkoppstein]

@liquidaty wrote:

Is there anything that appveyor provides now [?]

See https://github.com/stedolan/jq/wiki/Installation#windows-using-appveyor

[liquidaty]

@pkoppstein thank you. So is Appveyor used solely for building windows executables? If so, that is extremely easy to replace with a github action that runs mingw64 on vanilla linux. I'd be happy to submit a PR for that, which would kill 2 birds in one stone (fix windows build and remove a "heavy" and otherwise unnecessary dependency).

here's an example of a github action that builds a win version of an executable (and builds and installs a custom jq library!) at https://github.com/liquidaty/zsv via linux+mingw64 as specified in https://github.com/liquidaty/zsv/blob/main/.github/workflows/ci.yml.

[pkoppstein]

@liquidaty - From my point of view, the main significance of Appveyor was that it made it easy for Windows users to snarf any recent "snapshot" of jq of interest, not just the official versions.

[wader]

I would like to join the communication space of maintainers if exists (Slack or Discord?).

Me too

[liquidaty]

OK I've spent a lot more time than I had planned on looking into this appveyor issue and here are my conclusions:

1. the existing appveyor.xml is out of date. It tries to download non-existent files and apply out of date workarounds etc
2. I tried many fixes from various threads and thus far don't see any reasonable way to fix the broken issue, and I might not be alone in that view
3. The whole appveyor issue, and any solution, seem entirely arbitrary and fragile anyway, and offer a very narrow benefit (in theory-- not today since it is broken) for which better and more maintainable alternatives exist
4. This is a fundamental issue that needs to be fixed for anything to progress in a smooth manner. CI/CD with 99% pass and 1% fail is still of very little value
5. A better alternative, which even the MSYS developers themselves suggest to @nicowilliams ' question is: "use GitHub Actions instead". It also isn't hard for GitHub actions to be set up to provide access to nightly build artifacts

I also think, it is taking a disporportionate amount of time and attention write all these messages and discussions and attempt to get consensus, vs just fixing the issues. So, I have set up my own repo and am going to remove appveyor and do whatever other changes are needed for github actions to a minimal level of CI/CD. Until those basic changes are done, I am going to keep the repo private, because it's going to be in a broken state and I don't want anyone to try to use it and get turned off by that. Once CI/CD is working in that private repo, I will make it public and then begin porting the obvious and non-controversial PRs into that repo. If and when any of the maintainers of this repo or jqlang would like to take those changes, they are welcome to do so (or make me a maintainer and I'll do it directly).

If anyone would like to help with that, feel free to let me know.

I would like to join the communication space of maintainers if exists (Slack or Discord?).

Me too

[nicowilliams]

If you set up a slack or discord, I'll join it. EDIT: Owen will set it up and schedule it.

The other thing I need to do is set up a mirror from jqlang/jq to stedolan/jq and update the README to note that we're moving issues/PRs/dev activity to jqlang/jq but that stedolan/jq will remain canonical as long as there is one maintainer left with push rights for stedolan/jq.

[owenthereal]

👋 Feel free to join this discord 🎉 : https://discord.gg/yg6yjNmgAC. Please name your server nickname the same as your GitHub username so that we know who we are talking to 😄

[pkoppstein]

@nicowilliams - Thanks for your continuing efforts. I would like to ask a big favor that I'm hoping would not take too much of your time - to tag the current version with an "official" release number (e.g. 1.6.1). It would make a huge difference in several respects. I won't go into the details as I'm sure you can easily surmise them. (Of course, if you have time to sneak in any additional enhancements before creating 1.6.1, so much the better :-)

If you are don't have time to create a new version yourself, is there any chance that someone else could do so?

[wader]

A bit of wisdom for new maintainers:

Thanks for sharing. I got a bit curious about your wisdoms as i've tried (and failed) various ways to extend jq while working on https://github.com/wader/fq

• don't add any more non-function-like language elements like try/catch -- try/catch should have been a function try(exp; catch) and try(exp; catch; finally)

After writing lot of jq code i think i'm quite satisfied what it currently has. But if i could dream there are some things i've thought about that might be neat/useful:

• A bit more flexible destructing, ex rest of array and object (the ... in javascript)
• Object literal that can collect similar to array. Was some talk about it in Construct literal objects with subsequences of key-value pairs #2529
• A match syntax that can combine conditional and destructing
• Function references somehow.... but no idea how that would work :)

But as i said, i'm very pleased with current jq.

• ditto break
• don't add any more polymorphism -- polymorphism in a dynamically-typed language creates problems

By polymorphism you mean additional types or how different type interact with each other? both?

• do not add new types unless those are essentially sub-types of JSON types

  • for example, a "binary data" type should be indistinguishable from an array of small integers 0-255

  (this was something I discussed with @stedolan and he was quite adamant about this, and now so am I).

For fq i've added a binary type that can have both byte or bit units. First I naively exposed the type the to "standard jq" but it turned out be a bad idea, lots of weird behaviors happened (ex type/0 returned "binary"). This was later changed so the the type was indistinguishable to a string unless you use some new functions the know about the duality.

If I get some motivation and time it would be interesting to try the binary as array of ints idea for fq. It already kind of works like that if you use explode/0 or tobytes/0 (turns things into binary).

So I agree, adding a new type that is not JSON is probably a bad idea.

[nicowilliams]

@wader

By polymorphism you mean additional types or how different type interact with each other? both?

I mean adding behaviors to operators that work on multiple input types.

For fq i've added a binary type that can have both byte or bit units.

JSON only has the types that jq has, but we can have a binary type that is represented in jv/jq as an array of small unsigned integers (0..255) and internally use a byte array to represent it. Then we can have tobinary and toutf8 functions to convert from strings to binary and vice-versa, and we can have tobinary also accept arrays of numbers, and we can have a toarray to convert binary values that wouldn't allow setting/appending values that are not small integers (which would otherwise raise an error if the array was a binary). As well we'd have hex and base32 and base64 codecs. Maintaining the fiction that binary is an array of small integers is important when it comes to doing JSON I/O.

[...] lots of weird behaviors happened (ex type/0 returned "binary").

Exactly!

Binary has to be a "sub-type" of array (or string, but I think really of array). If a UTF-8 string is converted to binary, then that could be treated as a sub-type of string as long as you don't set or append elements that render it non-UTF-8, but checking that every time would be annoying, so I think it's best to go with binary as a sub-type of array.

[Farmbuyer]

@liquidaty If you're still looking for bugfix PRs, #1588 and #2116 are the same bug with trivially easy test cases (an error reporting routine is asserting and crashing).

[chickenandpork]

@chickenandpork wrote:

I’m still interested in helping

Great. In case you didn't notice, @nicowilliams asked in this thread:

is there a PR for making the Windows build work again?

If you could offer a PR addressing the issue(*), it might even be incorporated into the stedolan version!

Thanks.

(*) Presumably #2250 (AppVeyor continuous integration tests all fail with GPG errors)

@pkoppstein I hadn't noticed the response back in May; I haven't owned a windows box for likely 19 years, so I'm the opposite of helpful with windows. Now, bazel, that's got my interest, and it might re-enable some Windows compatibility. I cut a quick message into #general on the discord.

[itchyny]

Since jq 1.7 has been released, I'm closing this issue.