Please, untie HNix from Nix submodule

[Anton-Latukha]

Someone, who created the project 😉, said that there is not to much work to completely untie the HNix from Nix submodule.

It is said by the creator and observed by us that HNix mostly relies on use of the nix-instantiate functions of the Nix.

The solution would allow the project a new degree of freedom of progress, and would introduce the stability for our project in all kinds of software platforms.

---

Someone who would do the work may for effectivity during the analysis phase - look into just lazily update a submodule, maybe we can gain some information of Nix progress/changes from updating the submodule and seeing the test of the HNix-Nix behavior in that one move. Person would also may observe the additional points of Nix submodule use in the project.

---

P.S.

Closing this Report also would close Please, use modern Nix submodule, almost closes Please, fix HNix inside Nixpkgs, and allows the work towards clean CI: Test HNix Nixpkgs integration.

[sjakobi]

Someone, who created the project , said that there is not to much work to completely untie the HNix from Nix submodule.

It is said by the creator and observed by us that HNix mostly relies on use of the nix-instantiate functions of the Nix.

By "untying", do you mean that we should remove the submodule?

Note that, AFAICT, hnix uses the submodule only to access the test files in data/nix/tests/lang. Is there something wrong with that?

[sjakobi]

Note that, AFAICT, hnix uses the submodule only to access the test files in data/nix/tests/lang. Is there something wrong with that?

…apart from these files currently being outdated (#610), I mean.

[Anton-Latukha]

Well, what I know is: #641 (comment), that HNix also used nix-instantiate (main tool from Nix package) in tests the last time I checked it.

When there would be used only the tests - we would solve that easily.

[sjakobi]

nix-instantiate is not installed from the submodule though – it needs to be provided by the environment.

[Anton-Latukha]

After #672 I have all of the machinery ready (just made it yesterday) to slash the Submodule itself properly and merge it into our project, so we would preserve all author rights and follow the LICENSE agreement fully, and also would be free from the submodule itself.

Since I just hacked things yesterday I want to test it on one more case, before sharing, but here is the main part:

The main secres sauce of it - is this command nicely abstracted into a Fish shell script from https://github.com/Anton-Latukha/fish-functions.

function git-pass-filter

set passFilter ''

# Form a `-v pattern` for `rg`
for pattern in $argv

  set passFilter "$passFilter|$pattern"

end

# Remove the beginning |, no need to include the empty pattren in rg
set passFilter (string trim --chars='|' "$passFilter")

echo "
PassFilter: $passFilter
"

git filter-branch --prune-empty --index-filter \
  'git ls-tree -z -r --name-only --full-tree $GIT_COMMIT | \
  rg --null-data -v \''"$passFilter"'\' | \
  xargs -0 -r git rm --cached -r' -- --all


end

And then just:

git-pass-filter 'list' 'of' 'files' 'and' 'pa.*tterns'

[sjakobi]

@Anton-Latukha #672 is intriguing since it reveals that my initial stance that we only use the tests/lang files is wrong. I'm not sure when I'll make more progress on it though, or where that progress will eventually lead.

So you definitely don't have to wait for #672.

I still do not understand why you want to remove the submodule, and what problem you intend to solve by copying files into hnix proper. Could you explain your motivation a bit?

[Anton-Latukha]

For example.

User comes, and takes sources. People do Git clone automatically, and then complain about that they can not do something with the project. History of HNix already holds several of those Reports when people just did not know that projects with submodules exist, or how to get them, and/or did not read the headers of the README.

A submodule is always a structural point of failure.

We and all other maintainers on all software distribution platforms we would be present - would need to drag a submodule everywhere to be working.

Such things as submodule seem so simple, but so often in software stores and their specifics - can/would be the cause of the structural problems.

Also just the thing that HNix depends on Nix is just funny for everybody. If not the leverage for justification of maturity arguments.

There are several factors. If we merge the files into subdir:

1. We would be able just forget we had submodule.
2. Submodule - means problems with it - point of structural failure for users/devs/software stores.
3. We would preserve all rights properly and only have, use, bundle and ship the code properly, we do not drag/demand all Nix with us - we took its tests because we needed them for compatibility with it.
4. We would have much better time with the Nixpkgs/NixOS core members. If we use Nix - people would not get into the details when they want and can blame right of the gates, and point to that project uses Nix, so discredit the point of HNix existence. Nix community can be quite hostile and toxic, especially towards us. If we do not use submodule and not depend on Nix, and use only tests that put into subdir and we preserved all rights of authors - then our work would be rightfully justified, everything is done honestly and whatever happens - we can respect ourselves and the attitude towards obviously honest work would be quite different by many people. Democracy is a tyranny of the majority opinion. And the sway of even a couple of percent of the audience can mean all the difference between fail and success of the processes.
5. It would be much more simpler for us, especially, in the long run.

[sjakobi]

I agree that submodules can be challenging for people who aren't used to dealing with them. Since the submodule is only used for the testsuite, I think this affects only contributors though.

To me, the benefit of accessing the tests via a submodule vs. copying them into our source tree is that we can rely on the upstream VCS. Updating the tests should be easier, but also accessing any info regarding changes to the tests.

Also, lots of other projects use submodules too. The workflow is very well documented. I think that by switching to a home-grown alternative, we would probably make the project less accessible for new contributors, and increase the maintenance and documentation burden for ourselves.

A submodule is always a structural point of failure.

What do you mean here?

Also just the thing that HNix depends on Nix is just funny for everybody. If not the leverage for justification of maturity arguments.

Who thinks it's funny? I don't think it's funny. To me it seems completely normal that a new language implementation uses the testsuite of the reference implementation.

2. Submodule - means problems with it - point of structural failure for users/devs/software stores.

How does the submodule affect users or software stores?! Have there been any reports?

3. We would preserve all rights properly and only have, use, bundle and ship the code properly, we do not drag/demand all Nix with us - we took its tests because we needed them for compatibility with it.

What do you mean with "preserve all right properly"? Why can't we do this while we have the submodule?

4. We would have much better time with the Nixpkgs/NixOS core members. If we use Nix - people would not get into the details when they want and can blame right of the gates, and point to that project uses Nix, so discredit the point of HNix existence. Nix community can be quite hostile and toxic, especially towards us. If we do not use submodule and not depend on Nix, and use only tests that put into subdir and we preserved all rights of authors - then our work would be rightfully justified, everything is done honestly and whatever happens - we can respect ourselves and the attitude towards obviously honest work would be quite different by many people. Democracy is a tyranny of the majority opinion. And the sway of even a couple of percent of the audience can mean all the difference between fail and success of the processes.

I wasn't aware of any hostility towards hnix. What happened? Can you show me some links?

I also don't quite understand how copying files instead of accessing them via the submodule would change things. hnix would still rely on parts of the nix code base, just as it does today.

5. It would be much more simpler for us, especially, in the long run.

I think that especially in the long run, we'll want to update the language tests more frequently. I think the submodule will come in very handy then.

[Anton-Latukha]

Also just the thing that HNix depends on Nix is just funny for everybody. If not the leverage for justification of maturity arguments.

Who thinks it's funny? I don't think it's funny. To me it seems completely normal that a new language implementation uses the testsuite of the reference implementation.

Well, you have too much a professional view on it. The meritocracy view is: I arrived to look at HNix - seen has Nix as submodule - which means that project internally uses Nix, so why HNix exists.

We are in a tricky situation to Nix community, so better be very careful with positioning and representation of HNix.

It is also contrary to Nixpkgs practices, static deps in it is a bad taste, as also using submodule on other project.

If the submodule is not really needed - why have it - HNix would have much better social positioning without it. Since there would be Nix/HNix rivalry we can not avoid at some point - better to not depend on Nix so that argument would be populistically abused and advertised to third parties.

How does the submodule affect users or software stores?! Have there been any reports?

As I previously put, "Freedom 2: The freedom to redistribute and make copies so you can help your neighbour." It become much more abstract now - we do not know who where and where would package HNix, for example as Stack does not know: https://repology.org/project/haskell:stack/versions. So if we can free from Nix submodule for simplicity of shipment - we can.

3. We would preserve all rights properly and only have, use, bundle and ship the code properly, we do not drag/demand all Nix with us - we took its tests because we needed them for compatibility with it.

What do you mean with "preserve all right properly"? Why can't we do this while we have the submodule?

Well, again, critical community situation would prefer for us not to depend on Nix, or we would be pecked and ridiculed. You probably do not understand the magnitude of the question and rivalry and how strongly project strikes to too dear and too painful for them points, it is a core points of the whole community. They would arrive to us. I search a way to use what is needed, preserve all rights but do not depend on Nix explicitly, so when pecking starts - there is less things we can be accused of.

I also mean, that keeping all submodule - means attribution and using all authors work, from Nix community standpoint - exploiting BSD license, exploiting all of the project and not contributing back, and moreover - creating a weapon of destruction for the core Nix projects, submodule means use of it - exploitation of all the work done in Nix.

When in fact we can just quartally merge what is needed with its history and licence and keep attributions but also keep things more quite and implicit, merge-in only what is really needed. From community standpoint that would be much more fair work and fair use.

Please remember, most of the Nixpkgs community does not know Haskell (or even have good understanding of what "Purely functional lazy language" means paradigmically), majority of Nixpkgs community love Nix and do not understand the point of HNix existence except as a direct threat to the core - so they would judge superficially.

[sjakobi]

We are in a tricky situation to Nix community,

Can you expand on this?

If the submodule is not really needed - why have it

Some of the files in the submodule are required by the testsuite. The submodule is simply the best way to access them while keeping them easy to update.

HNix would have much better social positioning without it.

Frankly, I suspect that you're blowing the importance of the submodule way out of proportion. It would be helpful if other people who are familiar with the Nix community and its practices could weigh in on this point.

Since there would be Nix/HNix rivalry we can not avoid at some point - better to not depend on Nix so that argument would be populistically abused and advertised to third parties.

I have no idea whether your idea of a "rivalry" is likely to happen or not.

I think it's a very bad idea to let decisions about our internal setup be guided by paranoia about how these internals could be interpreted by misinformed outsiders. That just seems like a terrible distraction.

How does the submodule affect users or software stores?! Have there been any reports?

As I previously put, "Freedom 2: The freedom to redistribute and make copies so you can help your neighbour."

What previous comment are you referring to? Is this a quote?

It become much more abstract now - we do not know who where and where would package HNix, for example as Stack does not know: https://repology.org/project/haskell:stack/versions. So if we can free from Nix submodule for simplicity of shipment - we can.

I still don't understand the problem. How is this related to Stack?

---

You're really painting the Nix scene in some very dark colours here. Do you have anything to back up your claims about these negative perspectives on the hnix project?

[sjakobi]

Also, if the hnix project does have enemies, there's no point in trying to please them. They'd very likely quickly find something else to complain about. Let's rather focus on making hnix pleasant for existing and more inclined potential users [EDIT: and contributors too! :) ].

[Anton-Latukha]

Well, you get too in detail and too nitpicky about it.

The situation for us indeed - we are renegades to the Nix community. To not to say there is more.

I prefer not to post it all in public forum, I can tell it all privately, it is a sensitive topic for the community.

I prefer to make and post the automatic workflow on updating the needed files in the project, then to keep explicitly relying on Nix.