Skip to content
This repository has been archived by the owner on Sep 29, 2020. It is now read-only.

Test action megathread (pull request) #2

Closed
wants to merge 1 commit into from

Conversation

neverendingqs
Copy link
Owner

@neverendingqs neverendingqs commented Aug 22, 2020

Use this pull request to test the ask-so workflow.

Usage:

/so <query>

@neverendingqs neverendingqs changed the title chore: changes to create a test PR megathread. Test action megathread (pull request) Aug 22, 2020
@neverendingqs
Copy link
Owner Author

1

@neverendingqs
Copy link
Owner Author

2

@neverendingqs
Copy link
Owner Author

3

@neverendingqs
Copy link
Owner Author

4

@neverendingqs
Copy link
Owner Author

5

@github-actions
Copy link
Contributor

Received!

@neverendingqs
Copy link
Owner Author

6

@neverendingqs
Copy link
Owner Author

/so github

@github-actions
Copy link
Contributor

Received query 'github'!

@neverendingqs
Copy link
Owner Author

/so github actions

@github-actions
Copy link
Contributor

@neverendingqs
Copy link
Owner Author

/so github actions

@github-actions
Copy link
Contributor

Results for github actions

How to run GitHub Actions workflows locally?

Answers ### [By `iirekm` (Votes: 15)](https://stackoverflow.com/a/59988803)

There are tools like the already-mentioned act, but they are not perfect. You are not alone with this issue. Similar problems are:

  • how to test Jenkins builds locally
  • how to test GitLab CI builds locally
  • how to test Circle CI builds locally
  • how to test XXXX builds locally

And my solution for these problems is:

  • avoid functionalities provided by your CI tools (GitHub Actions, Gitlab CI, etc)
  • write as much as possible in CI-agnostic way (BASH scripts, PowerShell scripts, Gradle scripts, NPM scripts, Dockerfiles, Ansible scripts - anything you know)
  • invoke those scripts from your CI tool. In GitHub actions: run: your command to run
,### [By `Webber` (Votes: 9)](https://stackoverflow.com/a/60278805)

You can use nektos/act which supports yaml syntax since 0.2.0 (prerelease).

Check out their latest release.

,### [By `Jubair` (Votes: 3)](https://stackoverflow.com/a/59996378)

your best bet is https://github.com/nektos/act however it doesn't support yaml syntax yet, though there is a lot of interest aka: nektos/act#80 nektos/act#76 and nektos/act#74

Gitlab has gitlab-runner exec docker job-name but that's Gitlab :)

Github Actions Angular-cli

Answers ### [By `rams` (Votes: 2)](https://stackoverflow.com/a/59467447)

you seems to be new in Github Actions and deployments. With my experience, I assume you have reached the point of install Angular-CLI, due to ng not found issues occurred in action flow.

- uses: actions/checkout@v1
- name: Install Node
  uses: actions/setup-node@v1
     with:
        node-version: 12.8
- name: npm dependencies
   run: npm install     
- name: Build
   run: npm run build -- --prod

Fix Detail: Install Node first and then try npm install and npm build

Github Actions v4 API?

Answers ### [By `VonC` (Votes: 1)](https://stackoverflow.com/a/61430798)

Not yet: the GitHub Actions API announced last January 2020 is:

Note that there is a difference between:

For now, Actions are exposed through REST, not GraphQL

The request has been made of course:

As of now, the GitHub Actions API enables you to manage GitHub Actions only using the REST API.
I'll be sure to pass on your request for v4 support to the team for consideration.

(that was mid-April 2020)

@neverendingqs
Copy link
Owner Author

/so git rebase interactive

@github-actions
Copy link
Contributor

Results for git rebase interactive

How can I merge two commits into one if I already started rebase?

Question Text

I am trying to merge 2 commits into 1, so I followed “squashing commits with rebase” from git ready.

I ran

git rebase --interactive HEAD~2

In the resulting editor, I change pick to squash and then save-quit, but the rebase fails with the error

Cannot 'squash' without a previous commit

Now that my work tree has reached this state, I’m having trouble recovering.

The command git rebase --interactive HEAD~2 fails with:

Interactive rebase already started

and git rebase --continue fails with

Cannot 'squash' without a previous commit

Answers

By Greg Bacon (Votes: 1760)

Summary

The error message

Cannot 'squash' without a previous commit

means you likely attempted to “squash downward.” Git always squashes a newer commit into an older commit or “upward” as viewed on the interactive rebase todo list, that is into a commit on a previous line. Changing the command on your todo list’s very first line to squash will always produce this error as there is nothing for the first commit to squash into.

The Fix

First get back to where you started with

$ git rebase --abort

Say your history is

$ git log --pretty=oneline
a931ac7c808e2471b22b5bd20f0cad046b1c5d0d c
b76d157d507e819d7511132bdb5a80dd421d854f b
df239176e1a2ffac927d8b496ea00d5488481db5 a

That is, a was the first commit, then b, and finally c. After committing c we decide to squash b and c together:

(Note: Running git log pipes its output into a pager, less by default on most platforms. To quit the pager and return to your command prompt, press the q key.)

Running git rebase --interactive HEAD~2 gives you an editor with

pick b76d157 b
pick a931ac7 c

# Rebase df23917..a931ac7 onto df23917
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

(Notice that this todo list is in the reverse order as compared with the output of git log.)

Changing b’s pick to squash will result in the error you saw, but if instead you squash c into b (newer commit into the older or “squashing upward”) by changing the todo list to

pick   b76d157 b
squash a931ac7 c

and save-quitting your editor, you'll get another editor whose contents are

# This is a combination of 2 commits.
# The first commit's message is:

b

# This is the 2nd commit message:

c

When you save and quit, the contents of the edited file become commit message of the new combined commit:

$ git log --pretty=oneline
18fd73d3ce748f2a58d1b566c03dd9dafe0b6b4f b and c
df239176e1a2ffac927d8b496ea00d5488481db5 a

Note About Rewriting History

Interactive rebase rewrites history. Attempting to push to a remote that contains the old history will fail because it is not a fast-forward.

If the branch you rebased is a topic or feature branch in which you are working by yourself, no big deal. Pushing to another repository will require the --force option, or alternatively you may be able, depending on the remote repository’s permissions, to first delete the old branch and then push the rebased version. Examples of those commands that will potentially destroy work is outside the scope of this answer.

Rewriting already-published history on a branch in which you are working with other people without very good reason such as leaking a password or other sensitive details forces work onto your collaborators and is antisocial and will annoy other developers. The “Recovering From an Upstream Rebase” section in the git rebase documentation explains, with added emphasis.

Rebasing (or any other form of rewriting) a branch that others have based work on is a bad idea: anyone downstream of it is forced to manually fix their history. This section explains how to do the fix from the downstream’s point of view. The real fix, however, would be to avoid rebasing the upstream in the first place.

By user3828059 (Votes: 418)

If there are multiple commits, you can use git rebase -i to squash two commits into one.

If there are only two commits you want to merge, and they are the "most recent two", the following commands can be used to combine the two commits into one:

git reset --soft "HEAD^"
git commit --amend

By pambda (Votes: 119)

Rebase: You Ain't Gonna Need It:

A simpler way for most frequent scenario.

In most cases:

Actually if all you want is just simply combine several recent commits into one but do not need drop, reword and other rebase work.

you can simply do:

git reset --soft "HEAD~n"
  • Assuming ~n is number of commits to softly un-commit (i.e. ~1, ~2,...)

Then, use following command to modify the commit message.

git commit --amend

which is pretty much the same as a long range of squash and one pick.

And it works for n commits but not just two commits as above answer prompted.

Git - How to fix "corrupted" interactive rebase?

Question Text

I managed to create a little mess in my local git repository. I was trying to fix a broken commit by using the following instructions. Before running the "git commit --amend" (and after the git rebase --interactive) I decided that my changes were incorrect and so I executed "git reset HEAD --hard". Not a good idea, I tell you.

Now the interactive rebase seems to be "stuck". Git shows the current branch as (|REBASE-m). Every command (cd .., ls, git rebase...) inside my repository gives the following error:

cat: .git/rebase-merge/head-name: No such file or directory

Here's how git rebase --abort looks like:

$ git rebase --abort
cat: c:/_work/project/src/git/.git/rebase-merge/quiet: No such file or directory
cat: c:/_work/project/src/git/.git/rebase-merge/head-name: No such file or directory
cat: c:/_work/project/src/git/.git/rebase-merge/orig-head: No such file or directory
HEAD is now at 4c737fb Revert "Modified file names"
rm: cannot remove `c:/_work/project/src/git/.git/rebase-merge/done': Permission denied
rm: cannot remove directory `c:/_work/project/src/git/.git/rebase-merge': Directory
not empty
cat: .git/rebase-merge/head-name: No such file or directory

Here's the result of git rebase --continue:

$ git rebase --continue
cat: c:/_work/project/src/git/.git/rebase-merge/prev_head: No such file or directory
cat: c:/_work/project/src/git/.git/rebase-merge/end: No such file or directory
cat: c:/_work/project/src/git/.git/rebase-merge/msgnum: No such file or directory
cat: c:/_work/project/src/git/.git/rebase-merge/onto: No such file or directory
cat: c:/_work/project/src/git/.git/rebase-merge/quiet: No such file or directory
prev_head must be defined
cat: .git/rebase-merge/head-name: No such file or directory

Any ideas? I would like to reset the situation back to the state it was before I started my well-thought rebase operation.

Here's how git log --oneline shows the situation:

4c737fb Revert "Modified file names"
247ac02 Modified file names
33141e6 Message modifications
10a4a04 Modified db script

And this is fine.

I'm using msysgit v1.7.0.2.

Answers

By Laura Slocum (Votes: 218)

I got stuck in this. I created the head-name file, and then I ran into another error saying it couldn't find the onto file, so I created that file. Then I got another error saying could not read '.git/rebase-apply/onto': No such file or directory.

So I looked at the git documentation for rebasing and found another command:

git rebase --quit

This set me back on my branch with no changes, and I could start my rebase over again, good as new.

By Martin Owen (Votes: 163)

It looks like Git tried to remove the .git/rebase-merge directory but wasn't able to remove it completely. Have you tried copying that folder away? Also copy away the .git/rebase-apply folder if that is present.

By user584317 (Votes: 90)

I had a similar problem due to a zombie vim.exe process. Killing it in Task Manager, followed by a git rebase --abort fixed it.

How to abort a git rebase from inside vim during interactive editing

Question Text

When I do an interactive rebase, e.g.

git rebase -i HEAD~3

the rebase interactive editor (vim in my case) opens to let me edit the commits to rebase

pick c843ea2 Set Vim column limit to 80 (OS X)
pick fc32eac Add Bash alias for `pbcopy` (OS X)
....

If I now decide that I want to abort the rebase and quit vim using :q the rebase starts anyway. I'm using git version 1.9.0.msysgit.0 on windows.

Sure I can just delete all pick lines, but it might be a lot to do if I rebase a longer history. Is there another way?

How can I quit the rebase interactive editor (vim) and abort the rebase?

Answers

By Telmo Costa (Votes: 258)

If you exit the editor with an error code, the rebase will be aborted.

To exit with an error code on vim, do

:cq

See here and vimdoc here.

By Steven Penny (Votes: 44)

Just delete all the lines that are not comments from the file. Then save it to the default supplied path and exit the editor.

As a result git will just do nothing in this rebase.

To delete all lines in vim you can use

:%d|x

Then save and quit.

delete all lines of file in Vim

How to exit the Vim editor?

VIM - multiple commands on same line

By CodeManX (Votes: 9)

If you're using Notepad++ on Windows for instance:

CtrlA to select everything, then Del or Backspace to delete and CtrlS save. Git will abort the rebase if the file is empty.

You may also hit CtrlC in the command prompt where git is running to stop the current rebase command. Then run git rebase --abort to revert it.

@neverendingqs
Copy link
Owner Author

/so git reset

@github-actions
Copy link
Contributor

Results for git reset

Reset local repository branch to be just like remote repository HEAD

Question

How do I reset my local branch to be just like the branch on the remote repository?

I did:

git reset --hard HEAD

But when I run a git status,

On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)
      modified:   java/com/mycompany/TestContacts.java
      modified:   java/com/mycompany/TestParser.java

Can you please tell me why I have these 'modified'? I haven't touched these files? If I did, I want to remove those.

Answers

By Dan Moulding (Votes: 6947)

Setting your branch to exactly match the remote branch can be done in two steps:

git fetch origin
git reset --hard origin/master

If you want to save your current branch's state before doing this (just in case), you can do:

git commit -a -m "Saving my work, just in case"
git branch my-saved-work

Now your work is saved on the branch "my-saved-work" in case you decide you want it back (or want to look at it later or diff it against your updated branch).

Note that the first example assumes that the remote repo's name is "origin" and that the branch named "master" in the remote repo matches the currently checked-out branch in your local repo.

BTW, this situation that you're in looks an awful lot like a common case where a push has been done into the currently checked out branch of a non-bare repository. Did you recently push into your local repo? If not, then no worries -- something else must have caused these files to unexpectedly end up modified. Otherwise, you should be aware that it's not recommended to push into a non-bare repository (and not into the currently checked-out branch, in particular).

By Akavall (Votes: 446)

I needed to do (the solution in the accepted answer):

git fetch origin
git reset --hard origin/master

Followed by:

git clean -f

to remove local files

To see what files will be removed (without actually removing them):

git clean -n -f

By Acumenus (Votes: 323)

First, reset to the previously fetched HEAD of the corresponding upstream branch:

git reset --hard @{u}

The advantage of specifying @{u} or its verbose form @{upstream} is that the name of the remote repo and branch don't have to be explicitly specified. On Windows or with PowerShell, specify "@{u}" (with double quotes).

Next, as needed, remove untracked files, optionally also with -x:

git clean -df

Finally, as needed, get the latest changes:

git pull

Delete commits from a branch in Git

Question

I would like to know how to delete a commit.

By delete, I mean it is as if I didn't make that commit, and when I do a push in the future, my changes will not push to the remote branch.

I read git help, and I think the command I should use is git reset --hard HEAD. Is this correct?

Answers

By gahooa (Votes: 4216)

Careful: git reset --hard WILL DELETE YOUR WORKING DIRECTORY CHANGES. Be sure to stash any local changes you want to keep before running this command.

Assuming you are sitting on that commit, then this command will wack it...

git reset --hard HEAD~1

The HEAD~1 means the commit before head.

Or, you could look at the output of git log, find the commit id of the commit you want to back up to, and then do this:

git reset --hard <sha1-commit-id>

If you already pushed it, you will need to do a force push to get rid of it...

git push origin HEAD --force

However, if others may have pulled it, then you would be better off starting a new branch. Because when they pull, it will just merge it into their work, and you will get it pushed back up again.

If you already pushed, it may be better to use git revert, to create a "mirror image" commit that will undo the changes. However, both commits will be in the log.


FYI -- git reset --hard HEAD is great if you want to get rid of WORK IN PROGRESS. It will reset you back to the most recent commit, and erase all the changes in your working tree and index.


Lastly, if you need to find a commit that you "deleted", it is typically present in git reflog unless you have garbage collected your repository.

By Greg Hewgill (Votes: 717)

If you have not yet pushed the commit anywhere, you can use git rebase -i to remove that commit. First, find out how far back that commit is (approximately). Then do:

git rebase -i HEAD~N

The ~N means rebase the last N commits (N must be a number, for example HEAD~10). Then, you can edit the file that Git presents to you to delete the offending commit. On saving that file, Git will then rewrite all the following commits as if the one you deleted didn't exist.

The Git Book has a good section on rebasing with pictures and examples.

Be careful with this though, because if you change something that you have pushed elsewhere, another approach will be needed unless you are planning to do a force push.

By 1800 INFORMATION (Votes: 527)

Another possibility is one of my personal favorite commands:

git rebase -i <commit>~1

This will start the rebase in interactive mode -i at the point just before the commit you want to whack. The editor will start up listing all of the commits since then. Delete the line containing the commit you want to obliterate and save the file. Rebase will do the rest of the work, deleting only that commit, and replaying all of the others back into the log.

How to undo 'git reset'?

Question

What's the simplest way to undo the

git reset HEAD~

command? Currently, the only way I can think of is doing a "git clone http://..." from a remote repo.

Answers

By Mark Lodato (Votes: 2451)

Short answer:

git reset 'HEAD@{1}'

Long answer:

Git keeps a log of all ref updates (e.g., checkout, reset, commit, merge). You can view it by typing:

git reflog

Somewhere in this list is the commit that you lost. Let's say you just typed git reset HEAD~ and want to undo it. My reflog looks like this:

$ git reflog
3f6db14 HEAD@{0}: HEAD~: updating HEAD
d27924e HEAD@{1}: checkout: moving from d27924e0fe16776f0d0f1ee2933a0334a4787b4c
[...]

The first line says that HEAD 0 positions ago (in other words, the current position) is 3f6db14; it was obtained by resetting to HEAD~. The second line says that HEAD 1 position ago (in other words, the state before the reset) is d27924e. It was obtained by checking out a particular commit (though that's not important right now). So, to undo the reset, run git reset HEAD@{1} (or git reset d27924e).

If, on the other hand, you've run some other commands since then that update HEAD, the commit you want won't be at the top of the list, and you'll need to search through the reflog.

One final note: It may be easier to look at the reflog for the specific branch you want to un-reset, say master, rather than HEAD:

$ git reflog show master
c24138b master@{0}: merge origin/master: Fast-forward
90a2bf9 master@{1}: merge origin/master: Fast-forward
[...]

This should have less noise it in than the general HEAD reflog.

By Dan Fischer (Votes: 205)

Old question, and the posted answers work great. I'll chime in with another option though.

git reset ORIG_HEAD

ORIG_HEAD references the commit that HEAD previously referenced.

By zainengineer (Votes: 57)

My situation was slightly different, I did git reset HEAD~ three times.

To undo it I had to do

git reset HEAD@{3}

so you should be able to do

git reset HEAD@{N}

But if you have done git reset using

git reset HEAD~3

you will need to do

git reset HEAD@{1}

{N} represents the number of operations in reflog, as Mark pointed out in the comments.

@neverendingqs
Copy link
Owner Author

/so git one line graph

@github-actions
Copy link
Contributor

Results for git one line graph

Why does my git log graph grow one more line after I merge a branch each time?

Question

I'am used to use git log --oneline --graph --decorate --all as alias git ll to see the graph of commits in terminal

But a problem confuse me when every single time I merge my develop to master. The output of the command above may be like this:

* 0d1bf7b (HEAD -> master) Fix typo
*   f843224 Merge 'develop' to 'master'
|\
* | d673b76 (origin/master) Remove console.log for license information
* | 5080afc Remove all http url in production
* |   f28e74b Merge branch 'develop'
|\ \
* \ \   75c5b90 Merge branch 'develop'
|\ \ \
* \ \ \   ec189e6 Merge branch 'develop'
|\ \ \ \
* \ \ \ \   eb79c75 Merge branch 'develop'
|\ \ \ \ \
* \ \ \ \ \   74631ef Merge branch 'develop'
|\ \ \ \ \ \
| | | | | | | * f7a4155 (light) Fix typo
| | | | | | | *   1d6c411 Merge 'develop' to 'light'
| | | | | | | |\
| | | | | | | |/
| | | | | | |/|
| | | | | | * | 3715f47 (develop) Finish GroupCard in Setting page
| | | | | | | * e606e68 (origin/light) Remove console.log for license information
| | | | | | | * 676774c Remove all http url in production
| | | | | | | * c1bef16 Fix api url error

You can see there are too many lines generated after I merge develop to master. It is not a big problem for now, but it will become too many lines too obstruct me to see the commits someday.

So is there any thing I do wrong? Have you guys ever faced kind of this problem? How do you handle it?


[Editing on 2019/05/20]

Thank you guys. I appreciate your kind answers.

I want to fix my problem and make it clear a little bit. I use some GUI tools like Source tree and it show the git log as below. There are not many complicated lines with the same repository in this graph as you can see.

So is it possible if I want to show the graph like it in my command line interface?

git log in Source tree

Answers

By VonC (Votes: 2)

That is why squash and rebase do exists (for local commits of develop you have not pushed yet).
That would help keep the history linear, instead of git log showing you each develop merge in its separate track.

So is it possible if I want to show the graph like it in my command line interface?

In command-line, you can avoid all those extra lines by adding --no-merges:

git log --decorate --oneline --graph --no-merges --all --branches

By Rajni Kewlani (Votes: 0)

Of course you should use rebase and squash wherever applicable. Additionally, you can try following --no-merges option like:

git log --oneline --graph --decorate --all --no-merges

How to remove (squash) unnamed git branches

Question

I have a git repo with an absurdly complex tree. I want to simplify that tree so that all the unnamed branches will be a simple commit and just one line remains in the tree.

I have tried to:

  1. remove commits by interactive rebase
  2. squash all the commits in the unnamed branches

In both cases, I have issues with submodules conflicts. The repo over which I am working is this one: link.

For future reference, I leave here the present status of the git graph log:

*   9b0bc07 - (HEAD -> master, origin/master, origin/HEAD) Merge branch 'master' of https://framagit.org/sapo/ph.d.-project (16 hours ago) 
|\  
| * df5584b - Finished methods description and analysis (18 hours ago) 
| * 665d714 - Update comparison/README.md (20 hours ago) 
| * 6e31ac5 - Update comparison/README.md (5 days ago) 
| * b33094e - Update README.md (5 days ago) 
| *   336f5ad - Resolved merge (6 days ago) 
| |\  
| | * 1fba1c9 - Reorganized readme (7 days ago) 
| | * 12a313c - Still debugging, problem with polyphony (9 days ago) 
| | * 744ad4b - Still debugging runPlyAlignment (10 days ago) 
| | * 429fd13 - Added few modules. polyAlignment should work now, to be tested (13 days ago) 
| | *   25a83e6 - Finished broad_alingment, doing precise (13 days ago) 
| | |\  
| | | * 7a26990 - Update README.md (2 weeks ago) 
| | | * ccf7532 - Now tracking edited AMPACT code (3 weeks ago) 
| | | * 67a7d31 - Added configuration file for managing experiments (3 weeks ago) 
| | | * f2dafb9 - Error in Bach10 number of notes and loading sources (3 weeks ago) 
| | | * 16c4fe0 - Update comparison/README.md (4 weeks ago) 
| | | * 2983292 - updated datasets (4 weeks ago) 
| | | * 245f36e - Added symbolic link to utils (5 weeks ago) 
| | | * 202ee49 - Removed double submodule utils (5 weeks ago) 
| | | * 6a7ca53 - Updating  datasets (5 weeks ago) 
| | | * 0b25e03 - Trying to solve submodule issue (5 weeks ago) 
| | | * 96b4d90 - Moved utils to submodule (5 weeks ago) 
| | | * d4e7dc2 - Moved datasets to submodule (5 weeks ago) 
| | | * 667492c - Add LICENSE (5 weeks ago) 
| | | * 3882ad0 - Initial commit (5 weeks ago) 
| | * 253c3ea - Now tracking edited AMPACT code (3 weeks ago) 
| | * 022d483 - Added configuration file for managing experiments (3 weeks ago) 
| | * 557daba - Error in Bach10 number of notes and loading sources (3 weeks ago) 
| | * 434e3ae - Update comparison/README.md (4 weeks ago) 
| | * 66bc782 - updated datasets (4 weeks ago) 
| | * 1099294 - Added symbolic link to utils (5 weeks ago) 
| | * 7b43ae8 - Removed double submodule utils (5 weeks ago) 
| | * ae824ac - Updating  datasets (5 weeks ago) 
| | * 37dceed - Trying to solve submodule issue (5 weeks ago) 
| | * 2d83417 - Moved utils to submodule (5 weeks ago) 
| | * f11aa29 - Moved datasets to submodule (5 weeks ago) 
| | * 24a6b04 - Add LICENSE (5 weeks ago) 
| | * dd1e88d - Initial commit (5 weeks ago) 
| * a342f5e - Start tracking midiVelocity code and added dependencies for toyExampleRunScript (6 days ago) 
| * f2f8c48 - Still debugging, problem with polyphony (9 days ago) 
| * f7ede7a - Still debugging runPlyAlignment (10 days ago) 
| * 8f3b455 - Added few modules. polyAlignment should work now, to be tested (13 days ago) 
| *   415431e - Finished broad_alingment, doing precise (13 days ago) 
| |\  
| | * a6907a7 - Update README.md (2 weeks ago) 
| | * ab075bc - Now tracking edited AMPACT code (3 weeks ago) 
| | * 7f41e99 - Added configuration file for managing experiments (3 weeks ago) 
| | * cf925d6 - Error in Bach10 number of notes and loading sources (3 weeks ago) 
| | * eb2ffa4 - Update comparison/README.md (4 weeks ago) 
| | * a6b3992 - updated datasets (4 weeks ago) 
| | * 3f28358 - Added symbolic link to utils (5 weeks ago) 
| | * fb8c1db - Removed double submodule utils (5 weeks ago) 
| | * 99c6b86 - Updating  datasets (5 weeks ago) 
| | * 63646c3 - Trying to solve submodule issue (5 weeks ago) 
| | * 08cdad7 - Moved utils to submodule (5 weeks ago) 
| | * dc270c3 - Moved datasets to submodule (5 weeks ago) 
| | * 23a7cae - Add LICENSE (5 weeks ago) 
| | * 8aa959d - Initial commit (5 weeks ago) 
| * 95c1561 - Now tracking edited AMPACT code (3 weeks ago) 
| * 755842e - Added configuration file for managing experiments (3 weeks ago) 
| * 87ef237 - Error in Bach10 number of notes and loading sources (3 weeks ago) 
| * ad7de96 - Update comparison/README.md (4 weeks ago) 
| * e8306d8 - updated datasets (4 weeks ago) 
| * 6c56644 - Added symbolic link to utils (5 weeks ago) 
| * 83dc5e5 - Removed double submodule utils (5 weeks ago) 
| * 4d3480a - Updating  datasets (5 weeks ago) 
| * cd17adb - Trying to solve submodule issue (5 weeks ago) 
| * 8cee7fa - Moved utils to submodule (5 weeks ago) 
| * 51eb100 - Moved datasets to submodule (5 weeks ago) 
| * 654ad09 - Add LICENSE (5 weeks ago) 
| * ffe8f95 - Initial commit (5 weeks ago) 
* cde3680 - Added ewert-mueller-synch-method (16 hours ago) 
* 26ed69f - Finished methods description and analysis (18 hours ago) 
* de98e01 - Update comparison/README.md (20 hours ago) 
* 3a2f81c - Update comparison/README.md (5 days ago) 
* 1bbaf02 - Update README.md (5 days ago) 
*   08e1719 - Resolved merge (6 days ago) 
|\  
| * 3442cf3 - Reorganized readme (7 days ago) 
| * c9a382e - Still debugging, problem with polyphony (9 days ago) 
| * 6eeaa55 - Still debugging runPlyAlignment (10 days ago) 
| * 9ee5b09 - Added few modules. polyAlignment should work now, to be tested (13 days ago) 
| *   72cc527 - Finished broad_alingment, doing precise (13 days ago) 
| |\  
| | * 1fa24ce - Update README.md (2 weeks ago) 
| | * 65d8ec3 - Now tracking edited AMPACT code (3 weeks ago) 
| | * 90c2fcd - Added configuration file for managing experiments (3 weeks ago) 
| | * 0dfab99 - Error in Bach10 number of notes and loading sources (3 weeks ago) 
| | * 8adc63b - Update comparison/README.md (4 weeks ago) 
| | * 8f37d17 - updated datasets (4 weeks ago) 
| | * 2731ac6 - Added symbolic link to utils (5 weeks ago) 
| | * 7d3e966 - Removed double submodule utils (5 weeks ago) 
| | * 29486c3 - Updating  datasets (5 weeks ago) 
| | * 4da21a2 - Trying to solve submodule issue (5 weeks ago) 
| | * 113978b - Moved utils to submodule (5 weeks ago) 
| | * 147df61 - Moved datasets to submodule (5 weeks ago) 
| | * 0c49f6f - Add LICENSE (5 weeks ago) 
| | * b5edea7 - Initial commit (5 weeks ago) 
| * 647f169 - Now tracking edited AMPACT code (3 weeks ago) 
| * 06f03b8 - Added configuration file for managing experiments (3 weeks ago) 
| * d3da2bf - Error in Bach10 number of notes and loading sources (3 weeks ago) 
| * f2056cc - Update comparison/README.md (4 weeks ago) 
| * ed2d32b - updated datasets (4 weeks ago) 
| * 5f76af1 - Added symbolic link to utils (5 weeks ago) 
| * a36d228 - Removed double submodule utils (5 weeks ago) 
| * ff56e6c - Updating  datasets (5 weeks ago) 
| * a8113f6 - Trying to solve submodule issue (5 weeks ago) 
| * 439bbe6 - Moved utils to submodule (5 weeks ago) 
| * f1900ac - Moved datasets to submodule (5 weeks ago) 
| * 4b95c5c - Add LICENSE (5 weeks ago) 
| * 4452ced - Initial commit (5 weeks ago) 
* f6cf1b3 - Start tracking midiVelocity code and added dependencies for toyExampleRunScript (6 days ago) 
* bcc62dc - Still debugging, problem with polyphony (9 days ago) 
* 2e35a6e - Still debugging runPlyAlignment (10 days ago) 
* 671a59e - Added few modules. polyAlignment should work now, to be tested (13 days ago) 
*   84228aa - Finished broad_alingment, doing precise (13 days ago) 
|\  
| * 1e96218 - Update README.md (2 weeks ago) 
| * 04a420f - Now tracking edited AMPACT code (3 weeks ago) 
| * 6fbf6cf - Added configuration file for managing experiments (3 weeks ago) 
| * 9ba12b1 - Error in Bach10 number of notes and loading sources (3 weeks ago) 
| * 0652851 - Update comparison/README.md (4 weeks ago) 
| * a5d23af - updated datasets (4 weeks ago) 
| * 750e8bd - Added symbolic link to utils (5 weeks ago) 
| * 48f2509 - Removed double submodule utils (5 weeks ago) 
| * cf92067 - Updating  datasets (5 weeks ago) 
| * 79b804b - Trying to solve submodule issue (5 weeks ago) 
| * df419cb - Moved utils to submodule (5 weeks ago) 
| * 5f616a4 - Moved datasets to submodule (5 weeks ago) 
| * 9e07d9b - Add LICENSE (5 weeks ago) 
| * 04d3e95 - Initial commit (5 weeks ago) 
* 175279a - Now tracking edited AMPACT code (3 weeks ago) 
* 79b209f - Added configuration file for managing experiments (3 weeks ago) 
* 1489f41 - Error in Bach10 number of notes and loading sources (3 weeks ago) 
* 0cfdded - Update comparison/README.md (4 weeks ago) 
* 83c0006 - updated datasets (4 weeks ago) 
* d9ec598 - Added symbolic link to utils (5 weeks ago) 
* 00429a4 - Removed double submodule utils (5 weeks ago) 
* bd2c942 - Updating  datasets (5 weeks ago) 
* 88d7305 - Trying to solve submodule issue (5 weeks ago) 
* d3a0519 - Moved utils to submodule (5 weeks ago) 
* 3dcb36e - Moved datasets to submodule (5 weeks ago) 
* 84462c3 - Add LICENSE (5 weeks ago) 
* 5e49895 - Initial commit (5 weeks ago) 

Answers

By harmonica141 (Votes: 2)

Looking at your linked repository I see why the question is arising. A PhD thesis on git is a cool thing to do but something entirely different from the development of a software project. Nonetheless partly the same rules apply. Having a longer history provides you with backups. I strongly recommend not taking those away as you may regret it at some later point in time... (telling from experience...)

Git as a version tracking tool exists for the sole purpose of keeping history. So squashing commits simply to make the tree less complex is the equivalent of burning history books.

To reduce complexity on the repository and restore maintainability you may want to refactor in order to implement a consistent branching scheme. There are several out there for any purpose and complexity of project. Maybe you have a look around in the git documentation or over at the Bitbucket guys from Atlassian.

A very popular branching model though is git flow. The basic idea is to have a continuous development branch that you merge new features (each being developed on a separate branch) into and from which releases emerge into a stable branch. Each feature branch is closed after the feature is finished so the number of open branches is naturally limited. Basically in this workflow branches are used to sort code by its stability, starting from separate feature branches that can not even live on their own up to an ever trustworthy stable release branch. Please note that there are helper scripts to do the hard work for you in this workflow. It makes cl work really easy.

It also is important to recognize bad software. If there is a automated tool creating wild unnamed branches without order and permission this is definitely a sign of bad software. Start taking git responsibility yourself and use something you can control. If you do not want to use the command line for some reasons then at least make use of a good gui tool like Gitkraken or Git Tower.

Take some time to find a branching model that suits your needs and then refactor the repository to enforce it. This will help you keep track of the ongoing work.

By the way, what you show in the picture is far from complex. It is just getting lengthy. But as work continues on a project you will accumulate quite a number of commits. This will become more extreme the more developers work on the same repository. There is no need or reason to shorten history, only to enforce law and order to keep it comprehensible.

Git commit option to split branches in the network graph

Question

On my git network graph, I want to keep branches separated. However if I have a circumstance where I split my master to branch A, then make one or more commits to A, then merge it back to master, the graph shows a single line for both master and A, despite the fact that at least one commit point was not included.

In other words, my graph looks like this:

*------*------*------* (master/A)

And I want it to look like this:

*------*------*------* (master/A)
 \__________________/

I know there's an option either in commit or push to force this (and I did it once, ages ago), but for the life of me I can't find it.

Does anyone know the command to do this? And second, for extra credit, its location in Android Studio?

Thanks!

Answers

By Scott (Votes: 6)

As Mykola said in a comment, the answer is:

git merge --no-ff

The normal behavior of a git merge is to "fast forward" the base branch HEAD up to the place where you are on your new branch. This effectively erases your new branch (A in the example above). Specifying "no Fast Forward" adds a new commit and preserves the existence of Branch A for posterity.

Also see: http://www.relativesanity.com/articles/ffwd

Bonus Answer: In Android Studio, it's possible to use the no-ff option. However you cannot use the quick branch menu on the bottom right where you select the branch and click merge. You need to go the "long" way - From the top menu, select VCS/Git/Merge Changes. This will give a dialog box for the merge allowing you to set options, including "No fast forward"

@neverendingqs
Copy link
Owner Author

/so [git] reset

@github-actions
Copy link
Contributor

Results for [git] reset

How can I reset or revert a file to a specific revision?

Question

I have made some changes to a file which has been committed a few times as part of a group of files, but now want to reset/revert the changes on it back to a previous version.

I have done a git log along with a git diff to find the revision I need, but just have no idea how to get the file back to its former state in the past.

Answers

By Greg Hewgill (Votes: 6261)

Assuming the hash of the commit you want is c5f567:

git checkout c5f567 -- file1/to/restore file2/to/restore

The git checkout man page gives more information.

If you want to revert to the commit before c5f567, append ~1 (where 1 is the number of commits you want to go back, it can be anything):

git checkout c5f567~1 -- file1/to/restore file2/to/restore

As a side note, I've always been uncomfortable with this command because it's used for both ordinary things (changing between branches) and unusual, destructive things (discarding changes in the working directory).

By Chris Lloyd (Votes: 619)

You can quickly review the changes made to a file using the diff command:

git diff <commit hash> <filename>

Then to revert a specific file to that commit use the reset command:

git reset <commit hash> <filename>

You may need to use the --hard option if you have local modifications.

A good workflow for managaging waypoints is to use tags to cleanly mark points in your timeline. I can't quite understand your last sentence but what you may want is diverge a branch from a previous point in time. To do this, use the handy checkout command:

git checkout <commit hash>
git checkout -b <new branch name>

You can then rebase that against your mainline when you are ready to merge those changes:

git checkout <my branch>
git rebase master
git checkout master
git merge <my branch>

By foxxtrot (Votes: 369)

You can use any reference to a git commit, including the SHA-1 if that's most convenient. The point is that the command looks like this:

git checkout [commit-ref] -- [filename]

Reset local repository branch to be just like remote repository HEAD

Question

How do I reset my local branch to be just like the branch on the remote repository?

I did:

git reset --hard HEAD

But when I run a git status,

On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)
      modified:   java/com/mycompany/TestContacts.java
      modified:   java/com/mycompany/TestParser.java

Can you please tell me why I have these 'modified'? I haven't touched these files? If I did, I want to remove those.

Answers

By Dan Moulding (Votes: 6947)

Setting your branch to exactly match the remote branch can be done in two steps:

git fetch origin
git reset --hard origin/master

If you want to save your current branch's state before doing this (just in case), you can do:

git commit -a -m "Saving my work, just in case"
git branch my-saved-work

Now your work is saved on the branch "my-saved-work" in case you decide you want it back (or want to look at it later or diff it against your updated branch).

Note that the first example assumes that the remote repo's name is "origin" and that the branch named "master" in the remote repo matches the currently checked-out branch in your local repo.

BTW, this situation that you're in looks an awful lot like a common case where a push has been done into the currently checked out branch of a non-bare repository. Did you recently push into your local repo? If not, then no worries -- something else must have caused these files to unexpectedly end up modified. Otherwise, you should be aware that it's not recommended to push into a non-bare repository (and not into the currently checked-out branch, in particular).

By Akavall (Votes: 446)

I needed to do (the solution in the accepted answer):

git fetch origin
git reset --hard origin/master

Followed by:

git clean -f

to remove local files

To see what files will be removed (without actually removing them):

git clean -n -f

By Acumenus (Votes: 323)

First, reset to the previously fetched HEAD of the corresponding upstream branch:

git reset --hard @{u}

The advantage of specifying @{u} or its verbose form @{upstream} is that the name of the remote repo and branch don't have to be explicitly specified. On Windows or with PowerShell, specify "@{u}" (with double quotes).

Next, as needed, remove untracked files, optionally also with -x:

git clean -df

Finally, as needed, get the latest changes:

git pull

Move existing, uncommitted work to a new branch in Git

Question

I started some work on a new feature and after coding for a bit, I decided this feature should be on its own branch.

How do I move the existing uncommitted changes to a new branch and reset my current one?

I want to reset my current branch while preserving existing work on the new feature.

Answers

By knittl (Votes: 3717)

Use the following:

git checkout -b <new-branch>

This will leave your current branch as it is, create and checkout a new branch and keep all your changes. You can then stage changes in files to commit with:

git add <files>

and commit to your new branch with:

git commit -m "<Brief description of this commit>"

The changes in the working directory and changes staged in index do not belong to any branch yet. This changes the branch where those modifications would end in.

You don't reset your original branch, it stays as it is. The last commit on <old-branch> will still be the same. Therefore you checkout -b and then commit.


Update 2020 / Git 2.23

Git 2.23 adds the new switch subcommand in an attempt to clear some of the confusion that comes from the overloaded usage of checkout (switching branches, restoring files, detaching HEAD, etc.)

Starting with this version of Git, replace above's command with:

git switch -c <new-branch>

The behavior is identical and remains unchanged.

By Robin Qiu (Votes: 338)

Alternatively:

  1. Save current changes to a temp stash:

    $ git stash

  2. Create a new branch based on this stash, and switch to the new branch:

    $ git stash branch <new-branch> stash@{0}

Tip: use tab key to reduce typing the stash name.

By joeytwiddle (Votes: 50)

If you have been making commits on your main branch while you coded, but you now want to move those commits to a different branch, this is a quick way:

  1. Copy your current history onto a new branch, bringing along any uncommitted changes too:

     git checkout -b <new-feature-branch>
    
  2. Now force the original "messy" branch to roll back: (without switching to it)

     git branch -f <previous-branch> <earlier-commit-id>
    

    For example:

     git branch -f master origin/master
    

    or if you had made 4 commits:

     git branch -f master HEAD~4
    

Warning: git branch -f master origin/master will reset the tracking information for that branch. So if you have configured your master branch to push to somewhere other than origin/master then that configuration will be lost.

Warning: If you rebase after branching, there is a danger that some commits may be lost, which is described here. The only way to avoid that is to create a new history using cherry-pick. That link describes the safest fool-proof method, although less convenient. (If you have uncommitted changes, you may need to git stash at the start and git stash pop at the end.)

@neverendingqs
Copy link
Owner Author

/so reset

@github-actions
Copy link
Contributor

Results for reset

How can I reset or revert a file to a specific revision?

Question

I have made some changes to a file which has been committed a few times as part of a group of files, but now want to reset/revert the changes on it back to a previous version.

I have done a git log along with a git diff to find the revision I need, but just have no idea how to get the file back to its former state in the past.

Answers

By Greg Hewgill (Votes: 6261)

Assuming the hash of the commit you want is c5f567:

git checkout c5f567 -- file1/to/restore file2/to/restore

The git checkout man page gives more information.

If you want to revert to the commit before c5f567, append ~1 (where 1 is the number of commits you want to go back, it can be anything):

git checkout c5f567~1 -- file1/to/restore file2/to/restore

As a side note, I've always been uncomfortable with this command because it's used for both ordinary things (changing between branches) and unusual, destructive things (discarding changes in the working directory).

By Chris Lloyd (Votes: 619)

You can quickly review the changes made to a file using the diff command:

git diff <commit hash> <filename>

Then to revert a specific file to that commit use the reset command:

git reset <commit hash> <filename>

You may need to use the --hard option if you have local modifications.

A good workflow for managaging waypoints is to use tags to cleanly mark points in your timeline. I can't quite understand your last sentence but what you may want is diverge a branch from a previous point in time. To do this, use the handy checkout command:

git checkout <commit hash>
git checkout -b <new branch name>

You can then rebase that against your mainline when you are ready to merge those changes:

git checkout <my branch>
git rebase master
git checkout master
git merge <my branch>

By foxxtrot (Votes: 369)

You can use any reference to a git commit, including the SHA-1 if that's most convenient. The point is that the command looks like this:

git checkout [commit-ref] -- [filename]

Reset local repository branch to be just like remote repository HEAD

Question

How do I reset my local branch to be just like the branch on the remote repository?

I did:

git reset --hard HEAD

But when I run a git status,

On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)
      modified:   java/com/mycompany/TestContacts.java
      modified:   java/com/mycompany/TestParser.java

Can you please tell me why I have these 'modified'? I haven't touched these files? If I did, I want to remove those.

Answers

By Dan Moulding (Votes: 6947)

Setting your branch to exactly match the remote branch can be done in two steps:

git fetch origin
git reset --hard origin/master

If you want to save your current branch's state before doing this (just in case), you can do:

git commit -a -m "Saving my work, just in case"
git branch my-saved-work

Now your work is saved on the branch "my-saved-work" in case you decide you want it back (or want to look at it later or diff it against your updated branch).

Note that the first example assumes that the remote repo's name is "origin" and that the branch named "master" in the remote repo matches the currently checked-out branch in your local repo.

BTW, this situation that you're in looks an awful lot like a common case where a push has been done into the currently checked out branch of a non-bare repository. Did you recently push into your local repo? If not, then no worries -- something else must have caused these files to unexpectedly end up modified. Otherwise, you should be aware that it's not recommended to push into a non-bare repository (and not into the currently checked-out branch, in particular).

By Akavall (Votes: 446)

I needed to do (the solution in the accepted answer):

git fetch origin
git reset --hard origin/master

Followed by:

git clean -f

to remove local files

To see what files will be removed (without actually removing them):

git clean -n -f

By Acumenus (Votes: 323)

First, reset to the previously fetched HEAD of the corresponding upstream branch:

git reset --hard @{u}

The advantage of specifying @{u} or its verbose form @{upstream} is that the name of the remote repo and branch don't have to be explicitly specified. On Windows or with PowerShell, specify "@{u}" (with double quotes).

Next, as needed, remove untracked files, optionally also with -x:

git clean -df

Finally, as needed, get the latest changes:

git pull

Move existing, uncommitted work to a new branch in Git

Question

I started some work on a new feature and after coding for a bit, I decided this feature should be on its own branch.

How do I move the existing uncommitted changes to a new branch and reset my current one?

I want to reset my current branch while preserving existing work on the new feature.

Answers

By knittl (Votes: 3717)

Use the following:

git checkout -b <new-branch>

This will leave your current branch as it is, create and checkout a new branch and keep all your changes. You can then stage changes in files to commit with:

git add <files>

and commit to your new branch with:

git commit -m "<Brief description of this commit>"

The changes in the working directory and changes staged in index do not belong to any branch yet. This changes the branch where those modifications would end in.

You don't reset your original branch, it stays as it is. The last commit on <old-branch> will still be the same. Therefore you checkout -b and then commit.


Update 2020 / Git 2.23

Git 2.23 adds the new switch subcommand in an attempt to clear some of the confusion that comes from the overloaded usage of checkout (switching branches, restoring files, detaching HEAD, etc.)

Starting with this version of Git, replace above's command with:

git switch -c <new-branch>

The behavior is identical and remains unchanged.

By Robin Qiu (Votes: 338)

Alternatively:

  1. Save current changes to a temp stash:

    $ git stash

  2. Create a new branch based on this stash, and switch to the new branch:

    $ git stash branch <new-branch> stash@{0}

Tip: use tab key to reduce typing the stash name.

By joeytwiddle (Votes: 50)

If you have been making commits on your main branch while you coded, but you now want to move those commits to a different branch, this is a quick way:

  1. Copy your current history onto a new branch, bringing along any uncommitted changes too:

     git checkout -b <new-feature-branch>
    
  2. Now force the original "messy" branch to roll back: (without switching to it)

     git branch -f <previous-branch> <earlier-commit-id>
    

    For example:

     git branch -f master origin/master
    

    or if you had made 4 commits:

     git branch -f master HEAD~4
    

Warning: git branch -f master origin/master will reset the tracking information for that branch. So if you have configured your master branch to push to somewhere other than origin/master then that configuration will be lost.

Warning: If you rebase after branching, there is a danger that some commits may be lost, which is described here. The only way to avoid that is to create a new history using cherry-pick. That link describes the safest fool-proof method, although less convenient. (If you have uncommitted changes, you may need to git stash at the start and git stash pop at the end.)

@neverendingqs
Copy link
Owner Author

/so array

@github-actions
Copy link
Contributor

Results for array

Why is processing a sorted array faster than processing an unsorted array?

Question

Here is a piece of C++ code that shows some very peculiar behavior. For some strange reason, sorting the data miraculously makes the code almost six times faster:

#include <algorithm>
#include <ctime>
#include <iostream>

int main()
{
// Generate data
const unsigned arraySize = 32768;
int data[arraySize];

for (unsigned c = 0; c &lt; arraySize; ++c)
    data[c] = std::rand() % 256;

// !!! With this, the next loop runs faster.
std::sort(data, data + arraySize);

// Test
clock_t start = clock();
long long sum = 0;

for (unsigned i = 0; i &lt; 100000; ++i)
{
    // Primary loop
    for (unsigned c = 0; c &lt; arraySize; ++c)
    {
        if (data[c] &gt;= 128)
            sum += data[c];
    }
}

double elapsedTime = static_cast&lt;double&gt;(clock() - start) / CLOCKS_PER_SEC;

std::cout &lt;&lt; elapsedTime &lt;&lt; std::endl;
std::cout &lt;&lt; &quot;sum = &quot; &lt;&lt; sum &lt;&lt; std::endl;

}

  • Without std::sort(data, data + arraySize);, the code runs in 11.54 seconds.
  • With the sorted data, the code runs in 1.93 seconds.

Initially, I thought this might be just a language or compiler anomaly, so I tried Java:

import java.util.Arrays;
import java.util.Random;

public class Main
{
public static void main(String[] args)
{
// Generate data
int arraySize = 32768;
int data[] = new int[arraySize];

    Random rnd = new Random(0);
    for (int c = 0; c &lt; arraySize; ++c)
        data[c] = rnd.nextInt() % 256;

    // !!! With this, the next loop runs faster
    Arrays.sort(data);

    // Test
    long start = System.nanoTime();
    long sum = 0;

    for (int i = 0; i &lt; 100000; ++i)
    {
        // Primary loop
        for (int c = 0; c &lt; arraySize; ++c)
        {
            if (data[c] &gt;= 128)
                sum += data[c];
        }
    }

    System.out.println((System.nanoTime() - start) / 1000000000.0);
    System.out.println(&quot;sum = &quot; + sum);
}

}

With a similar but less extreme result.


My first thought was that sorting brings the data into the cache, but then I thought how silly that was because the array was just generated.

  • What is going on?
  • Why is processing a sorted array faster than processing an unsorted array?

The code is summing up some independent terms, so the order should not matter.

Answers

By Mysticial (Votes: 32157)

You are a victim of branch prediction fail.


What is Branch Prediction?

Consider a railroad junction:

Image showing a railroad junction Image by Mecanismo, via Wikimedia Commons. Used under the CC-By-SA 3.0 license.

Now for the sake of argument, suppose this is back in the 1800s - before long distance or radio communication.

You are the operator of a junction and you hear a train coming. You have no idea which way it is supposed to go. You stop the train to ask the driver which direction they want. And then you set the switch appropriately.

Trains are heavy and have a lot of inertia. So they take forever to start up and slow down.

Is there a better way? You guess which direction the train will go!

  • If you guessed right, it continues on.
  • If you guessed wrong, the captain will stop, back up, and yell at you to flip the switch. Then it can restart down the other path.

If you guess right every time, the train will never have to stop.
If you guess wrong too often, the train will spend a lot of time stopping, backing up, and restarting.


Consider an if-statement: At the processor level, it is a branch instruction:

Screenshot of compiled code containing an if statement

You are a processor and you see a branch. You have no idea which way it will go. What do you do? You halt execution and wait until the previous instructions are complete. Then you continue down the correct path.

Modern processors are complicated and have long pipelines. So they take forever to "warm up" and "slow down".

Is there a better way? You guess which direction the branch will go!

  • If you guessed right, you continue executing.
  • If you guessed wrong, you need to flush the pipeline and roll back to the branch. Then you can restart down the other path.

If you guess right every time, the execution will never have to stop.
If you guess wrong too often, you spend a lot of time stalling, rolling back, and restarting.


This is branch prediction. I admit it's not the best analogy since the train could just signal the direction with a flag. But in computers, the processor doesn't know which direction a branch will go until the last moment.

So how would you strategically guess to minimize the number of times that the train must back up and go down the other path? You look at the past history! If the train goes left 99% of the time, then you guess left. If it alternates, then you alternate your guesses. If it goes one way every three times, you guess the same...

In other words, you try to identify a pattern and follow it. This is more or less how branch predictors work.

Most applications have well-behaved branches. So modern branch predictors will typically achieve >90% hit rates. But when faced with unpredictable branches with no recognizable patterns, branch predictors are virtually useless.

Further reading: "Branch predictor" article on Wikipedia.


As hinted from above, the culprit is this if-statement:

if (data[c] >= 128)
    sum += data[c];

Notice that the data is evenly distributed between 0 and 255. When the data is sorted, roughly the first half of the iterations will not enter the if-statement. After that, they will all enter the if-statement.

This is very friendly to the branch predictor since the branch consecutively goes the same direction many times. Even a simple saturating counter will correctly predict the branch except for the few iterations after it switches direction.

Quick visualization:

T = branch taken
N = branch not taken

data[] = 0, 1, 2, 3, 4, ... 126, 127, 128, 129, 130, ... 250, 251, 252, ...
branch = N N N N N ... N N T T T ... T T T ...

   = NNNNNNNNNNNN ... NNNNNNNTTTTTTTTT ... TTTTTTTTTT  (easy to predict)

However, when the data is completely random, the branch predictor is rendered useless, because it can't predict random data. Thus there will probably be around 50% misprediction (no better than random guessing).

data[] = 226, 185, 125, 158, 198, 144, 217, 79, 202, 118,  14, 150, 177, 182, 133, ...
branch =   T,   T,   N,   T,   T,   T,   T,  N,   T,   N,   N,   T,   T,   T,   N  ...
   = TTNTTTTNTNNTTTN ...   (completely random - hard to predict)


So what can be done?

If the compiler isn't able to optimize the branch into a conditional move, you can try some hacks if you are willing to sacrifice readability for performance.

Replace:

if (data[c] >= 128)
    sum += data[c];

with:

int t = (data[c] - 128) >> 31;
sum += ~t & data[c];

This eliminates the branch and replaces it with some bitwise operations.

(Note that this hack is not strictly equivalent to the original if-statement. But in this case, it's valid for all the input values of data[].)

Benchmarks: Core i7 920 @ 3.5 GHz

C++ - Visual Studio 2010 - x64 Release

//  Branch - Random
seconds = 11.777

// Branch - Sorted
seconds = 2.352

// Branchless - Random
seconds = 2.564

// Branchless - Sorted
seconds = 2.587

Java - NetBeans 7.1.1 JDK 7 - x64

//  Branch - Random
seconds = 10.93293813

// Branch - Sorted
seconds = 5.643797077

// Branchless - Random
seconds = 3.113581453

// Branchless - Sorted
seconds = 3.186068823

Observations:

  • With the Branch: There is a huge difference between the sorted and unsorted data.
  • With the Hack: There is no difference between sorted and unsorted data.
  • In the C++ case, the hack is actually a tad slower than with the branch when the data is sorted.

A general rule of thumb is to avoid data-dependent branching in critical loops (such as in this example).


Update:

  • GCC 4.6.1 with -O3 or -ftree-vectorize on x64 is able to generate a conditional move. So there is no difference between the sorted and unsorted data - both are fast.

    (Or somewhat fast: for the already-sorted case, cmov can be slower especially if GCC puts it on the critical path instead of just add, especially on Intel before Broadwell where cmov has 2 cycle latency: gcc optimization flag -O3 makes code slower than -O2)

  • VC++ 2010 is unable to generate conditional moves for this branch even under /Ox.

  • Intel C++ Compiler (ICC) 11 does something miraculous. It interchanges the two loops, thereby hoisting the unpredictable branch to the outer loop. So not only is it immune to the mispredictions, it is also twice as fast as whatever VC++ and GCC can generate! In other words, ICC took advantage of the test-loop to defeat the benchmark...

  • If you give the Intel compiler the branchless code, it just out-right vectorizes it... and is just as fast as with the branch (with the loop interchange).

This goes to show that even mature modern compilers can vary wildly in their ability to optimize code...

By Daniel Fischer (Votes: 4153)

Branch prediction.

With a sorted array, the condition data[c] >= 128 is first false for a streak of values, then becomes true for all later values. That's easy to predict. With an unsorted array, you pay for the branching cost.

By WiSaGaN (Votes: 3358)

The reason why performance improves drastically when the data is sorted is that the branch prediction penalty is removed, as explained beautifully in Mysticial's answer.

Now, if we look at the code

if (data[c] >= 128)
    sum += data[c];

we can find that the meaning of this particular if... else... branch is to add something when a condition is satisfied. This type of branch can be easily transformed into a conditional move statement, which would be compiled into a conditional move instruction: cmovl, in an x86 system. The branch and thus the potential branch prediction penalty is removed.

In C, thus C++, the statement, which would compile directly (without any optimization) into the conditional move instruction in x86, is the ternary operator ... ? ... : .... So we rewrite the above statement into an equivalent one:

sum += data[c] >=128 ? data[c] : 0;

While maintaining readability, we can check the speedup factor.

On an Intel Core i7-2600K @ 3.4 GHz and Visual Studio 2010 Release Mode, the benchmark is (format copied from Mysticial):

x86

//  Branch - Random
seconds = 8.885

//  Branch - Sorted
seconds = 1.528

//  Branchless - Random
seconds = 3.716

//  Branchless - Sorted
seconds = 3.71

x64

//  Branch - Random
seconds = 11.302

//  Branch - Sorted
 seconds = 1.830

//  Branchless - Random
seconds = 2.736

//  Branchless - Sorted
seconds = 2.737

The result is robust in multiple tests. We get a great speedup when the branch result is unpredictable, but we suffer a little bit when it is predictable. In fact, when using a conditional move, the performance is the same regardless of the data pattern.

Now let's look more closely by investigating the x86 assembly they generate. For simplicity, we use two functions max1 and max2.

max1 uses the conditional branch if... else ...:

int max1(int a, int b) {
    if (a > b)
        return a;
    else
        return b;
}

max2 uses the ternary operator ... ? ... : ...:

int max2(int a, int b) {
    return a > b ? a : b;
}

On a x86-64 machine, GCC -S generates the assembly below.

:max1
    movl    %edi, -4(%rbp)
    movl    %esi, -8(%rbp)
    movl    -4(%rbp), %eax
    cmpl    -8(%rbp), %eax
    jle     .L2
    movl    -4(%rbp), %eax
    movl    %eax, -12(%rbp)
    jmp     .L4
.L2:
    movl    -8(%rbp), %eax
    movl    %eax, -12(%rbp)
.L4:
    movl    -12(%rbp), %eax
    leave
    ret

:max2
    movl    %edi, -4(%rbp)
    movl    %esi, -8(%rbp)
    movl    -4(%rbp), %eax
    cmpl    %eax, -8(%rbp)
    cmovge  -8(%rbp), %eax
    leave
    ret

max2 uses much less code due to the usage of instruction cmovge. But the real gain is that max2 does not involve branch jumps, jmp, which would have a significant performance penalty if the predicted result is not right.

So why does a conditional move perform better?

In a typical x86 processor, the execution of an instruction is divided into several stages. Roughly, we have different hardware to deal with different stages. So we do not have to wait for one instruction to finish to start a new one. This is called pipelining.

In a branch case, the following instruction is determined by the preceding one, so we cannot do pipelining. We have to either wait or predict.

In a conditional move case, the execution conditional move instruction is divided into several stages, but the earlier stages like Fetch and Decode does not depend on the result of the previous instruction; only latter stages need the result. Thus, we wait a fraction of one instruction's execution time. This is why the conditional move version is slower than the branch when prediction is easy.

The book Computer Systems: A Programmer's Perspective, second edition explains this in detail. You can check Section 3.6.6 for Conditional Move Instructions, entire Chapter 4 for Processor Architecture, and Section 5.11.2 for a special treatment for Branch Prediction and Misprediction Penalties.

Sometimes, some modern compilers can optimize our code to assembly with better performance, sometimes some compilers can't (the code in question is using Visual Studio's native compiler). Knowing the performance difference between branch and conditional move when unpredictable can help us write code with better performance when the scenario gets so complex that the compiler can not optimize them automatically.

How can I remove a specific item from an array?

Question

I have an array of numbers and I'm using the .push() method to add elements to it.

Is there a simple way to remove a specific element from an array?

I'm looking for the equivalent of something like:

array.remove(number);

I have to use core JavaScript. Frameworks are not allowed.

Answers

By Tom Wadley (Votes: 12310)

Find the index of the array element you want to remove using indexOf, and then remove that index with splice.

The splice() method changes the contents of an array by removing existing elements and/or adding new elements.

const array = [2, 5, 9];

console.log(array);

const index = array.indexOf(5);
if (index > -1) {
array.splice(index, 1);
}

// array = [2, 9]
console.log(array);

The second parameter of splice is the number of elements to remove. Note that splice modifies the array in place and returns a new array containing the elements that have been removed.


For the reason of completeness, here are functions. The first function removes only a single occurrence (i.e. removing the first match of 5 from [2,5,9,1,5,8,5]), while the second function removes all occurrences:

function removeItemOnce(arr, value) {
  var index = arr.indexOf(value);
  if (index > -1) {
    arr.splice(index, 1);
  }
  return arr;
}

function removeItemAll(arr, value) {
var i = 0;
while (i < arr.length) {
if (arr[i] === value) {
arr.splice(i, 1);
} else {
++i;
}
}
return arr;
}
//Usage
console.log(removeItemOnce([2,5,9,1,5,8,5], 5))
console.log(removeItemAll([2,5,9,1,5,8,5], 5))

By Peter Olson (Votes: 1331)

I don't know how you are expecting array.remove(int) to behave. There are three possibilities I can think of that you might want.

To remove an element of an array at an index i:

array.splice(i, 1);

If you want to remove every element with value number from the array:

for(var i = array.length - 1; i >= 0; i--) {
    if(array[i] === number) {
        array.splice(i, 1);
    }
}

If you just want to make the element at index i no longer exist, but you don't want the indexes of the other elements to change:

delete array[i];

By ujeenator (Votes: 1289)

Edited on 2016 October

  • Do it simple, intuitive and explicit (Occam's razor)
  • Do it immutable (original array stay unchanged)
  • Do it with standard JavaScript functions, if your browser doesn't support them - use polyfill

In this code example I use "array.filter(...)" function to remove unwanted items from an array. This function doesn't change the original array and creates a new one. If your browser doesn't support this function (e.g. Internet Explorer before version 9, or Firefox before version 1.5), consider using the filter polyfill from Mozilla.

Removing item (ECMA-262 Edition 5 code aka oldstyle JavaScript)

var value = 3

var arr = [1, 2, 3, 4, 5, 3]

arr = arr.filter(function(item) {
    return item !== value
})

console.log(arr)
// [ 1, 2, 4, 5 ]

Removing item (ECMAScript 6 code)

let value = 3

let arr = [1, 2, 3, 4, 5, 3]

arr = arr.filter(item => item !== value)

console.log(arr)
// [ 1, 2, 4, 5 ]

IMPORTANT ECMAScript 6 "() => {}" arrow function syntax is not supported in Internet Explorer at all, Chrome before 45 version, Firefox before 22 version, and Safari before 10 version. To use ECMAScript 6 syntax in old browsers you can use BabelJS.


Removing multiple items (ECMAScript 7 code)

An additional advantage of this method is that you can remove multiple items

let forDeletion = [2, 3, 5]

let arr = [1, 2, 3, 4, 5, 3]

arr = arr.filter(item => !forDeletion.includes(item))
// !!! Read below about array.includes(...) support !!!

console.log(arr)
// [ 1, 4 ]

IMPORTANT "array.includes(...)" function is not supported in Internet Explorer at all, Chrome before 47 version, Firefox before 43 version, Safari before 9 version, and Edge before 14 version so here is polyfill from Mozilla.

Removing multiple items (in the future, maybe)

If the "This-Binding Syntax" proposal is ever accepted, you'll be able to do this:

// array-lib.js

export function remove(...forDeletion) {
    return this.filter(item => !forDeletion.includes(item))
}

// main.js

import { remove } from './array-lib.js'

let arr = [1, 2, 3, 4, 5, 3]

// :: This-Binding Syntax Proposal
// using "remove" function as "virtual method"
// without extending Array.prototype
arr = arr::remove(2, 3, 5)

console.log(arr)
// [ 1, 4 ]

Try it yourself in BabelJS :)

Reference

For-each over an array in JavaScript

Question

How can I loop through all the entries in an array using JavaScript?

I thought it was something like this:

forEach(instance in theArray)

Where theArray is my array, but this seems to be incorrect.

Answers

By T.J. Crowder (Votes: 7117)

TL;DR

  • Don't use for-in unless you use it with safeguards or are at least aware of why it might bite you.
  • Your best bets are usually

    • a for-of loop (ES2015+ only),
    • Array#forEach (spec | MDN) (or its relatives some and such) (ES5+ only),
    • a simple old-fashioned for loop,
    • or for-in with safeguards.

But there's lots more to explore, read on...


JavaScript has powerful semantics for looping through arrays and array-like objects. I've split the answer into two parts: Options for genuine arrays, and options for things that are just array-like, such as the arguments object, other iterable objects (ES2015+), DOM collections, and so on.

I'll quickly note that you can use the ES2015 options now, even on ES5 engines, by transpiling ES2015 to ES5. Search for "ES2015 transpiling" / "ES6 transpiling" for more...

Okay, let's look at our options:

For Actual Arrays

You have three options in ECMAScript 5 ("ES5"), the version most broadly supported at the moment, and two more added in ECMAScript 2015 ("ES2015", "ES6"):

  1. Use forEach and related (ES5+)
  2. Use a simple for loop
  3. Use for-in correctly
  4. Use for-of (use an iterator implicitly) (ES2015+)
  5. Use an iterator explicitly (ES2015+)

Details:

1. Use forEach and related

In any vaguely-modern environment (so, not IE8) where you have access to the Array features added by ES5 (directly or using polyfills), you can use forEach (spec | MDN):

var a = ["a", "b", "c"];
a.forEach(function(entry) {
    console.log(entry);
});

forEach accepts a callback function and, optionally, a value to use as this when calling that callback (not used above). The callback is called for each entry in the array, in order, skipping non-existent entries in sparse arrays. Although I only used one argument above, the callback is called with three: The value of each entry, the index of that entry, and a reference to the array you're iterating over (in case your function doesn't already have it handy).

Unless you're supporting obsolete browsers like IE8 (which NetApps shows at just over 4% market share as of this writing in September 2016), you can happily use forEach in a general-purpose web page without a shim. If you do need to support obsolete browsers, shimming/polyfilling forEach is easily done (search for "es5 shim" for several options).

forEach has the benefit that you don't have to declare indexing and value variables in the containing scope, as they're supplied as arguments to the iteration function, and so nicely scoped to just that iteration.

If you're worried about the runtime cost of making a function call for each array entry, don't be; details.

Additionally, forEach is the "loop through them all" function, but ES5 defined several other useful "work your way through the array and do things" functions, including:

  • every (stops looping the first time the callback returns false or something falsey)
  • some (stops looping the first time the callback returns true or something truthy)
  • filter (creates a new array including elements where the filter function returns true and omitting the ones where it returns false)
  • map (creates a new array from the values returned by the callback)
  • reduce (builds up a value by repeatedly calling the callback, passing in previous values; see the spec for the details; useful for summing the contents of an array and many other things)
  • reduceRight (like reduce, but works in descending rather than ascending order)

2. Use a simple for loop

Sometimes the old ways are the best:

var index;
var a = ["a", "b", "c"];
for (index = 0; index < a.length; ++index) {
    console.log(a[index]);
}

If the length of the array won't change during the loop, and it's in performance-sensitive code (unlikely), a slightly more complicated version grabbing the length up front might be a tiny bit faster:

var index, len;
var a = ["a", "b", "c"];
for (index = 0, len = a.length; index < len; ++index) {
    console.log(a[index]);
}

And/or counting backward:

var index;
var a = ["a", "b", "c"];
for (index = a.length - 1; index >= 0; --index) {
    console.log(a[index]);
}

But with modern JavaScript engines, it's rare you need to eke out that last bit of juice.

In ES2015 and higher, you can make your index and value variables local to the for loop:

let a = ["a", "b", "c"];
for (let index = 0; index < a.length; ++index) {
    let value = a[index];
    console.log(index, value);
}
//console.log(index);   // would cause "ReferenceError: index is not defined"
//console.log(value);   // would cause "ReferenceError: value is not defined"

let a = ["a", "b", "c"];
for (let index = 0; index < a.length; ++index) {
    let value = a[index];
    console.log(index, value);
}
try {
    console.log(index);
} catch (e) {
    console.error(e);   // "ReferenceError: index is not defined"
}
try {
    console.log(value);
} catch (e) {
    console.error(e);   // "ReferenceError: value is not defined"
}

And when you do that, not just value but also index is recreated for each loop iteration, meaning closures created in the loop body keep a reference to the index (and value) created for that specific iteration:

let divs = document.querySelectorAll("div");
for (let index = 0; index < divs.length; ++index) {
    divs[index].addEventListener('click', e => {
        console.log("Index is: " + index);
    });
}

let divs = document.querySelectorAll("div");
for (let index = 0; index < divs.length; ++index) {
    divs[index].addEventListener('click', e => {
        console.log("Index is: " + index);
    });
}
<div>zero</div>
<div>one</div>
<div>two</div>
<div>three</div>
<div>four</div>

If you had five divs, you'd get "Index is: 0" if you clicked the first and "Index is: 4" if you clicked the last. This does not work if you use var instead of let.

3. Use for-in correctly

You'll get people telling you to use for-in, but that's not what for-in is for. for-in loops through the enumerable properties of an object, not the indexes of an array. The order is not guaranteed, not even in ES2015 (ES6). ES2015+ does define an order to object properties (via [[OwnPropertyKeys]], [[Enumerate]], and things that use them like Object.getOwnPropertyKeys), but it didn't define that for-in would follow that order; ES2020 did, though. (Details in this other answer.)

The only real use cases for for-in on an array are:

  • It's a sparse arrays with massive gaps in it, or
  • You're using non-element properties and you want to include them in the loop

Looking only at that first example: You can use for-in to visit those sparse array elements if you use appropriate safeguards:

// `a` is a sparse array
var key;
var a = [];
a[0] = "a";
a[10] = "b";
a[10000] = "c";
for (key in a) {
    if (a.hasOwnProperty(key)  &&        // These checks are
        /^0$|^[1-9]\d*$/.test(key) &&    // explained
        key <= 4294967294                // below
        ) {
        console.log(a[key]);
    }
}

Note the three checks:

  1. That the object has its own property by that name (not one it inherits from its prototype), and

  2. That the key is all decimal digits (e.g., normal string form, not scientific notation), and

  3. That the key's value when coerced to a number is <= 2^32 - 2 (which is 4,294,967,294). Where does that number come from? It's part of the definition of an array index in the specification. Other numbers (non-integers, negative numbers, numbers greater than 2^32 - 2) are not array indexes. The reason it's 2^32 - 2 is that that makes the greatest index value one lower than 2^32 - 1, which is the maximum value an array's length can have. (E.g., an array's length fits in a 32-bit unsigned integer.) (Props to RobG for pointing out in a comment on my blog post that my previous test wasn't quite right.)

You wouldn't do that in inline code, of course. You'd write a utility function. Perhaps:

// Utility function for antiquated environments without `forEach`
var hasOwn = Object.prototype.hasOwnProperty;
var rexNum = /^0$|^[1-9]\d*$/;
function sparseEach(array, callback, thisArg) {
    var index;
    for (var key in array) {
        index = +key;
        if (hasOwn.call(a, key) &&
            rexNum.test(key) &&
            index <= 4294967294
            ) {
            callback.call(thisArg, array[key], index, array);
        }
    }
}

var a = [];
a[5] = "five";
a[10] = "ten";
a[100000] = "one hundred thousand";
a.b = "bee";

sparseEach(a, function(value, index) {
console.log("Value at " + index + " is " + value);
});

4. Use for-of (use an iterator implicitly) (ES2015+)

ES2015 added iterators to JavaScript. The easiest way to use iterators is the new for-of statement. It looks like this:

const a = ["a", "b", "c"];
for (const val of a) {
    console.log(val);
}

Under the covers, that gets an iterator from the array and loops through it, getting the values from it. This doesn't have the issue that using for-in has, because it uses an iterator defined by the object (the array), and arrays define that their iterators iterate through their entries (not their properties). Unlike for-in in ES5, the order in which the entries are visited is the numeric order of their indexes.

5. Use an iterator explicitly (ES2015+)

Sometimes, you might want to use an iterator explicitly. You can do that, too, although it's a lot clunkier than for-of. It looks like this:

const a = ["a", "b", "c"];
const it = a.values();
let entry;
while (!(entry = it.next()).done) {
    console.log(entry.value);
}

The iterator is an object matching the Iterator definition in the specification. Its next method returns a new result object each time you call it. The result object has a property, done, telling us whether it's done, and a property value with the value for that iteration. (done is optional if it would be false, value is optional if it would be undefined.)

The meaning of value varies depending on the iterator; arrays support (at least) three functions that return iterators:

  • values(): This is the one I used above. It returns an iterator where each value is the array entry for that iteration ("a", "b", and "c" in the example earlier).
  • keys(): Returns an iterator where each value is the key for that iteration (so for our a above, that would be "0", then "1", then "2").
  • entries(): Returns an iterator where each value is an array in the form [key, value] for that iteration.

For Array-Like Objects

Aside from true arrays, there are also array-like objects that have a length property and properties with numeric names: NodeList instances, the arguments object, etc. How do we loop through their contents?

Use any of the options above for arrays

At least some, and possibly most or even all, of the array approaches above frequently apply equally well to array-like objects:

  1. Use forEach and related (ES5+)

    The various functions on Array.prototype are "intentionally generic" and can usually be used on array-like objects via Function#call or Function#apply. (See the Caveat for host-provided objects at the end of this answer, but it's a rare issue.)

    Suppose you wanted to use forEach on a Node's childNodes property. You'd do this:

    Array.prototype.forEach.call(node.childNodes, function(child) {
        // Do something with `child`
    });
    

    If you're going to do that a lot, you might want to grab a copy of the function reference into a variable for reuse, e.g.:

    // (This is all presumably in some scoping function)
    var forEach = Array.prototype.forEach;
    
    // Then later...
    forEach.call(node.childNodes, function(child) {
        // Do something with `child`
    });
    
  2. Use a simple for loop

    Obviously, a simple for loop applies to array-like objects.

  3. Use for-in correctly

    for-in with the same safeguards as with an array should work with array-like objects as well; the caveat for host-provided objects on Test action megathread (issue) #1 above may apply.

  4. Use for-of (use an iterator implicitly) (ES2015+)

    for-of uses the iterator provided by the object (if any). That includes host-provided objects. For instance, the specification for the NodeList from querySelectorAll was updated to support iteration. The spec for the HTMLCollection from getElementsByTagName was not.

  5. Use an iterator explicitly (ES2015+)

    See chore: update deps (2020-09-01) #4.

Create a true array

Other times, you may want to convert an array-like object into a true array. Doing that is surprisingly easy:

  1. Use the slice method of arrays

    We can use the slice method of arrays, which like the other methods mentioned above is "intentionally generic" and so can be used with array-like objects, like this:

    var trueArray = Array.prototype.slice.call(arrayLikeObject);
    

    So for instance, if we want to convert a NodeList into a true array, we could do this:

    var divs = Array.prototype.slice.call(document.querySelectorAll("div"));
    

    See the Caveat for host-provided objects below. In particular, note that this will fail in IE8 and earlier, which don't let you use host-provided objects as this like that.

  2. Use spread syntax (...)

    It's also possible to use ES2015's spread syntax with JavaScript engines that support this feature. Like for-of, this uses the iterator provided by the object (see chore: update deps (2020-09-01) #4 in the previous section):

    var trueArray = [...iterableObject];
    

    So for instance, if we want to convert a NodeList into a true array, with spread syntax this becomes quite succinct:

    var divs = [...document.querySelectorAll("div")];
    
  3. Use Array.from

    Array.from (spec) | (MDN) (ES2015+, but easily polyfilled) creates an array from an array-like object, optionally passing the entries through a mapping function first. So:

    var divs = Array.from(document.querySelectorAll("div"));
    

    Or if you wanted to get an array of the tag names of the elements with a given class, you'd use the mapping function:

    // Arrow function (ES2015):
    var divs = Array.from(document.querySelectorAll(".some-class"), element => element.tagName);
    
    // Standard function (since `Array.from` can be shimmed):
    var divs = Array.from(document.querySelectorAll(".some-class"), function(element) {
        return element.tagName;
    });
    

Caveat for host-provided objects

If you use Array.prototype functions with host-provided array-like objects (DOM lists and other things provided by the browser rather than the JavaScript engine), you need to be sure to test in your target environments to make sure the host-provided object behaves properly. Most do behave properly (now), but it's important to test. The reason is that most of the Array.prototype methods you're likely to want to use rely on the host-provided object giving an honest answer to the abstract [[HasProperty]] operation. As of this writing, browsers do a very good job of this, but the 5.1 spec did allow for the possibility a host-provided object may not be honest. It's in §8.6.2, several paragraphs below the big table near the beginning of that section), where it says:

Host objects may implement these internal methods in any manner unless specified otherwise; for example, one possibility is that [[Get]] and [[Put]] for a particular host object indeed fetch and store property values but [[HasProperty]] always generates false.

(I couldn't find the equivalent verbiage in the ES2015 spec, but it's bound to still be the case.) Again, as of this writing the common host-provided array-like objects in modern browsers [NodeList instances, for instance] do handle [[HasProperty]] correctly, but it's important to test.)

By PatrikAkerstrand (Votes: 524)

Note: This answer is hopelessly out-of-date. For a more modern approach, look at the methods available on an array. Methods of interest might be:

  • forEach
  • map
  • filter
  • zip
  • reduce
  • every
  • some

The standard way to iterate an array in JavaScript is a vanilla for-loop:

var length = arr.length,
    element = null;
for (var i = 0; i < length; i++) {
  element = arr[i];
  // Do something with element
}

Note, however, that this approach is only good if you have a dense array, and each index is occupied by an element. If the array is sparse, then you can run into performance problems with this approach, since you will iterate over a lot of indices that do not really exist in the array. In this case, a for .. in-loop might be a better idea. However, you must use the appropriate safeguards to ensure that only the desired properties of the array (that is, the array elements) are acted upon, since the for..in-loop will also be enumerated in legacy browsers, or if the additional properties are defined as enumerable.

In ECMAScript 5 there will be a forEach method on the array prototype, but it is not supported in legacy browsers. So to be able to use it consistently you must either have an environment that supports it (for example, Node.js for server side JavaScript), or use a "Polyfill". The Polyfill for this functionality is, however, trivial and since it makes the code easier to read, it is a good polyfill to include.

By Poonam (Votes: 246)

If you’re using the jQuery library, you can use jQuery.each:

$.each(yourArray, function(index, value) {
  // do your stuff here
});

EDIT :

As per question, user want code in javascript instead of jquery so the edit is

var length = yourArray.length;   
for (var i = 0; i < length; i++) {
  // Do something with yourArray[i].
}

@neverendingqs
Copy link
Owner Author

/so [python] array

@github-actions
Copy link
Contributor

Results for [python] array

Convert bytes to a string

Question

I'm using this code to get standard output from an external program:

>>> from subprocess import *
>>> command_stdout = Popen(['ls', '-l'], stdout=PIPE).communicate()[0]

The communicate() method returns an array of bytes:

>>> command_stdout
b'total 0\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2\n'

However, I'd like to work with the output as a normal Python string. So that I could print it like this:

>>> print(command_stdout)
-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1
-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2

I thought that's what the binascii.b2a_qp() method is for, but when I tried it, I got the same byte array again:

>>> binascii.b2a_qp(command_stdout)
b'total 0\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file1\n-rw-rw-r-- 1 thomas thomas 0 Mar  3 07:03 file2\n'

How do I convert the bytes value back to string? I mean, using the "batteries" instead of doing it manually. And I'd like it to be OK with Python 3.

Answers

By Aaron Maenpaa (Votes: 3865)

You need to decode the bytes object to produce a string:

>>> b"abcde"
b'abcde'

# utf-8 is used here because it is a very common encoding, but you
# need to use the encoding your data is actually in.
>>> b"abcde".decode("utf-8") 
'abcde'

By dF. (Votes: 230)

You need to decode the byte string and turn it in to a character (Unicode) string.

On Python 2

encoding = 'utf-8'
'hello'.decode(encoding)

or

unicode('hello', encoding)

On Python 3

encoding = 'utf-8'
b'hello'.decode(encoding)

or

str(b'hello', encoding)

By Sisso (Votes: 190)

I think this way is easy:

>>> bytes_data = [112, 52, 52]
>>> "".join(map(chr, bytes_data))
'p44'

How do I sort a list of dictionaries by a value of the dictionary?

Question

I have a list of dictionaries and want each item to be sorted by a specific property values.

Take into consideration the array below,

[{'name':'Homer', 'age':39}, {'name':'Bart', 'age':10}]

When sorted by name, should become

[{'name':'Bart', 'age':10}, {'name':'Homer', 'age':39}]
Answers

By Mario F (Votes: 2540)

It may look cleaner using a key instead a cmp:

newlist = sorted(list_to_be_sorted, key=lambda k: k['name']) 

or as J.F.Sebastian and others suggested,

from operator import itemgetter
newlist = sorted(list_to_be_sorted, key=itemgetter('name')) 

For completeness (as pointed out in comments by fitzgeraldsteele), add reverse=True to sort descending

newlist = sorted(l, key=itemgetter('name'), reverse=True)

By vemury (Votes: 174)

import operator

To sort the list of dictionaries by key='name':

list_of_dicts.sort(key=operator.itemgetter('name'))

To sort the list of dictionaries by key='age':

list_of_dicts.sort(key=operator.itemgetter('age'))

By pjz (Votes: 70)

my_list = [{'name':'Homer', 'age':39}, {'name':'Bart', 'age':10}]

my_list.sort(lambda x,y : cmp(x['name'], y['name']))

my_list will now be what you want.

Or better:

The new key argument is more efficient and neater:

my_list = sorted(my_list, key=lambda k: k['name'])

...the lambda is, IMO, easier to understand than operator.itemgetter, but your mileage may vary.

Peak detection in a 2D array

Question

I'm helping a veterinary clinic measuring pressure under a dogs paw. I use Python for my data analysis and now I'm stuck trying to divide the paws into (anatomical) subregions.

I made a 2D array of each paw, that consists of the maximal values for each sensor that has been loaded by the paw over time. Here's an example of one paw, where I used Excel to draw the areas I want to 'detect'. These are 2 by 2 boxes around the sensor with local maxima's, that together have the largest sum.

alt text

So I tried some experimenting and decide to simply look for the maximums of each column and row (can't look in one direction due to the shape of the paw). This seems to 'detect' the location of the separate toes fairly well, but it also marks neighboring sensors.

alt text

So what would be the best way to tell Python which of these maximums are the ones I want?

Note: The 2x2 squares can't overlap, since they have to be separate toes!

Also I took 2x2 as a convenience, any more advanced solution is welcome, but I'm simply a human movement scientist, so I'm neither a real programmer or a mathematician, so please keep it 'simple'.

Here's a version that can be loaded with np.loadtxt


Results

So I tried @jextee's solution (see the results below). As you can see, it works very on the front paws, but it works less well for the hind legs.

More specifically, it can't recognize the small peak that's the fourth toe. This is obviously inherent to the fact that the loop looks top down towards the lowest value, without taking into account where this is.

Would anyone know how to tweak @jextee's algorithm, so that it might be able to find the 4th toe too?

alt text

Since I haven't processed any other trials yet, I can't supply any other samples. But the data I gave before were the averages of each paw. This file is an array with the maximal data of 9 paws in the order they made contact with the plate.

This image shows how they were spatially spread out over the plate.

alt text

Update:

I have set up a blog for anyone interested and I have setup a SkyDrive with all the raw measurements. So to anyone requesting more data: more power to you!


New update:

So after the help I got with my questions regarding paw detection and paw sorting, I was finally able to check the toe detection for every paw! Turns out, it doesn't work so well in anything but paws sized like the one in my own example. Off course in hindsight, it's my own fault for choosing the 2x2 so arbitrarily.

Here's a nice example of where it goes wrong: a nail is being recognized as a toe and the 'heel' is so wide, it gets recognized twice!

alt text

The paw is too large, so taking a 2x2 size with no overlap, causes some toes to be detected twice. The other way around, in small dogs it often fails to find a 5th toe, which I suspect is being caused by the 2x2 area being too large.

After trying the current solution on all my measurements I came to the staggering conclusion that for nearly all my small dogs it didn't find a 5th toe and that in over 50% of the impacts for the large dogs it would find more!

So clearly I need to change it. My own guess was changing the size of the neighborhood to something smaller for small dogs and larger for large dogs. But generate_binary_structure wouldn't let me change the size of the array.

Therefore, I'm hoping that anyone else has a better suggestion for locating the toes, perhaps having the toe area scale with the paw size?

Answers

By Ivan (Votes: 340)

I detected the peaks using a local maximum filter. Here is the result on your first dataset of 4 paws: Peaks detection result

I also ran it on the second dataset of 9 paws and it worked as well.

Here is how you do it:

import numpy as np
from scipy.ndimage.filters import maximum_filter
from scipy.ndimage.morphology import generate_binary_structure, binary_erosion
import matplotlib.pyplot as pp

#for some reason I had to reshape. Numpy ignored the shape header.
paws_data = np.loadtxt("paws.txt").reshape(4,11,14)

#getting a list of images
paws = [p.squeeze() for p in np.vsplit(paws_data,4)]


def detect_peaks(image):
    """
    Takes an image and detect the peaks usingthe local maximum filter.
    Returns a boolean mask of the peaks (i.e. 1 when
    the pixel's value is the neighborhood maximum, 0 otherwise)
    """

    # define an 8-connected neighborhood
    neighborhood = generate_binary_structure(2,2)

    #apply the local maximum filter; all pixel of maximal value 
    #in their neighborhood are set to 1
    local_max = maximum_filter(image, footprint=neighborhood)==image
    #local_max is a mask that contains the peaks we are 
    #looking for, but also the background.
    #In order to isolate the peaks we must remove the background from the mask.

    #we create the mask of the background
    background = (image==0)

    #a little technicality: we must erode the background in order to 
    #successfully subtract it form local_max, otherwise a line will 
    #appear along the background border (artifact of the local maximum filter)
    eroded_background = binary_erosion(background, structure=neighborhood, border_value=1)

    #we obtain the final mask, containing only peaks, 
    #by removing the background from the local_max mask (xor operation)
    detected_peaks = local_max ^ eroded_background

    return detected_peaks


#applying the detection and plotting results
for i, paw in enumerate(paws):
    detected_peaks = detect_peaks(paw)
    pp.subplot(4,2,(2*i+1))
    pp.imshow(paw)
    pp.subplot(4,2,(2*i+2) )
    pp.imshow(detected_peaks)

pp.show()

All you need to do after is use scipy.ndimage.measurements.label on the mask to label all distinct objects. Then you'll be able to play with them individually.

Note that the method works well because the background is not noisy. If it were, you would detect a bunch of other unwanted peaks in the background. Another important factor is the size of the neighborhood. You will need to adjust it if the peak size changes (the should remain roughly proportional).

By sastanin (Votes: 53)

Solution

Data file: paw.txt. Source code:

from scipy import *
from operator import itemgetter

n = 5  # how many fingers are we looking for

d = loadtxt("paw.txt")
width, height = d.shape

# Create an array where every element is a sum of 2x2 squares.

fourSums = d[:-1,:-1] + d[1:,:-1] + d[1:,1:] + d[:-1,1:]

# Find positions of the fingers.

# Pair each sum with its position number (from 0 to width*height-1),

pairs = zip(arange(width*height), fourSums.flatten())

# Sort by descending sum value, filter overlapping squares

def drop_overlapping(pairs):
    no_overlaps = []
    def does_not_overlap(p1, p2):
        i1, i2 = p1[0], p2[0]
        r1, col1 = i1 / (width-1), i1 % (width-1)
        r2, col2 = i2 / (width-1), i2 % (width-1)
        return (max(abs(r1-r2),abs(col1-col2)) >= 2)
    for p in pairs:
        if all(map(lambda prev: does_not_overlap(p,prev), no_overlaps)):
            no_overlaps.append(p)
    return no_overlaps

pairs2 = drop_overlapping(sorted(pairs, key=itemgetter(1), reverse=True))

# Take the first n with the heighest values

positions = pairs2[:n]

# Print results

print d, "\n"

for i, val in positions:
    row = i / (width-1)
    column = i % (width-1)
    print "sum = %f @ %d,%d (%d)" % (val, row, column, i)
    print d[row:row+2,column:column+2], "\n"

Output without overlapping squares. It seems that the same areas are selected as in your example.

Some comments

The tricky part is to calculate sums of all 2x2 squares. I assumed you need all of them, so there might be some overlapping. I used slices to cut the first/last columns and rows from the original 2D array, and then overlapping them all together and calculating sums.

To understand it better, imaging a 3x3 array:

>>> a = arange(9).reshape(3,3) ; a
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])

Then you can take its slices:

>>> a[:-1,:-1]
array([[0, 1],
       [3, 4]])
>>> a[1:,:-1]
array([[3, 4],
       [6, 7]])
>>> a[:-1,1:]
array([[1, 2],
       [4, 5]])
>>> a[1:,1:]
array([[4, 5],
       [7, 8]])

Now imagine you stack them one above the other and sum elements at the same positions. These sums will be exactly the same sums over the 2x2 squares with the top-left corner in the same position:

>>> sums = a[:-1,:-1] + a[1:,:-1] + a[:-1,1:] + a[1:,1:]; sums
array([[ 8, 12],
       [20, 24]])

When you have the sums over 2x2 squares, you can use max to find the maximum, or sort, or sorted to find the peaks.

To remember positions of the peaks I couple every value (the sum) with its ordinal position in a flattened array (see zip). Then I calculate row/column position again when I print the results.

Notes

I allowed for the 2x2 squares to overlap. Edited version filters out some of them such that only non-overlapping squares appear in the results.

Choosing fingers (an idea)

Another problem is how to choose what is likely to be fingers out of all the peaks. I have an idea which may or may not work. I don't have time to implement it right now, so just pseudo-code.

I noticed that if the front fingers stay on almost a perfect circle, the rear finger should be inside of that circle. Also, the front fingers are more or less equally spaced. We may try to use these heuristic properties to detect the fingers.

Pseudo code:

select the top N finger candidates (not too many, 10 or 12)
consider all possible combinations of 5 out of N (use itertools.combinations)
for each combination of 5 fingers:
    for each finger out of 5:
        fit the best circle to the remaining 4
        => position of the center, radius
        check if the selected finger is inside of the circle
        check if the remaining four are evenly spread
        (for example, consider angles from the center of the circle)
        assign some cost (penalty) to this selection of 4 peaks + a rear finger
        (consider, probably weighted:
             circle fitting error,
             if the rear finger is inside,
             variance in the spreading of the front fingers,
             total intensity of 5 peaks)
choose a combination of 4 peaks + a rear peak with the lowest penalty

This is a brute-force approach. If N is relatively small, then I think it is doable. For N=12, there are C_12^5 = 792 combinations, times 5 ways to select a rear finger, so 3960 cases to evaluate for every paw.

By CakeMaster (Votes: 34)

This is an image registration problem. The general strategy is:

  • Have a known example, or some kind of prior on the data.
  • Fit your data to the example, or fit the example to your data.
  • It helps if your data is roughly aligned in the first place.

Here's a rough and ready approach, "the dumbest thing that could possibly work":

  • Start with five toe coordinates in roughly the place you expect.
  • With each one, iteratively climb to the top of the hill. i.e. given current position, move to maximum neighbouring pixel, if its value is greater than current pixel. Stop when your toe coordinates have stopped moving.

To counteract the orientation problem, you could have 8 or so initial settings for the basic directions (North, North East, etc). Run each one individually and throw away any results where two or more toes end up at the same pixel. I'll think about this some more, but this kind of thing is still being researched in image processing - there are no right answers!

Slightly more complex idea: (weighted) K-means clustering. It's not that bad.

  • Start with five toe coordinates, but now these are "cluster centres".

Then iterate until convergence:

  • Assign each pixel to the closest cluster (just make a list for each cluster).
  • Calculate the center of mass of each cluster. For each cluster, this is: Sum(coordinate * intensity value)/Sum(coordinate)
  • Move each cluster to the new centre of mass.

This method will almost certainly give much better results, and you get the mass of each cluster which may help in identifying the toes.

(Again, you've specified the number of clusters up front. With clustering you have to specify the density one way or another: Either choose the number of clusters, appropriate in this case, or choose a cluster radius and see how many you end up with. An example of the latter is mean-shift.)

Sorry about the lack of implementation details or other specifics. I would code this up but I've got a deadline. If nothing else has worked by next week let me know and I'll give it a shot.

@neverendingqs
Copy link
Owner Author

/so

@neverendingqs
Copy link
Owner Author

/so

@github-actions
Copy link
Contributor

Results for ``

Why is processing a sorted array faster than processing an unsorted array?

Question

Here is a piece of C++ code that shows some very peculiar behavior. For some strange reason, sorting the data miraculously makes the code almost six times faster:

#include <algorithm>
#include <ctime>
#include <iostream>

int main()
{
// Generate data
const unsigned arraySize = 32768;
int data[arraySize];

for (unsigned c = 0; c &lt; arraySize; ++c)
    data[c] = std::rand() % 256;

// !!! With this, the next loop runs faster.
std::sort(data, data + arraySize);

// Test
clock_t start = clock();
long long sum = 0;

for (unsigned i = 0; i &lt; 100000; ++i)
{
    // Primary loop
    for (unsigned c = 0; c &lt; arraySize; ++c)
    {
        if (data[c] &gt;= 128)
            sum += data[c];
    }
}

double elapsedTime = static_cast&lt;double&gt;(clock() - start) / CLOCKS_PER_SEC;

std::cout &lt;&lt; elapsedTime &lt;&lt; std::endl;
std::cout &lt;&lt; &quot;sum = &quot; &lt;&lt; sum &lt;&lt; std::endl;

}

  • Without std::sort(data, data + arraySize);, the code runs in 11.54 seconds.
  • With the sorted data, the code runs in 1.93 seconds.

Initially, I thought this might be just a language or compiler anomaly, so I tried Java:

import java.util.Arrays;
import java.util.Random;

public class Main
{
public static void main(String[] args)
{
// Generate data
int arraySize = 32768;
int data[] = new int[arraySize];

    Random rnd = new Random(0);
    for (int c = 0; c &lt; arraySize; ++c)
        data[c] = rnd.nextInt() % 256;

    // !!! With this, the next loop runs faster
    Arrays.sort(data);

    // Test
    long start = System.nanoTime();
    long sum = 0;

    for (int i = 0; i &lt; 100000; ++i)
    {
        // Primary loop
        for (int c = 0; c &lt; arraySize; ++c)
        {
            if (data[c] &gt;= 128)
                sum += data[c];
        }
    }

    System.out.println((System.nanoTime() - start) / 1000000000.0);
    System.out.println(&quot;sum = &quot; + sum);
}

}

With a similar but less extreme result.


My first thought was that sorting brings the data into the cache, but then I thought how silly that was because the array was just generated.

  • What is going on?
  • Why is processing a sorted array faster than processing an unsorted array?

The code is summing up some independent terms, so the order should not matter.

Answers

By Mysticial (Votes: 32157)

You are a victim of branch prediction fail.


What is Branch Prediction?

Consider a railroad junction:

Image showing a railroad junction Image by Mecanismo, via Wikimedia Commons. Used under the CC-By-SA 3.0 license.

Now for the sake of argument, suppose this is back in the 1800s - before long distance or radio communication.

You are the operator of a junction and you hear a train coming. You have no idea which way it is supposed to go. You stop the train to ask the driver which direction they want. And then you set the switch appropriately.

Trains are heavy and have a lot of inertia. So they take forever to start up and slow down.

Is there a better way? You guess which direction the train will go!

  • If you guessed right, it continues on.
  • If you guessed wrong, the captain will stop, back up, and yell at you to flip the switch. Then it can restart down the other path.

If you guess right every time, the train will never have to stop.
If you guess wrong too often, the train will spend a lot of time stopping, backing up, and restarting.


Consider an if-statement: At the processor level, it is a branch instruction:

Screenshot of compiled code containing an if statement

You are a processor and you see a branch. You have no idea which way it will go. What do you do? You halt execution and wait until the previous instructions are complete. Then you continue down the correct path.

Modern processors are complicated and have long pipelines. So they take forever to "warm up" and "slow down".

Is there a better way? You guess which direction the branch will go!

  • If you guessed right, you continue executing.
  • If you guessed wrong, you need to flush the pipeline and roll back to the branch. Then you can restart down the other path.

If you guess right every time, the execution will never have to stop.
If you guess wrong too often, you spend a lot of time stalling, rolling back, and restarting.


This is branch prediction. I admit it's not the best analogy since the train could just signal the direction with a flag. But in computers, the processor doesn't know which direction a branch will go until the last moment.

So how would you strategically guess to minimize the number of times that the train must back up and go down the other path? You look at the past history! If the train goes left 99% of the time, then you guess left. If it alternates, then you alternate your guesses. If it goes one way every three times, you guess the same...

In other words, you try to identify a pattern and follow it. This is more or less how branch predictors work.

Most applications have well-behaved branches. So modern branch predictors will typically achieve >90% hit rates. But when faced with unpredictable branches with no recognizable patterns, branch predictors are virtually useless.

Further reading: "Branch predictor" article on Wikipedia.


As hinted from above, the culprit is this if-statement:

if (data[c] >= 128)
    sum += data[c];

Notice that the data is evenly distributed between 0 and 255. When the data is sorted, roughly the first half of the iterations will not enter the if-statement. After that, they will all enter the if-statement.

This is very friendly to the branch predictor since the branch consecutively goes the same direction many times. Even a simple saturating counter will correctly predict the branch except for the few iterations after it switches direction.

Quick visualization:

T = branch taken
N = branch not taken

data[] = 0, 1, 2, 3, 4, ... 126, 127, 128, 129, 130, ... 250, 251, 252, ...
branch = N N N N N ... N N T T T ... T T T ...

   = NNNNNNNNNNNN ... NNNNNNNTTTTTTTTT ... TTTTTTTTTT  (easy to predict)

However, when the data is completely random, the branch predictor is rendered useless, because it can't predict random data. Thus there will probably be around 50% misprediction (no better than random guessing).

data[] = 226, 185, 125, 158, 198, 144, 217, 79, 202, 118,  14, 150, 177, 182, 133, ...
branch =   T,   T,   N,   T,   T,   T,   T,  N,   T,   N,   N,   T,   T,   T,   N  ...
   = TTNTTTTNTNNTTTN ...   (completely random - hard to predict)


So what can be done?

If the compiler isn't able to optimize the branch into a conditional move, you can try some hacks if you are willing to sacrifice readability for performance.

Replace:

if (data[c] >= 128)
    sum += data[c];

with:

int t = (data[c] - 128) >> 31;
sum += ~t & data[c];

This eliminates the branch and replaces it with some bitwise operations.

(Note that this hack is not strictly equivalent to the original if-statement. But in this case, it's valid for all the input values of data[].)

Benchmarks: Core i7 920 @ 3.5 GHz

C++ - Visual Studio 2010 - x64 Release

//  Branch - Random
seconds = 11.777

// Branch - Sorted
seconds = 2.352

// Branchless - Random
seconds = 2.564

// Branchless - Sorted
seconds = 2.587

Java - NetBeans 7.1.1 JDK 7 - x64

//  Branch - Random
seconds = 10.93293813

// Branch - Sorted
seconds = 5.643797077

// Branchless - Random
seconds = 3.113581453

// Branchless - Sorted
seconds = 3.186068823

Observations:

  • With the Branch: There is a huge difference between the sorted and unsorted data.
  • With the Hack: There is no difference between sorted and unsorted data.
  • In the C++ case, the hack is actually a tad slower than with the branch when the data is sorted.

A general rule of thumb is to avoid data-dependent branching in critical loops (such as in this example).


Update:

  • GCC 4.6.1 with -O3 or -ftree-vectorize on x64 is able to generate a conditional move. So there is no difference between the sorted and unsorted data - both are fast.

    (Or somewhat fast: for the already-sorted case, cmov can be slower especially if GCC puts it on the critical path instead of just add, especially on Intel before Broadwell where cmov has 2 cycle latency: gcc optimization flag -O3 makes code slower than -O2)

  • VC++ 2010 is unable to generate conditional moves for this branch even under /Ox.

  • Intel C++ Compiler (ICC) 11 does something miraculous. It interchanges the two loops, thereby hoisting the unpredictable branch to the outer loop. So not only is it immune to the mispredictions, it is also twice as fast as whatever VC++ and GCC can generate! In other words, ICC took advantage of the test-loop to defeat the benchmark...

  • If you give the Intel compiler the branchless code, it just out-right vectorizes it... and is just as fast as with the branch (with the loop interchange).

This goes to show that even mature modern compilers can vary wildly in their ability to optimize code...

By Daniel Fischer (Votes: 4153)

Branch prediction.

With a sorted array, the condition data[c] >= 128 is first false for a streak of values, then becomes true for all later values. That's easy to predict. With an unsorted array, you pay for the branching cost.

By WiSaGaN (Votes: 3358)

The reason why performance improves drastically when the data is sorted is that the branch prediction penalty is removed, as explained beautifully in Mysticial's answer.

Now, if we look at the code

if (data[c] >= 128)
    sum += data[c];

we can find that the meaning of this particular if... else... branch is to add something when a condition is satisfied. This type of branch can be easily transformed into a conditional move statement, which would be compiled into a conditional move instruction: cmovl, in an x86 system. The branch and thus the potential branch prediction penalty is removed.

In C, thus C++, the statement, which would compile directly (without any optimization) into the conditional move instruction in x86, is the ternary operator ... ? ... : .... So we rewrite the above statement into an equivalent one:

sum += data[c] >=128 ? data[c] : 0;

While maintaining readability, we can check the speedup factor.

On an Intel Core i7-2600K @ 3.4 GHz and Visual Studio 2010 Release Mode, the benchmark is (format copied from Mysticial):

x86

//  Branch - Random
seconds = 8.885

//  Branch - Sorted
seconds = 1.528

//  Branchless - Random
seconds = 3.716

//  Branchless - Sorted
seconds = 3.71

x64

//  Branch - Random
seconds = 11.302

//  Branch - Sorted
 seconds = 1.830

//  Branchless - Random
seconds = 2.736

//  Branchless - Sorted
seconds = 2.737

The result is robust in multiple tests. We get a great speedup when the branch result is unpredictable, but we suffer a little bit when it is predictable. In fact, when using a conditional move, the performance is the same regardless of the data pattern.

Now let's look more closely by investigating the x86 assembly they generate. For simplicity, we use two functions max1 and max2.

max1 uses the conditional branch if... else ...:

int max1(int a, int b) {
    if (a > b)
        return a;
    else
        return b;
}

max2 uses the ternary operator ... ? ... : ...:

int max2(int a, int b) {
    return a > b ? a : b;
}

On a x86-64 machine, GCC -S generates the assembly below.

:max1
    movl    %edi, -4(%rbp)
    movl    %esi, -8(%rbp)
    movl    -4(%rbp), %eax
    cmpl    -8(%rbp), %eax
    jle     .L2
    movl    -4(%rbp), %eax
    movl    %eax, -12(%rbp)
    jmp     .L4
.L2:
    movl    -8(%rbp), %eax
    movl    %eax, -12(%rbp)
.L4:
    movl    -12(%rbp), %eax
    leave
    ret

:max2
    movl    %edi, -4(%rbp)
    movl    %esi, -8(%rbp)
    movl    -4(%rbp), %eax
    cmpl    %eax, -8(%rbp)
    cmovge  -8(%rbp), %eax
    leave
    ret

max2 uses much less code due to the usage of instruction cmovge. But the real gain is that max2 does not involve branch jumps, jmp, which would have a significant performance penalty if the predicted result is not right.

So why does a conditional move perform better?

In a typical x86 processor, the execution of an instruction is divided into several stages. Roughly, we have different hardware to deal with different stages. So we do not have to wait for one instruction to finish to start a new one. This is called pipelining.

In a branch case, the following instruction is determined by the preceding one, so we cannot do pipelining. We have to either wait or predict.

In a conditional move case, the execution conditional move instruction is divided into several stages, but the earlier stages like Fetch and Decode does not depend on the result of the previous instruction; only latter stages need the result. Thus, we wait a fraction of one instruction's execution time. This is why the conditional move version is slower than the branch when prediction is easy.

The book Computer Systems: A Programmer's Perspective, second edition explains this in detail. You can check Section 3.6.6 for Conditional Move Instructions, entire Chapter 4 for Processor Architecture, and Section 5.11.2 for a special treatment for Branch Prediction and Misprediction Penalties.

Sometimes, some modern compilers can optimize our code to assembly with better performance, sometimes some compilers can't (the code in question is using Visual Studio's native compiler). Knowing the performance difference between branch and conditional move when unpredictable can help us write code with better performance when the scenario gets so complex that the compiler can not optimize them automatically.

How do I undo the most recent local commits in Git?

Question

I accidentally committed the wrong files to Git, but I haven't pushed the commit to the server yet.

How can I undo those commits from the local repository?

Answers

By Esko Luontola (Votes: 23261)

Undo a commit and redo

$ git commit -m "Something terribly misguided"             # (1)
$ git reset HEAD~                                          # (2)
<< edit files as necessary >>                              # (3)
$ git add ...                                              # (4)
$ git commit -c ORIG_HEAD                                  # (5)
  1. This is what you want to undo.
  2. This does nothing to your working tree (the state of your files on disk), but undoes the commit and leaves the changes you committed unstaged (so they'll appear as "Changes not staged for commit" in git status, so you'll need to add them again before committing). If you only want to add more changes to the previous commit, or change the commit message1, you could use git reset --soft HEAD~ instead, which is like git reset HEAD~2 but leaves your existing changes staged.
  3. Make corrections to working tree files.
  4. git add anything that you want to include in your new commit.
  5. Commit the changes, reusing the old commit message. reset copied the old head to .git/ORIG_HEAD; commit with -c ORIG_HEAD will open an editor, which initially contains the log message from the old commit and allows you to edit it. If you do not need to edit the message, you could use the -C option.

Beware, however, that if you have added any new changes to the index, using commit --amend will add them to your previous commit.

If the code is already pushed to your server and you have permissions to overwrite history (rebase) then:

git push origin master --force

You can also look at this answer:

How can I move HEAD back to a previous location? (Detached head) & Undo commits

The above answer will show you git reflog, which you can use to determine the SHA-1 for the commit to which you wish to revert. Once you have this value, use the sequence of commands as explained above.


1 Note, however, that you don't need to reset to an earlier commit if you just made a mistake in your commit message. The easier option is to git reset (to unstage any changes you've made since) and then git commit --amend, which will open your default commit message editor pre-populated with the last commit message.

2 HEAD~ is the same as HEAD~1. Also, see What is the HEAD in git?. It's helpful if you want to uncommit multiple commits.

By Ryan Lundy (Votes: 10860)

Undoing a commit is a little scary if you don't know how it works. But it's actually amazingly easy if you do understand.

Say you have this, where C is your HEAD and (F) is the state of your files.

   (F)
A-B-C
    ↑
  master

You want to nuke commit C and never see it again and lose all the changes in locally modified files. You do this:

git reset --hard HEAD~1

The result is:

 (F)
A-B
  ↑
master

Now B is the HEAD. Because you used --hard, your files are reset to their state at commit B.

Ah, but suppose commit C wasn't a disaster, but just a bit off. You want to undo the commit but keep your changes for a bit of editing before you do a better commit. Starting again from here, with C as your HEAD:

   (F)
A-B-C
    ↑
  master

You can do this, leaving off the --hard:

git reset HEAD~1

In this case the result is:

   (F)
A-B-C
  ↑
master

In both cases, HEAD is just a pointer to the latest commit. When you do a git reset HEAD~1, you tell Git to move the HEAD pointer back one commit. But (unless you use --hard) you leave your files as they were. So now git status shows the changes you had checked into C. You haven't lost a thing!

For the lightest touch, you can even undo your commit but leave your files and your index:

git reset --soft HEAD~1

This not only leaves your files alone, it even leaves your index alone. When you do git status, you'll see that the same files are in the index as before. In fact, right after this command, you could do git commit and you'd be redoing the same commit you just had.

One more thing: Suppose you destroy a commit as in the first example, but then discover you needed it after all? Tough luck, right?

Nope, there's still a way to get it back. Type git reflog and you'll see a list of (partial) commit shas (that is, hashes) that you've moved around in. Find the commit you destroyed, and do this:

git checkout -b someNewBranchName shaYouDestroyed

You've now resurrected that commit. Commits don't actually get destroyed in Git for some 90 days, so you can usually go back and rescue one you didn't mean to get rid of.

By Andrew (Votes: 2185)

There are two ways to "undo" your last commit, depending on whether or not you have already made your commit public (pushed to your remote repository):

How to undo a local commit

Let's say I committed locally, but now I want to remove that commit.

git log
    commit 101: bad commit    # Latest commit. This would be called 'HEAD'.
    commit 100: good commit   # Second to last commit. This is the one we want.

To restore everything back to the way it was prior to the last commit, we need to reset to the commit before HEAD:

git reset --soft HEAD^     # Use --soft if you want to keep your changes
git reset --hard HEAD^     # Use --hard if you don't care about keeping the changes you made

Now git log will show that our last commit has been removed.

How to undo a public commit

If you have already made your commits public, you will want to create a new commit which will "revert" the changes you made in your previous commit (current HEAD).

git revert HEAD

Your changes will now be reverted and ready for you to commit:

git commit -m 'restoring the file I removed by accident'
git log
    commit 102: restoring the file I removed by accident
    commit 101: removing a file we don't need
    commit 100: adding a file that we need

For more information, check out Git Basics - Undoing Things.

How do I delete a Git branch locally and remotely?

Question

I want to delete a branch both locally and remotely.

Failed Attempts to Delete a Remote Branch

git branch -d remotes/origin/bugfix

error: branch 'remotes/origin/bugfix' not found.

git branch -d origin/bugfix

error: branch 'origin/bugfix' not found.

git branch -rd origin/bugfix

Deleted remote branch origin/bugfix (was 2a14ef7).

git push

Everything up-to-date

git pull

From github.com:gituser/gitproject

* [new branch] bugfix -> origin/bugfix

Already up-to-date.

What should I do differently to successfully delete the remotes/origin/bugfix branch both locally and remotely?

Answers

By Matthew Rankin (Votes: 21677)

Executive Summary

$ git push -d <remote_name> <branch_name>
$ git branch -d <branch_name>

Note that in most cases the remote name is origin. In such a case you'll have to use the command like so.

$ git push -d origin <branch_name>

Delete Local Branch

To delete the local branch use one of the following:

$ git branch -d branch_name
$ git branch -D branch_name

Note: The -d option is an alias for --delete, which only deletes the branch if it has already been fully merged in its upstream branch. You could also use -D, which is an alias for --delete --force, which deletes the branch "irrespective of its merged status." [Source: man git-branch]

Delete Remote Branch [Updated on 8-Sep-2017]

As of Git v1.7.0, you can delete a remote branch using

$ git push <remote_name> --delete <branch_name>

which might be easier to remember than

$ git push <remote_name> :<branch_name>

which was added in Git v1.5.0 "to delete a remote branch or a tag."

Starting on Git v2.8.0 you can also use git push with the -d option as an alias for --delete.

Therefore, the version of Git you have installed will dictate whether you need to use the easier or harder syntax.

Delete Remote Branch [Original Answer from 5-Jan-2010]

From Chapter 3 of Pro Git by Scott Chacon:

Deleting Remote Branches

Suppose you’re done with a remote branch — say, you and your collaborators are finished with a feature and have merged it into your remote’s master branch (or whatever branch your stable code-line is in). You can delete a remote branch using the rather obtuse syntax git push [remotename] :[branch]. If you want to delete your server-fix branch from the server, you run the following:

$ git push origin :serverfix
To [email protected]:schacon/simplegit.git
 - [deleted]         serverfix

Boom. No more branches on your server. You may want to dog-ear this page, because you’ll need that command, and you’ll likely forget the syntax. A way to remember this command is by recalling the git push [remotename] [localbranch]:[remotebranch] syntax that we went over a bit earlier. If you leave off the [localbranch] portion, then you’re basically saying, “Take nothing on my side and make it be [remotebranch].”

I issued git push origin: bugfix and it worked beautifully. Scott Chacon was right—I will want to dog ear that page (or virtually dog ear by answering this on Stack Overflow).

Then you should execute this on other machines

# Fetch changes from all remotes and locally delete 
# remote deleted branches/tags etc
# --prune will do the job :-;
git fetch --all --prune

to propagate changes.

By Eric Brotto (Votes: 3349)

Matthew's answer is great for removing remote branches and I also appreciate the explanation, but to make a simple distinction between the two commands:

To remove a local branch from your machine:

git branch -d {the_local_branch} (use -D instead to force deleting the branch without checking merged status)

To remove a remote branch from the server:

git push origin --delete {the_remote_branch}

Reference: Git: Delete a branch (local or remote)

By user456814 (Votes: 2080)

The short answers

If you want more detailed explanations of the following commands, then see the long answers in the next section.

Deleting a remote branch

git push origin --delete <branch>  # Git version 1.7.0 or newer
git push origin -d <branch>        # Shorter version (Git 1.7.0 or newer)
git push origin :<branch>          # Git versions older than 1.7.0

Deleting a local branch

git branch --delete <branch>
git branch -d <branch> # Shorter version
git branch -D <branch> # Force-delete un-merged branches

Deleting a local remote-tracking branch

git branch --delete --remotes <remote>/<branch>
git branch -dr <remote>/<branch> # Shorter

git fetch <remote> --prune # Delete multiple obsolete remote-tracking branches
git fetch <remote> -p      # Shorter

The long answer: there are three different branches to delete!

When you're dealing with deleting branches both locally and remotely, keep in mind that there are three different branches involved:

  1. The local branch X.
  2. The remote origin branch X.
  3. The local remote-tracking branch origin/X that tracks the remote branch X.

Visualization of three branches

The original poster used:

git branch -rd origin/bugfix

Which only deleted his local remote-tracking branch origin/bugfix, and not the actual remote branch bugfix on origin.

Diagram 2

To delete that actual remote branch, you need

git push origin --delete bugfix

Diagram 3

Additional details

The following sections describe additional details to consider when deleting your remote and remote-tracking branches.

Pushing to delete remote branches also removes remote-tracking branches

Note that deleting the remote branch X from the command line using a git push will also remove the local remote-tracking branch origin/X, so it is not necessary to prune the obsolete remote-tracking branch with git fetch --prune or git fetch -p. However, it wouldn't hurt if you did it anyway.

You can verify that the remote-tracking branch origin/X was also deleted by running the following:

# View just remote-tracking branches
git branch --remotes
git branch -r

# View both strictly local as well as remote-tracking branches
git branch --all
git branch -a

Pruning the obsolete local remote-tracking branch origin/X

If you didn't delete your remote branch X from the command line (like above), then your local repository will still contain (a now obsolete) remote-tracking branch origin/X. This can happen if you deleted a remote branch directly through GitHub's web interface, for example.

A typical way to remove these obsolete remote-tracking branches (since Git version 1.6.6) is to simply run git fetch with the --prune or shorter -p. Note that this removes all obsolete local remote-tracking branches for any remote branches that no longer exist on the remote:

git fetch origin --prune
git fetch origin -p # Shorter

Here is the relevant quote from the 1.6.6 release notes (emphasis mine):

"git fetch" learned --all and --multipleoptions, to run fetch from many repositories, and --prune option to remove remote tracking branches that went stale. These make "git remote update" and "git remote prune" less necessary (there is no plan to remove "remote update" nor "remote prune", though).

Alternative to above automatic pruning for obsolete remote-tracking branches

Alternatively, instead of pruning your obsolete local remote-tracking branches through git fetch -p, you can avoid making the extra network operation by just manually removing the branch(es) with the --remote or -r flags:

git branch --delete --remotes origin/X
git branch -dr origin/X # Shorter

See Also

@neverendingqs
Copy link
Owner Author

fdasfasd

@neverendingqs
Copy link
Owner Author

/so [javascript] array sort

@github-actions
Copy link
Contributor

Results for [javascript] array sort

Sort array of objects by string property value

Question

I have an array of JavaScript objects:

var objs = [ 
    { first_nom: 'Lazslo', last_nom: 'Jamf'     },
    { first_nom: 'Pig',    last_nom: 'Bodine'   },
    { first_nom: 'Pirate', last_nom: 'Prentice' }
];

How can I sort them by the value of last_nom in JavaScript?

I know about sort(a,b), but that only seems to work on strings and numbers. Do I need to add a toString() method to my objects?

Answers

By Wogan (Votes: 4049)

It's easy enough to write your own comparison function:

function compare( a, b ) {
  if ( a.last_nom < b.last_nom ){
    return -1;
  }
  if ( a.last_nom > b.last_nom ){
    return 1;
  }
  return 0;
}

objs.sort( compare );

Or inline (c/o Marco Demaio):

objs.sort((a,b) => (a.last_nom > b.last_nom) ? 1 : ((b.last_nom > a.last_nom) ? -1 : 0)); 

By Ege &#214;zcan (Votes: 859)

You can also create a dynamic sort function that sorts objects by their value that you pass:

function dynamicSort(property) {
    var sortOrder = 1;
    if(property[0] === "-") {
        sortOrder = -1;
        property = property.substr(1);
    }
    return function (a,b) {
        /* next line works with strings and numbers, 
         * and you may want to customize it to your needs
         */
        var result = (a[property] < b[property]) ? -1 : (a[property] > b[property]) ? 1 : 0;
        return result * sortOrder;
    }
}

So you can have an array of objects like this:

var People = [
    {Name: "Name", Surname: "Surname"},
    {Name:"AAA", Surname:"ZZZ"},
    {Name: "Name", Surname: "AAA"}
];

...and it will work when you do:

People.sort(dynamicSort("Name"));
People.sort(dynamicSort("Surname"));
People.sort(dynamicSort("-Surname"));

Actually this already answers the question. Below part is written because many people contacted me, complaining that it doesn't work with multiple parameters.

Multiple Parameters

You can use the function below to generate sort functions with multiple sort parameters.

function dynamicSortMultiple() {
    /*
     * save the arguments object as it will be overwritten
     * note that arguments object is an array-like object
     * consisting of the names of the properties to sort by
     */
    var props = arguments;
    return function (obj1, obj2) {
        var i = 0, result = 0, numberOfProperties = props.length;
        /* try getting a different result from 0 (equal)
         * as long as we have extra properties to compare
         */
        while(result === 0 && i < numberOfProperties) {
            result = dynamicSort(props[i])(obj1, obj2);
            i++;
        }
        return result;
    }
}

Which would enable you to do something like this:

People.sort(dynamicSortMultiple("Name", "-Surname"));

Subclassing Array

For the lucky among us who can use ES6, which allows extending the native objects:

class MyArray extends Array {
    sortBy(...args) {
        return this.sort(dynamicSortMultiple.apply(null, args));
    }
}

That would enable this:

MyArray.from(People).sortBy("Name", "-Surname");

By Vlad Bezden (Votes: 455)

In ES6/ES2015 or later you can do this way:

objs.sort((a, b) => a.last_nom.localeCompare(b.last_nom));

Prior to ES6/ES2015

objs.sort(function(a, b) {
    return a.last_nom.localeCompare(b.last_nom)
});

Sorting an array of objects by property values

Question

I've got the following objects using AJAX and stored them in an array:

var homes = [
    {
        "h_id": "3",
        "city": "Dallas",
        "state": "TX",
        "zip": "75201",
        "price": "162500"
    }, {
        "h_id": "4",
        "city": "Bevery Hills",
        "state": "CA",
        "zip": "90210",
        "price": "319250"
    }, {
        "h_id": "5",
        "city": "New York",
        "state": "NY",
        "zip": "00010",
        "price": "962500"
    }
];

How do I create a function to sort the objects by the price property in ascending or descending order using JavaScript only?

Answers

By Stobor (Votes: 1715)

Sort homes by price in ascending order:

homes.sort(function(a, b) {
    return parseFloat(a.price) - parseFloat(b.price);
});

Or after ES6 version:

homes.sort((a, b) => parseFloat(a.price) - parseFloat(b.price));

Some documentation can be found here.

For descending order, you may use

homes.sort((a, b) => parseFloat(b.price) - parseFloat(a.price));

By Triptych (Votes: 676)

Here's a more flexible version, which allows you to create reusable sort functions, and sort by any field.

const sort_by = (field, reverse, primer) => {

const key = primer ?
function(x) {
return primer(x[field])
} :
function(x) {
return x[field]
};

reverse = !reverse ? 1 : -1;

return function(a, b) {
return a = key(a), b = key(b), reverse * ((a > b) - (b > a));
}
}

//Now you can sort by any field at will...

const homes=[{h_id:"3",city:"Dallas",state:"TX",zip:"75201",price:"162500"},{h_id:"4",city:"Bevery Hills",state:"CA",zip:"90210",price:"319250"},{h_id:"5",city:"New York",state:"NY",zip:"00010",price:"962500"}];

// Sort by price high to low
console.log(homes.sort(sort_by('price', true, parseInt)));

// Sort by city, case-insensitive, A-Z
console.log(homes.sort(sort_by('city', false, (a) => a.toUpperCase()
)));

By Ricardo Marimon (Votes: 134)

To sort it you need to create a comparator function taking two arguments. Then call the sort function with that comparator function as follows:

// a and b are object elements of your array
function mycomparator(a,b) {
  return parseInt(a.price, 10) - parseInt(b.price, 10);
}
homes.sort(mycomparator);

If you want to sort ascending switch the expressions on each side of the minus sign.

How to sort an array of integers correctly

Question

Trying to get the highest and lowest value from an array that I know will contain only integers seems to be harder than I thought.

var numArray = [140000, 104, 99];
numArray = numArray.sort();
console.log(numArray)

I'd expect this to show 99, 104, 140000. Instead it shows 104, 140000, 99. So it seems the sort is handling the values as strings.

Is there a way to get the sort function to actually sort on integer value?

Answers

By aks (Votes: 1281)

By default, the sort method sorts elements alphabetically. To sort numerically just add a new method which handles numeric sorts (sortNumber, shown below) -

var numArray = [140000, 104, 99];
numArray.sort(function(a, b) {
  return a - b;
});

console.log(numArray);

In ES6, you can simplify this with arrow functions:

numArray.sort((a, b) => a - b); // For ascending sort
numArray.sort((a, b) => b - a); // For descending sort

Documentation:

Mozilla Array.prototype.sort() recommends this compare function for arrays that don't contain Infinity or NaN. (Because Inf - Inf is NaN, not 0).

Also examples of sorting objects by key.

By MarzSocks (Votes: 176)

Just building on all of the above answers, they can also be done in one line like this:

var numArray = [140000, 104, 99];

// ES5
numArray = numArray.sort(function (a, b) {  return a - b;  });

// ES2015
numArray = numArray.sort((a, b) => a - b);

//outputs: 99, 104, 140000

By Paul Dixon (Votes: 73)

array.sort does a lexicographic sort by default, for a numeric sort, provide your own function. Here's a simple example:

function compareNumbers(a, b)
{
    return a - b;
}

numArray.sort(compareNumbers);

Also note that sort works "in place", there's no need for the assignment.

@neverendingqs neverendingqs force-pushed the master branch 2 times, most recently from 1c6148f to ddc9d73 Compare August 23, 2020 02:23
@neverendingqs
Copy link
Owner Author

/so [github-actions] hub

@github-actions
Copy link
Contributor

Results for [github-actions] hub

Using elm-test in a git hub action (Votes: 2)

Asked by Ackdari.

Question

I want to use the an git hub action to test and build my elm package whenever a commit is pushed to the master branch for this my action .yml file looks like this

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - run: |
        sudo npm install -g elm-test # this fails
        elm-test
        elm make

for testing I want to use elm-test which can be installed via npm but the command sudo npm install -g elm-test fails with

/usr/local/bin/elm-test -> /usr/local/lib/node_modules/elm-test/bin/elm-test

> [email protected] install /usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json
> binwrap-install

ERR Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json/unpacked_bin'
    at Object.mkdirSync (fs.js:823:3)
    at /usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:46:10
    at new Promise (<anonymous>)
    at untgz (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:21:10)
    at binstall (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:11:12)
    at install (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/install.js:20:10)
    at Object.install (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/index.js:14:14)
    at Object.<anonymous> (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/bin/binwrap-install:18:9)
    at Module._compile (internal/modules/cjs/loader.js:955:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:991:10)
    at Module.load (internal/modules/cjs/loader.js:811:32)
    at Function.Module._load (internal/modules/cjs/loader.js:723:14)
    at Function.Module.runMain (internal/modules/cjs/loader.js:1043:10)
    at internal/main/run_main_module.js:17:11 {
  errno: -13,
  syscall: 'mkdir',
  code: 'EACCES',
  path: '/usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json/unpacked_bin'
}
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/elm-test/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `binwrap-install`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2020-02-03T17_50_06_232Z-debug.log

Any advice on how to install elm-test inside a git hub action?

Edit: Without the sudo the error becomes

npm WARN checkPermissions Missing write access to /usr/local/lib/node_modules
npm ERR! code EACCES
npm ERR! syscall access
npm ERR! path /usr/local/lib/node_modules
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, access '/usr/local/lib/node_modules'
npm ERR!  [Error: EACCES: permission denied, access '/usr/local/lib/node_modules'] {
npm ERR!   stack: "Error: EACCES: permission denied, access '/usr/local/lib/node_modules'",
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'access',
npm ERR!   path: '/usr/local/lib/node_modules'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2020-02-04T13_41_34_534Z-debug.log
Answers

By Ackdari (Votes: 3)

Thanks to @toastal comment I found another solution. That is to set up a package.json file using npm and then adding elm-test as a dependency with the command

npm install -D elm-test

also you might want to add node_modules to your .gitignore to ignore npms installation folder.

Then in the yml file you can just run the command npm install and elm-test gets installed. Then you can call it with

./node_modules/elm-test/bin/elm-test

My yml file now looks like this

name: Tests

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - name: install npm dependencies
      run: npm install
    - name: Test
      run: ./node_modules/elm-test/bin/elm-test
    - name: check documentation
      run: |
        elm make --docs=docs.json
        rm docs.json

By Ackdari (Votes: 2)

Thanks to @glennsl I found an solution. The solution is to specify where npm installs package globally. Since the runner that runs the action has no access to the /usr/local/lib npm can't install anything on a global level. The solution (as described here) is to create an folder as a "global" installation folder and configuring npm to use it. It is also necessary to add the new folder to the PATH environment variable.

So my yml file now looks like this

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - run: |
        mkdir ~/.npm-global
        npm config set prefix '~/.npm-global'
        PATH=~/.npm-global/bin:$PATH
        npm install -g elm-test
        elm-test
        elm make

Where should you put the git hub workflow directory for actions in a full-stack project? (Votes: 1)

Asked by Austin S.

Question

I am building a full-stack react project with a node backend and would like to implement a git hub actions to automate pull requests for continuous integration. Currently, in my root directory, I have my project split into two directories a front-end directory holding the client-side application and a back-end directory container the serverside. My question is where should I place the .github directory and associated .workflow directory in the project? At the root level or in either the client-side or server-side directories?

enter image description here

Answers

By Lucas Golven (Votes: 1)

From Github documentation:

You must store workflows in the .github/workflows directory in the root of your repository.

(workflow file cannot be inside subdirectory, it must be in .github/workflows)

You can define a naming convention for your repository: To keep the same logic that you were trying to implement with the subfolders, you could have .github/workflows/economic_growth.back-end.yaml and .github/workflows/economic_growth.front-end.yaml.

Can I run a docker container with GitHub actions? (Votes: 1)

Asked by Bruno Brant.

Question

I'm want to perform CI with tests that depends on a container that is published in Docker Hub. Is it possible? Can I start the container and run tests that depends on it?

Answers

By scthi (Votes: 4)

Yes, here is an example using docker-compose:

  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v1

      - name: Login to Docker Registry
        run: docker login "$DOCKER_REGISTRY" -u "$DOCKER_USERNAME" --password-stdin <<< "$DOCKER_PASSWORD"
        env:
          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
          DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
      - name: Start env
        run: docker-compose up
      - name: Run tests
        run: ...

@neverendingqs
Copy link
Owner Author

This can be used even if the pull request is closed, so closing it for now.

@neverendingqs neverendingqs deleted the chore/testprmegathread branch August 31, 2020 17:40
@neverendingqs
Copy link
Owner Author

/so [github-actions] hub

@github-actions
Copy link
Contributor

Results for [github-actions] hub

Using elm-test in a git hub action (Votes: 2)

Asked by Ackdari.

Question

I want to use the an git hub action to test and build my elm package whenever a commit is pushed to the master branch for this my action .yml file looks like this

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - run: |
        sudo npm install -g elm-test # this fails
        elm-test
        elm make

for testing I want to use elm-test which can be installed via npm but the command sudo npm install -g elm-test fails with

/usr/local/bin/elm-test -> /usr/local/lib/node_modules/elm-test/bin/elm-test

> [email protected] install /usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json
> binwrap-install

ERR Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json/unpacked_bin'
    at Object.mkdirSync (fs.js:823:3)
    at /usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:46:10
    at new Promise (<anonymous>)
    at untgz (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:21:10)
    at binstall (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:11:12)
    at install (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/install.js:20:10)
    at Object.install (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/index.js:14:14)
    at Object.<anonymous> (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/bin/binwrap-install:18:9)
    at Module._compile (internal/modules/cjs/loader.js:955:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:991:10)
    at Module.load (internal/modules/cjs/loader.js:811:32)
    at Function.Module._load (internal/modules/cjs/loader.js:723:14)
    at Function.Module.runMain (internal/modules/cjs/loader.js:1043:10)
    at internal/main/run_main_module.js:17:11 {
  errno: -13,
  syscall: 'mkdir',
  code: 'EACCES',
  path: '/usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json/unpacked_bin'
}
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/elm-test/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `binwrap-install`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2020-02-03T17_50_06_232Z-debug.log

Any advice on how to install elm-test inside a git hub action?

Edit: Without the sudo the error becomes

npm WARN checkPermissions Missing write access to /usr/local/lib/node_modules
npm ERR! code EACCES
npm ERR! syscall access
npm ERR! path /usr/local/lib/node_modules
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, access '/usr/local/lib/node_modules'
npm ERR!  [Error: EACCES: permission denied, access '/usr/local/lib/node_modules'] {
npm ERR!   stack: "Error: EACCES: permission denied, access '/usr/local/lib/node_modules'",
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'access',
npm ERR!   path: '/usr/local/lib/node_modules'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2020-02-04T13_41_34_534Z-debug.log
Answers

By Ackdari (Votes: 3)

Thanks to `@toastal` comment I found another solution. That is to set up a package.json file using npm and then adding elm-test as a dependency with the command

npm install -D elm-test

also you might want to add node_modules to your .gitignore to ignore npms installation folder.

Then in the yml file you can just run the command npm install and elm-test gets installed. Then you can call it with

./node_modules/elm-test/bin/elm-test

My yml file now looks like this

name: Tests

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - name: install npm dependencies
      run: npm install
    - name: Test
      run: ./node_modules/elm-test/bin/elm-test
    - name: check documentation
      run: |
        elm make --docs=docs.json
        rm docs.json

By Ackdari (Votes: 2)

Thanks to `@glennsl` I found an solution. The solution is to specify where npm installs package globally. Since the runner that runs the action has no access to the /usr/local/lib npm can't install anything on a global level. The solution (as described here) is to create an folder as a "global" installation folder and configuring npm to use it. It is also necessary to add the new folder to the PATH environment variable.

So my yml file now looks like this

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - run: |
        mkdir ~/.npm-global
        npm config set prefix '~/.npm-global'
        PATH=~/.npm-global/bin:$PATH
        npm install -g elm-test
        elm-test
        elm make

Where should you put the git hub workflow directory for actions in a full-stack project? (Votes: 1)

Asked by Austin S.

Question

I am building a full-stack react project with a node backend and would like to implement a git hub actions to automate pull requests for continuous integration. Currently, in my root directory, I have my project split into two directories a front-end directory holding the client-side application and a back-end directory container the serverside. My question is where should I place the .github directory and associated .workflow directory in the project? At the root level or in either the client-side or server-side directories?

enter image description here

Answers

By Lucas Golven (Votes: 1)

From Github documentation:

You must store workflows in the .github/workflows directory in the root of your repository.

(workflow file cannot be inside subdirectory, it must be in .github/workflows)

You can define a naming convention for your repository: To keep the same logic that you were trying to implement with the subfolders, you could have .github/workflows/economic_growth.back-end.yaml and .github/workflows/economic_growth.front-end.yaml.

Can I run a docker container with GitHub actions? (Votes: 1)

Asked by Bruno Brant.

Question

I'm want to perform CI with tests that depends on a container that is published in Docker Hub. Is it possible? Can I start the container and run tests that depends on it?

Answers

By scthi (Votes: 4)

Yes, here is an example using docker-compose:

  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v1

      - name: Login to Docker Registry
        run: docker login "$DOCKER_REGISTRY" -u "$DOCKER_USERNAME" --password-stdin <<< "$DOCKER_PASSWORD"
        env:
          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
          DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
      - name: Start env
        run: docker-compose up
      - name: Run tests
        run: ...

@neverendingqs
Copy link
Owner Author

/so [github-actions] hub

@github-actions
Copy link
Contributor

github-actions bot commented Sep 1, 2020

Results for [github-actions] hub

Using elm-test in a git hub action (Votes: 2)

Asked by Ackdari.

Question

I want to use the an git hub action to test and build my elm package whenever a commit is pushed to the master branch for this my action .yml file looks like this

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - run: |
        sudo npm install -g elm-test # this fails
        elm-test
        elm make

for testing I want to use elm-test which can be installed via npm but the command sudo npm install -g elm-test fails with

/usr/local/bin/elm-test -> /usr/local/lib/node_modules/elm-test/bin/elm-test

> [email protected] install /usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json
> binwrap-install

ERR Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json/unpacked_bin'
    at Object.mkdirSync (fs.js:823:3)
    at /usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:46:10
    at new Promise (<anonymous>)
    at untgz (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:21:10)
    at binstall (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/binstall.js:11:12)
    at install (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/install.js:20:10)
    at Object.install (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/index.js:14:14)
    at Object.<anonymous> (/usr/local/lib/node_modules/elm-test/node_modules/binwrap/bin/binwrap-install:18:9)
    at Module._compile (internal/modules/cjs/loader.js:955:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:991:10)
    at Module.load (internal/modules/cjs/loader.js:811:32)
    at Function.Module._load (internal/modules/cjs/loader.js:723:14)
    at Function.Module.runMain (internal/modules/cjs/loader.js:1043:10)
    at internal/main/run_main_module.js:17:11 {
  errno: -13,
  syscall: 'mkdir',
  code: 'EACCES',
  path: '/usr/local/lib/node_modules/elm-test/node_modules/elmi-to-json/unpacked_bin'
}
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/elm-test/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `binwrap-install`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2020-02-03T17_50_06_232Z-debug.log

Any advice on how to install elm-test inside a git hub action?

Edit: Without the sudo the error becomes

npm WARN checkPermissions Missing write access to /usr/local/lib/node_modules
npm ERR! code EACCES
npm ERR! syscall access
npm ERR! path /usr/local/lib/node_modules
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, access '/usr/local/lib/node_modules'
npm ERR!  [Error: EACCES: permission denied, access '/usr/local/lib/node_modules'] {
npm ERR!   stack: "Error: EACCES: permission denied, access '/usr/local/lib/node_modules'",
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'access',
npm ERR!   path: '/usr/local/lib/node_modules'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/runner/.npm/_logs/2020-02-04T13_41_34_534Z-debug.log
Answers

By Ackdari (Votes: 3)

Thanks to `@toastal` comment I found another solution. That is to set up a package.json file using npm and then adding elm-test as a dependency with the command

npm install -D elm-test

also you might want to add node_modules to your .gitignore to ignore npms installation folder.

Then in the yml file you can just run the command npm install and elm-test gets installed. Then you can call it with

./node_modules/elm-test/bin/elm-test

My yml file now looks like this

name: Tests

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - name: install npm dependencies
      run: npm install
    - name: Test
      run: ./node_modules/elm-test/bin/elm-test
    - name: check documentation
      run: |
        elm make --docs=docs.json
        rm docs.json

By Ackdari (Votes: 2)

Thanks to `@glennsl` I found an solution. The solution is to specify where npm installs package globally. Since the runner that runs the action has no access to the /usr/local/lib npm can't install anything on a global level. The solution (as described here) is to create an folder as a "global" installation folder and configuring npm to use it. It is also necessary to add the new folder to the PATH environment variable.

So my yml file now looks like this

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Elm environment
      uses: JorelAli/setup-elm@v1
      with:
        # Version of Elm to use. E.g. 0.19.1
        elm-version: 0.19.1
    - run: |
        mkdir ~/.npm-global
        npm config set prefix '~/.npm-global'
        PATH=~/.npm-global/bin:$PATH
        npm install -g elm-test
        elm-test
        elm make

Where should you put the git hub workflow directory for actions in a full-stack project? (Votes: 1)

Asked by Austin S.

Question

I am building a full-stack react project with a node backend and would like to implement a git hub actions to automate pull requests for continuous integration. Currently, in my root directory, I have my project split into two directories a front-end directory holding the client-side application and a back-end directory container the serverside. My question is where should I place the .github directory and associated .workflow directory in the project? At the root level or in either the client-side or server-side directories?

enter image description here

Answers

By Lucas Golven (Votes: 1)

From Github documentation:

You must store workflows in the .github/workflows directory in the root of your repository.

(workflow file cannot be inside subdirectory, it must be in .github/workflows)

You can define a naming convention for your repository: To keep the same logic that you were trying to implement with the subfolders, you could have .github/workflows/economic_growth.back-end.yaml and .github/workflows/economic_growth.front-end.yaml.

Can I run a docker container with GitHub actions? (Votes: 1)

Asked by Bruno Brant.

Question

I'm want to perform CI with tests that depends on a container that is published in Docker Hub. Is it possible? Can I start the container and run tests that depends on it?

Answers

By scthi (Votes: 4)

Yes, here is an example using docker-compose:

  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v1

      - name: Login to Docker Registry
        run: docker login "$DOCKER_REGISTRY" -u "$DOCKER_USERNAME" --password-stdin <<< "$DOCKER_PASSWORD"
        env:
          DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
          DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
          DOCKER_REGISTRY: ${{ secrets.DOCKER_REGISTRY }}
      - name: Start env
        run: docker-compose up
      - name: Run tests
        run: ...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant