lfs server usage

[mschubert]

Hi,

I'm playing around with the lfs server a bit, and I can't seem to make it work as I would expect: my lfs files are commited to the git repository instead. This is most likely me not understanding how to use the lfs server correctly, so please tell me if I'm doing something obviously wrong.

My setup is the following:

• Create a repository on Github, clone it locally
• Set up lfs to track all files ending in *.lfs

git lfs track "*.lfs"

• Start the lfs-test-server

LFS_ADMINUSER=test LFS_ADMINPASS=test lfs-test-server

• Add my [user] + password using the /mgmt interface
• Set up .gitconfig in the local repository for my lfs test server.

[lfs]
url = "http://localhost:8080/myuser"

• Add an lfs-tracked file

echo > mytest.lfs
git add mytest.lfs
git commit -m "add lfs-tracked file"
git push origin master

The expected result here is that lfs tracks the file and adds it to the lfs server directory.
Instead, it is added to the git repository as a normal, git-tracked file.

[shiftkey]

@mschubert thanks for the detailed repro. Could you tell me if you've configured the .gitattributes file for the filters, as per the Configuration section of the README?

*.lfs filter=lfs -crlf

Without that section, the filters probably won't run, which sounds like what is happening here.

[mschubert]

As far as I understood, git lfs track "*.lfs" will create the required .gitattributes file. - Mine looks exactly like your quote.

[shiftkey]

@mschubert good to know. Does git lfs ls-files include your file as well?

[mschubert]

The output of git lfs ls-files is empty.

I found an error in my tracking of files (no quotes). I still can't push, will look if solution for @mkulagowski solves my problem as well.

(@shiftkey, note the update here)

[shiftkey]

My gut feel at this point is that this is a git-lfs problem, rather than related to the test server itself.

[mschubert]

Yes, you are right. The file not being listed makes it a git-lfs issue most likely. I will try figure out what's going wrong on their end, and the retry running the server.

[mkulagowski]

Hey, I didn't want to open new issue with the same name, so I write here.
I'm trying to get git-lfs working with lfs-test-server, yet to no avail.
Unlike the most I tried to make it work the normal way - with VPS instead of locally :P

So...I downloaded 64bit test-serv binary to my VPS, changed LFS_HOST="vps_host_name:8080" and started it. All was done through ssh.

Then I downloaded git-lfs locally, git init, git lfs init and changed .gitattributes to track single file (it was made through git lfs track "file.zip"). I've also changed .git/config file by adding:
[lfs] url = "http://vps_host_name:8080/mkkulagowski/folder_I_created_just_for_this"

Next I git add file.zip, commit it and push...aaand the grand finale:
gerrit asks me for credentials :P, then vps asks for credentials - I give the same credentials, I used to connect through ssh, so they should be working...yet I get:
Authorization error: http://vps_host_name:8080/mkkulagowski/folder_I_created_just_for_this/objects Check that you have proper access to the repository

On VPS I get messages in lfs-serv:
2015-04-20T08:48:57Z vz31640 lfs[3167] [proc.go:63]: fn=main msg=listening pid=3167 addr=tcp://:8080 version=0.1.2

2015-04-20T08:49:50Z vz31640 lfs[3167] [server.go:262]: method=POST url=/mkkulagowski/gopath/objects status=401 request_id=64e49c46-7b9d-e6ed-4700-1883babf99af

2015-04-20T08:50:49Z vz31640 lfs[3167] [server.go:262]: request_id=c8b080f7-400e-6926-9edf-891f30f23db2 method=POST url=/mkkulagowski/gopath/objects status=401

Any ideas?

[rubyist]

@mkulagowski Did you add a user to the lfs server? Since it's just a little stand alone test thing it uses its own user database, separate from anything else on the system. If you start the server with the environment variables LFS_ADMINUSER and LFS_ADMINPASS you'll be able to load the management interface at http://vps_host_name:8080/mgmt in a browser. From there you can add users and set their passwords. These are the credentials you'll use when prompted for the credentials.

It's a little confusing, for sure, a real Git LFS backend will have better integration with the repository's own authentication. For example, when using Git LFS on GitHub, you'll just use your GitHub credentials throughout. Let me know if that doesn't work out for you!

[mkulagowski]

@rubyist Well...that kind of helped - there wasn't problem with credentials and while pushing I even caught a glimpse of loading bar :D
Unfortunately I got few errors:

SERVER:
2015-04-20T14:55:00Z vz31640 lfs[4413] [server.go:164]: method=POST url=/mkkulagowski/gopath/objects status=202 request_id=d2618a9a-081c-de0e-a359-8a26a0e45d8c

LOCAL OUTPUT:
Username for 'http://vps_host_name:8080': user
Password for 'http://user@vps_host_name:8080':
0 B / 3.43 KB [-------------------------------------------------------] 0.00 %
Error for PUT http://localhost:8080/mkkulagowski/gopath/objects/fe0701c5029fc3c637123f7ccc4114b719e959b4cfb115b54c0463d4a71f49d0

git lfs logs last:
LocalWorkingDir=D:\gfhgh
LocalGitDir=D:\gfhgh\.git
LocalMediaDir=D:\gfhgh\.git\lfs\objects
TempDir=D:\gfhgh\.git\lfs\tmp
GIT_PREFIX=

Error for PUT http://localhost:8080/mkkulagowski/gopath/objects/fe0701c5029fc3c637123f7ccc4114b719e959b4cfb115b54c0463d4a71f49d0

Put http://localhost:8080/mkkulagowski/gopath/objects/fe0701c5029fc3c637123f7ccc4114b719e959b4cfb115b54c0463d4a71f49d0: dial tcp 127.0.0.1:8080: ConnectEx tcp:
Nie można nawiązać połączenia, ponieważ komputer docelowy aktywnie go odmawia.
goroutine 16 [running]:
github.com/github/git-lfs/lfs.Stack(0x0, 0x0, 0x0)
/Users/rick/github/git-lfs/.vendor/src/github.com/github/git-lfs/lfs/errors.go:85 +0x90
github.com/github/git-lfs/commands.logPanic(0x333288, 0xc082057260, 0xc082055a00, 0x0, 0x0)
/Users/rick/github/git-lfs/.vendor/src/github.com/github/git-lfs/commands/commands.go:148 +0xf1e
github.com/github/git-lfs/commands.handlePanic(0x333288, 0xc082057260, 0x0, 0x0)
/Users/rick/github/git-lfs/.vendor/src/github.com/github/git-lfs/commands/commands.go:100 +0x73
github.com/github/git-lfs/commands.LoggedError(0x333288, 0xc082057260, 0xc08204c380, 0x80, 0x0, 0x0, 0x0)
/Users/rick/github/git-lfs/.vendor/src/github.com/github/git-lfs/commands/commands.go:65 +0x90
github.com/github/git-lfs/commands.Panic(0x333288, 0xc082057260, 0xc08204c380, 0x80, 0x0, 0x0, 0x0)
/Users/rick/github/git-lfs/.vendor/src/github.com/github/git-lfs/commands/commands.go:75 +0x6f
github.com/github/git-lfs/commands.pushCommand(0x9a5e60, 0xc082056cc0, 0x2, 0x3)
/Users/rick/github/git-lfs/.vendor/src/github.com/github/git-lfs/commands/command_push.go:120 +0x579
github.com/spf13/cobra.(*Command).execute(0x9a5e60, 0xc082004740, 0x3, 0x4, 0x0, 0x0)
/Users/rick/github/git-lfs/.vendor/src/github.com/spf13/cobra/command.go:355 +0x183
github.com/spf13/cobra.(*Command).findAndExecute(0x9a6560, 0xc08200a330, 0x4, 0x4, 0x0, 0x0)
/Users/rick/github/git-lfs/.vendor/src/github.com/spf13/cobra/command.go:334 +0xda
github.com/spf13/cobra.(*Command).Execute(0x9a6560, 0x0, 0x0)
/Users/rick/github/git-lfs/.vendor/src/github.com/spf13/cobra/command.go:408 +0x7b7
github.com/github/git-lfs/commands.Run()
/Users/rick/github/git-lfs/.vendor/src/github.com/github/git-lfs/commands/commands.go:80 +0x31
main.main()
/Users/rick/github/git-lfs/cmd/git-lfs.go:8 +0x21

This part:
Nie można nawiązać połączenia, ponieważ komputer docelowy aktywnie go odmawia. means in my language:
No connection could be made because the target machine actively refused it

Any ideas? Got 202 on server for POST...but nothing for PUT.

[rubyist]

@mkulagowski Hmm, it looks like for the PUT it was trying to connect to 127.0.0.1 instead of your vps host. When the client makes the POST, the server will hand back some json telling it where to do the PUT. This is where the LFS_HOST variable comes in. When you started the server back up were you still setting the LFS_HOST variable?

[mkulagowski]

@rubyist Thank You very much sir, I think we're at the finishing line:
LFS_HOST was indeed wrong, cause I used export from cmd line - after another logon it went back to default.

This time I changed it, started up lfs-test-server and pushed sth from my local machine and VOILA! It went through. I pushed a ~25Mb wmv file as well.
Yet there are things that I do not fully understand:
[1] When I upload a new file it asks me for credentials 3 times - queries were POST, POST, PUT (for a 10kb file it asked 2 times - POST, PUT). Can we count it as an issue :P ?

[2] Where can I find uploaded files? I can't see anything on the specified path on vps o.O, yet when I make git reset and then fetch commit with these files I get them in a snap and it doesn't ask me for credentials...I'm not sure if it actually went to my vps :/

Output from vps:
2015-04-22T06:15:47Z vz31640 lfs[16870] [proc.go:63]: fn=main msg=listening pid=16870 addr=tcp://:8080 version=0.1.2

2015-04-22T06:16:18Z vz31640 lfs[16870] [server.go:164]: url=/mkkulagowski/gopath/objects status=202 request_id=fda15df5-77c7-070c-8a6f-0a78a09fd53d method=POST
2015-04-22T06:16:27Z vz31640 lfs[16870] [server.go:186]: method=PUT url=/mkkulagowski/gopath/objects/fe0701c5029fc3c637123f7ccc4114b719e959b4cfb115b54c0463d4a71f49d0 status=200 request_id=1099580c-04a8-a626-efbc-c84bf24e741a

2015-04-22T06:18:05Z vz31640 lfs[16870] [server.go:164]: method=POST url=/mkkulagowski/gopath/objects status=200 request_id=cf17fa93-fac4-4dcd-34af-0aefff5dd49d
2015-04-22T06:18:09Z vz31640 lfs[16870] [server.go:164]: method=POST url=/mkkulagowski/gopath/objects status=202 request_id=35568656-ceac-405c-a2ac-d90726c14350
2015-04-22T06:19:17Z vz31640 lfs[16870] [server.go:186]: method=PUT url=/mkkulagowski/gopath/objects/05aefc9502534cac696a25e5020ae40abbc2a2bc918a9902a60673849a0857b4 status=200 request_id=291e3d9a-7f72-8dcf-00b4-bbbd8e20d485

[mkulagowski]

I guess I could've been ignored due to my issue being almost the same as the original issue from mschubert. Yet for me git lfs ls-files list files correctly so it may not be git-lfs's issue but this test-server's.

[rubyist]

Hi @mkulagowski - sorry, I totally missed this. For your questions:

1. The git lfs client can be a little repetitive asking for credentials right now, especially when using it against the test server. The client itself does not cache credentials, but uses git's credential cache helper. Do you have one configured? You can check by running git config credential.helper. There is an issue to make this a little more streamlined.
2. Unless you set LFS_CONTENTPATH then the uploaded content will be stored in whatever directory the test server was run from under a subdirectory called lfs-content. The reason it runs so faster after you do a reset and fetch is because the lfs objects are already cached in your repository under .git/lfs. We do this to avoid re-downloading files when changing branches and such. If you do a fresh clone of your repository you should see the files download (and see GET entries in the server log).

[mkulagowski]

@rubyist Ok, great - I've found those files 👍
One more question...what is the difference between LFS_CONTENTPATH and path from .git/config: 'http://vps_host_name:8080/path' ?

[rubyist]

LFS_CONTENTPATH is only telling the lfs-test-server where to store the uploaded content. You could stick it anywhere on your file system. The path part of the url in the config is designed to work with the API itself.

Technically, the lfs-test-server doesn't have to require a user/repo part of the path, the API really only defines the parts after that. It's really an artifact of and older setup we had, but it can also serve as a demonstration of how a server might use that kind of information for authentication or data partitioning. You could modify the server code to have more or no path pieces, and as long as the /objects/{oid} part stays intact at the end it will be compliant with the API.

For example, you could set that line to be:

route := "/objects/{oid}"

And then your git config could be:

[lfs]
    url = "http://vps_host:8080/"

Or:

route := "/repo/objects/{oid}"

And

[lfs]
    url = "http://vps_host:8080/myrepo"

Does that make sense? Feel free to keep asking questions if it doesn't, it's been a while since I've had to explain any of this to someone who hasn't been working on it for a while so keep prodding me if it doesn't make sense 😄

[mkulagowski]

@rubyist Well... I'm not yet sure I understand...
So right now I could set .git/config to:
[lfs]
url = "http://vps_host:8080/user1/repo2"
and LFS_CONTENTPATH to:
"/user2/repo3"
and my files would go to user2/repo3 right?

I understand the concept of CONTENTPATH, as it is simple ("show me folder to store files") yet I miss what is the current use of .git/config path and how it works alongside with LFS_CONTENTPATH :/

[rubyist]

@mkulagowski You're right about the LFS_CONTENTPATH, it's only telling the server where to store the files on disk. It is not related to the .git/config path in any way.

[marcpires]

I have configured the Test Server, but uploading a 71MB file is taking to long. Any suggestions ?

[rubyist]

@marcpiresrj What kind of set up are you using? Is the test server running on a separate host or locally? How long is it taking?