How to debug failed oxidized push to remote git repo

[ghost]

Hi,

I'm trying to debug a problem with Oxidized pushing to a remote GIT repository. Here are the relevant bits of my configuration:

output:
default: git
git:
user: "Matthew Watkins"
email: "[email protected]"
repo: "/var/lib/oxidized/oxidized.git"
hooks:
push_to_remote:
type: githubrepo
events: [post_store]
remote_repo: [email protected]:matthew.watkins/oxidized.git
username: git
publickey: /root/gitlab_id_rsa.pub
privatekey: /root/gitlab_id_rsa

When I run oxidized, the first device configuration retrieves successfully. This triggers a push to my remote repository, but the device configuration file never arrives, I don't see an update. The remote git repository is brand new and empty, containing no data.

Here's the standard log output:

root@debian-oxidized-git:~# tail -f log.txt

Logfile created on 2019-08-06 11:37:26 +0000 by logger.rb/61378

I, [2019-08-06T11:37:26.424762 #14] INFO -- : Oxidized starting, running as pid 14
I, [2019-08-06T11:37:26.426239 #14] INFO -- : lib/oxidized/nodes.rb: Loading nodes
I, [2019-08-06T11:37:26.542498 #14] INFO -- : lib/oxidized/nodes.rb: Loaded 159 nodes
I, [2019-08-06T11:37:35.862643 #14] INFO -- : Configuration updated for /172.20.6.48
I, [2019-08-06T11:37:35.862929 #14] INFO -- : GithubRepo: Pushing local repository(/var/lib/oxidized/oxidized.git/)...
I, [2019-08-06T11:37:35.863877 #14] INFO -- : GithubRepo: to remote: [email protected]:matthew.watkins/oxidized.git

I see no further activity. When I enable debugging in the Oxidized configuration file, and restart the process, I see the same line repeated over and over again, very rapidly. The failure seems to run in a tight loop.

D, [2019-08-06T11:29:03.469117 #17] DEBUG -- : GithubRepo: Authenticating using ssh keys as 'git'
D, [2019-08-06T11:29:03.477886 #17] DEBUG -- : GithubRepo: Authenticating using ssh keys as 'git'
D, [2019-08-06T11:29:03.487504 #17] DEBUG -- : GithubRepo: Authenticating using ssh keys as 'git'
D, [2019-08-06T11:29:03.496390 #17] DEBUG -- : GithubRepo: Authenticating using ssh keys as 'git'
D, [2019-08-06T11:29:03.505360 #17] DEBUG -- : GithubRepo: Authenticating using ssh keys as 'git'

I can see the local GIT repository is being created fine:

root@debian-oxidized-git:/var/lib/oxidized/oxidized.git# ls
HEAD config description hooks index info objects refs

This looks like a generic authentication failure, and I'm a bit worried that the tight loop could be generating load on either the network or the remote git server. If I clone the remote repository from my shell prompt, it completes successfully, saying I cloned an empty repository.

root@debian-oxidized-git:~# cd /tmp
root@debian-oxidized-git:/tmp# git clone [email protected]:matthew.watkins/oxidized.git
Cloning into 'oxidized'...
warning: You appear to have cloned an empty repository.

My local user account has an SSH config file as follows:

host gitlab-test.internaltest.com
HostName gitlab-test.internaltest.com
IdentityFile /root/gitlab_id_rsa
User git

host gitlab.internaltest.com
HostName gitlab.internaltest.com
IdentityFile /root/gitlab_id_rsa
User git

My SSH private key permissions are set appropriately (og-wrx).

Any ideas how I might be able to debug this further?

Thanks in advance for any help...

[tyler-8]

Check that your keys have been generated as true RSA/PEM private keys.

https://serverfault.com/questions/939909/ssh-keygen-does-not-create-rsa-private-key

Converting my openssh key to PEM format solved the issue.

[ghost]

Thanks for the tip, I'll check the key format and report my results back here. I thought my problem might be because of known issue with a specific release of libssh, but it doesn't seem like I'm running the affected version.

[ghost]

Looks like the SSH key format might have been one of a couple of problems that was preventing pushes to remote repositories. I have everything working now, so thanks for your help.