Git repositories not indexed if directory name contains .git

[ben8p]

Hi,
We have a setup where both SVN and GIT repositories are fetched.
While SVN repositories are well indexed and visible on the web page, GIT repositories seems to be indexed but remain invisible.

In the logs, I can see:

Oct 27, 2015 8:22:02 AM org.opensolaris.opengrok.util.Executor exec
FINE: Executing command [/opt/tools/git-active/bin/git, --help] in directory /opt/tools/scripts
Oct 27, 2015 8:22:02 AM org.opensolaris.opengrok.util.Executor exec
FINE: Finished command [/opt/tools/git-active/bin/git, --help] in directory /opt/tools/scripts
Oct 27, 2015 8:22:02 AM org.opensolaris.opengrok.util.Executor exec
FINE: Executing command [/opt/tools/subversion-active/bin/svn, --help] in directory /opt/tools/scripts
Oct 27, 2015 8:22:02 AM org.opensolaris.opengrok.util.Executor exec
FINE: Finished command [/opt/tools/subversion-active/bin/svn, --help] in directory /opt/tools/scripts

[...]

Oct 27, 2015 8:22:14 AM org.opensolaris.opengrok.util.Executor exec
FINE: Finished command [/opt/tools/subversion-active/bin/svn, info, --xml] in directory /appl/itrf/opengrok/src/whaeversvnproject
Oct 27, 2015 8:22:14 AM org.opensolaris.opengrok.history.HistoryGuru addRepositories
CONFIG: Adding <org.opensolaris.opengrok.history.SubversionRepository> repository: </appl/itrf/opengrok/src/whaeversvnproject>
Oct 27, 2015 8:22:14 AM org.opensolaris.opengrok.util.Executor exec
FINE: Executing command [/opt/tools/git-active/bin/git, --help] in directory /opt/tools/scripts
Oct 27, 2015 8:22:14 AM org.opensolaris.opengrok.util.Executor exec
FINE: Finished command [/opt/tools/git-active/bin/git, --help] in directory /opt/tools/scripts
Oct 27, 2015 8:22:14 AM org.opensolaris.opengrok.history.HistoryGuru addRepositories
CONFIG: Adding <org.opensolaris.opengrok.history.GitRepository> repository: </appl/itrf/opengrok/src/whaevergitproject.git>
Oct 27, 2015 8:22:14 AM org.opensolaris.opengrok.util.Executor exec

[...]

Oct 27, 2015 8:22:41 AM org.opensolaris.opengrok.util.Executor exec
FINE: Finished command [/opt/tools/subversion-active/bin/svn, log, --trust-server-cert, --non-interactive, --xml, -v, -r, BASE:675] in directory /appl/itrf/opengrok/src/whaeversvnproject
Oct 27, 2015 8:22:41 AM org.opensolaris.opengrok.util.Statistics report
INFO: Done historycache for /appl/itrf/opengrok/src/whaeversvnproject (took 24 ms)
Oct 27, 2015 8:22:41 AM org.opensolaris.opengrok.history.HistoryGuru createCache
INFO: Creating historycache for /appl/itrf/opengrok/src/whaevergitproject.git (GitRepository)
Oct 27, 2015 8:22:41 AM org.opensolaris.opengrok.util.Executor exec
FINE: Executing command [/opt/tools/git-active/bin/git, log, --abbrev-commit, --abbrev=8, --name-only, --pretty=fuller, ac891a28..] in directory /appl/itrf/opengrok/src/whaevergitproject.git
Oct 27, 2015 8:22:41 AM org.opensolaris.opengrok.util.Executor exec

I noticed openGrok append .. at the end of the revision number.
On command line, if run
/opt/tools/git-active/bin/git log --abbrev-commit --abbrev=8 --name-only --pretty=fuller ac891a28.. then nothing is displayed
but if i run /opt/tools/git-active/bin/git log --abbrev-commit --abbrev=8 --name-only --pretty=fuller ac891a28 (without ..) then I can see the history.

According to git documentation, git log supports what is describes in "revision range" section (https://git-scm.com/docs/gitrevisions).
In this page, the only occurence of .. is with 2 revisions: r1..r2

Am I doing something wrong or is it a bug in the Git indexer ?

Thanks

[tarzanek]

which versions is this?
what is the git version? (I noticed you use some custom build/install ;) )
We index our own opengrok repo, which is git without any troubles ... (with stock OS components, but it's Solaris 11.2, so I can try with your config to see what is the misbehaviour, ev. your .gitrc / .gitconfig has some special stuff ? )

[ben8p]

Hi,
We use git 2.6.1 built from source (no rights to install the binaries).
It was built with the following options:

./configure --prefix=/opt/tools/git-2.6.1 \
    --with-openssl \
     --with-expat \
    --with-curl=/opt/tools/curl-active \
    --with-zlib=/opt/tools/zlib-active \
    --with-libpcre \
    --with-python=/opt/tools/python-active

I tried opengrok 0.12.1.5 release and master (commit 7933205) built from source.
the git config contains only

[http]
        sslVerify = false

the linux version is :

Linux 3.0.101-0.47.55-default #1 SMP Thu May 28 08:25:11 UTC 2015 (dc083ee) x86_64 x86_64 x86_64 GNU/Linux 
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3

Note:

• curl version for git: 7.19.7
• python version for git: 2.7.9

[ben8p]

Anyone has an idea ?
@tarzanek with the extra details i gave do you have a clue ?

I am really out of ideas, so any little clue would help...
I also tried to setup opengrok for git only and then I get an empty list on the webpage

[tarzanek]

just to understand the prob definition - so basically you say, that subdirectories that are git repos are not shown on the project list page ?
Or the problem is that the projects with git don't show any history buttons, resp. history is not working ?

Since if the projects are not shown, then it means the generated config.xml is lacking them (check etc under /var/opengrok if it shows correct project list)
If it's about history, we need to understand the commands more as you suggested

(just clarifying the issue here)

[ben8p]

@tarzanek indeed, project are not even listed in the project list page.

The config.xml looks ok:

<void property="cmds">
   <object class="java.util.Collections" method="unmodifiableMap">
    <object class="java.util.HashMap">
     <void method="put">
      <string>org.opensolaris.opengrok.history.PerforceRepository</string>
      <string>p4</string>
     </void>
     <void method="put">
      <string>org.opensolaris.opengrok.history.SubversionRepository</string>
      <string>/opt/tools/subversion-active/bin/svn</string>
     </void>
     <void method="put">
      <string>org.opensolaris.opengrok.history.ClearCaseRepository</string>
      <string>cleartool</string>
     </void>
     <void method="put">
      <string>org.opensolaris.opengrok.history.GitRepository</string>
      <string>/opt/tools/git-active/bin/git</string>
     </void>
    </object>
   </object>
  </void>

[...]

<void method="add">
    <object class="org.opensolaris.opengrok.history.RepositoryInfo">
     <void property="datePattern">
      <string>EEE MMM dd hh:mm:ss yyyy ZZZZ</string>
     </void>
     <void property="directoryName">
      <string>/appl/itrf/opengrok/src/whaevergitproject.git</string>
     </void>
     <void property="type">
      <string>git</string>
     </void>
    </object>
   </void>
 <void method="add">
    <object class="org.opensolaris.opengrok.history.RepositoryInfo">
     <void property="datePattern">
      <string>yyyy-MM-dd&apos;T&apos;HH:mm:ss.SSS&apos;Z&apos;</string>
     </void>
     <void property="directoryName">
      <string>/appl/itrf/opengrok/src/whaeversvnproject</string>
     </void>
     <void property="type">
      <string>Subversion</string>
     </void>
    </object>
   </void>

If that can help, this is how we generate the config:

#!/bin/bash
LD_LIBRARY_PATH=/opt/tools/subversion-active/lib:/opt/tools/openssl-active/lib:/opt/tools/zlib-active/lib
export LD_LIBRARY_PATH
export PATH=$PATH:/opt/tools/subversion-active/bin:/opt/tools/git-active/bin
export LD_LIBRARY_PATH_64==/opt/tools/subversion-active/lib:/opt/tools/openssl-active/lib:/opt/tools/zlib-active/lib
export SVN=/opt/tools/subversion-active/bin/svn
export GIT=/opt/tools/git-active/bin/git
export JAVA_HOME=/appl/it/java/jdk1.7.0_51
export EXUBERANT_CTAGS=/opt/tools/ctags-active/bin/ctags

SRC_ROOT=/appl/it/opengrok/src
DATA_ROOT=/appl/it/opengrok/data

JAR="-jar /opt/tools/opengrok-active/lib/opengrok.jar"

LOGGER=-Djava.util.logging.config.file=/opt/tools/scripts/opengrok.logging.properties
LOGGER_SVN=-Djava.util.logging.config.file=/opt/tools/scripts/opengrok.svn.logging.properties

$JAVA_HOME/bin/java -Xmx2048m -Dorg.opensolaris.opengrok.history.Subversion=$SVN \
-Dorg.opensolaris.opengrok.history.git=$GIT \
$LOGGER_SVN \
$JAR \
-S \
-r off \
-v \
-c $EXUBERANT_CTAGS \
-a on \
-l off \
-B https://wwww.someinternalpages.com# \
-R /appl/it/opengrok/etc/configuration.base.xml \
-W /opt/tools/scripts/etc/configuration.xml \
-H \
-s $SRC_ROOT \
-d $DATA_ROOT > /opt/tools/scripts/logs/index-svn.log

$JAVA_HOME/bin/java -Dorg.opensolaris.opengrok.history.Subversion=$SVN \
-Dorg.opensolaris.opengrok.history.git=$GIT \
$LOGGER -Xmx512m \
$JAR \
-R /opt/tools/scripts/etc/configuration.xml \
-U localhost:2424 > /opt/tools/scripts/logs/autoupdate.log \
-n

[vladak]

The rev.. notation is used for fetching partial history (introduced with cset 49c1125) in GitRepository.java#getHistoryLogExecutor(). I think the equivalent of rev.. should be rev..HEAD.

git help log says:

     <since>..<until>
         Show only commits between the named two commits. When
         either <since> or <until> is omitted, it defaults to
         HEAD, i.e. the tip of the current branch. For a more
         complete list of ways to spell <since> and <until>, see
         gitrevisions(5).

I tried git-2.6.2 compiled from source (using only --prefix) and it behaves the same as 1.7.9.2 I normally use, at least when used on the OpenGrok git repo.

The only case when there is no output using the rev.. notation is when the revision is HEAD which makes sense - we only want to output revisions since (but not including) that revision when performing incremental indexing.

If you're using file based history cache, the latest indexed revision is stored in /var/opengrok/data/historycache/<your_project>/OpenGroklatestRev. If that file contains the revision number from the log then history indexer is probably working as expected.

It seems that the visibility problem has different cause.

[vladak]

I am assuming that by "remain invisible" you mean it is missing in the select box on the front page but it is present when you go to /source/xref/ in the web application. AFAIK the select box is available only when projects are enabled. In the above indexer command line the projects are not enabled (the -P option is not used) which is probably the reason why it is missing in the project picker.

[vladak]

In configuration.xml the projects are visible under the projects property, e.g.:

  <void property="projects">
   <void method="add">
    <object class="org.opensolaris.opengrok.configuration.Project">
     <void property="description">
      <string>OpenGrok</string>
     </void>
     <void property="path">
      <string>/OpenGrok</string>
     </void>
    </object>
   </void>
...
  </void>

[ben8p]

@vladak thanks for the details.
According to your message, my first assumption was wrong and the indexer is ok.

cat /appl/it/opengrok/data/historycache/whaevergitproject.git/OpenGroklatestRev

returns ac891a28

By "remain invisible" i mean it is NOT present when you go to /source/xref/ (on server or on webpage)

ls /appl/it/opengrok/data/xref/whaevergitproject.git

returns ls: cannot access /appl/it/opengrok/data/xref/whaevergitproject.git: No such file or directory

Also, there is no <void property="projects"> in our configuration . We don't use this feature.

Then it seems that somehow the projects is not fetched into /appl/it/opengrok/data/xref/...
Any idea on what could cause that ?

Thanks

[vladak]

Assuming the permissions in the xref directory are okay. IndexDatabase.java#addFile() reports any problems with INFO and FINE level so they might escape with default logging config. You can try to edit your logging.properties file (normally it resides under /var/opengrok/ directory) so that it contains this line:

.level= ALL

rerun the indexer and inspect the log for any problems.

[vladak]

Also, another idea would be to index the same git repo in some virtual environment with OpenGrok installation done to the default paths.

[ben8p]

@vladak thanks
I inspected the log but I cannot see anything wrong.

Probably not related to my problem but I noticed that in DATA_ROOT/historycache there are repositories removed from SRC_ROOT
Is this expected ?

For tests, I added the -P option to the indexer.
After running it, the git repository was visible in the project selector box but still not in the xref.
Also clicking on the git repository was forwarding to a 404 repository not found

So, somehow -P see the repository, but it is never copied into xref folder.

[tarzanek]

Could it be that your repos haven name from the default ignore paths?

[ben8p]

@tarzanek the repositories names are:

• itrf-ckeditor (from svn, available in xref)
• itrf-ckeditor.git (from git, NOT available in xref)

I don't think there is any default ignore on ".git" pattern. Is there?

[tarzanek]

Lol
I think there is
Can you please try to get rid of .git in name as test?
What version of grok is this?
Now I am not sure if we released the improved way for ignore ...