Add Git LFS Support (#180)

This pull request introduces support for Git LFS (Large File Storage) 

- Added the necessary steps to install Git LFS in the docker/entrypoint.
- Updated the `_clone` and `_update` methods to fetch Git LFS objects during cloning and updating operations.
- Modified the `push` method to push Git LFS objects along with regular repository content

Signed-off-by: kareem zarka <[email protected]>
Co-authored-by: kareem zarka <[email protected]>
Co-authored-by: Yikun Jiang <[email protected]>

Dockerfile

@@ -1,6 +1,6 @@
 FROM ubuntu
 
-RUN apt update && apt install git python3 python3-pip -y && \
+RUN apt update && apt install git git-lfs python3 python3-pip -y && \
   echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config
 
 ADD *.sh /

README.md

@@ -50,6 +50,7 @@ steps:
 - `debug` 默认为false, 配置后，启用debug开关，会显示所有执行命令。
 - `timeout` 默认为'30m', 用于设置每个git命令的超时时间，'600'=>600s, '30m'=>30 mins, '1h'=>1 hours
 - `mappings` 源仓库映射规则，比如'A=>B, C=>CC', A会被映射为B，C会映射为CC，映射不具有传递性。主要用于源和目的仓库名不同的镜像。
+- `lfs` 提供[git lfs](https://git-lfs.com/)支持, 默认为false, 配置为true后，调用`git lfs fetch --all`和`git lfs push --all`进行同步。
 
 ## 举些例子
 
@@ -182,6 +183,18 @@ steps:
     static_list: ${{ steps.repo.outputs.repoList }}
 ```
 
+#### 支持LFS同步
+```yaml
+- name: Mirror with lfs (git lfs fetch/push --all)
+  uses: Yikun/hub-mirror-action@master
+  with:
+    src: github/Yikun
+    dst: gitee/yikunkero
+    dst_key: ${{ secrets.GITEE_PRIVATE_KEY }}
+    dst_token: ${{ secrets.GITEE_TOKEN }}
+    lfs: true
+```
+
 ## FAQ
 
 - 如何在secrets添加dst_token和dst_key？

README_en.md

@@ -47,6 +47,7 @@ More than [100+](https://github.com/search?p=2&q=hub-mirror-action+%22account_ty
 - `force_update` (optional) Force to update the destination repo, use '-f' flag do 'git push'
 - `timeout` (optional) Default is '30m', set the timeout for every git command, like '600'=>600s, '30m'=>30 mins, '1h'=>1 hours
 - `mappings` (optional) Default is empty, the source repos mappings, such as 'A=>B, C=>CC', source repo name would be mapped follow the rule: A to B, C to CC. Mapping is not transitive.
+- `lfs` (optional) Default is false, support [git lfs](https://git-lfs.com/), call `git lfs fetch --all` and `git lfs push --all` to support lfs mirror.
 
 ## Scenarios
 
@@ -167,6 +168,18 @@ Note: please configure the public key of `dst_key` to the source (github in here
     static_list: ${{ steps.repo.outputs.repoList }}
 ```
 
+#### Support LFS mirror
+```yaml
+- name: Mirror with lfs (git lfs fetch/push --all)
+  uses: Yikun/hub-mirror-action@master
+  with:
+    src: github/Yikun
+    dst: gitee/yikunkero
+    dst_key: ${{ secrets.GITEE_PRIVATE_KEY }}
+    dst_token: ${{ secrets.GITEE_TOKEN }}
+    lfs: true
+```
+
 ## FAQ
 - How to use `secrets` to add token and key?
 

action.yml

@@ -53,6 +53,9 @@ inputs:
   mappings:
     description: "The source repos mappings, such as 'A=>B, C=>CC', source repo name would be mapped follow the rule: A to B, C to CC. Mapping is not transitive."
     default: ''
+  lfs:
+    description: "Enable Git LFS support."
+    default: false
 runs:
   using: "docker"
   image: "Dockerfile"
@@ -73,3 +76,4 @@ runs:
     - ${{ inputs.debug}}
     - ${{ inputs.timeout}}
     - ${{ inputs.mappings}}
+    - ${{ inputs.lfs }}

entrypoint.sh

@@ -12,6 +12,8 @@ chmod 600 /root/.ssh/id_rsa
 
 pip3 install -r /hub-mirror/requirements.txt
 
+git lfs install
+
 python3 /hub-mirror/hubmirror.py --src "${INPUT_SRC}" --dst "${INPUT_DST}" \
 --dst-token "${INPUT_DST_TOKEN}" \
 --account-type "${INPUT_ACCOUNT_TYPE}" \
@@ -25,7 +27,8 @@ python3 /hub-mirror/hubmirror.py --src "${INPUT_SRC}" --dst "${INPUT_DST}" \
 --force-update "${INPUT_FORCE_UPDATE}" \
 --debug "${INPUT_DEBUG}" \
 --timeout  "${INPUT_TIMEOUT}" \
---mappings  "${INPUT_MAPPINGS}"
+--mappings  "${INPUT_MAPPINGS}" \
+--lfs "${INPUT_LFS}"
 
 # Skip original code
 exit $?

hub-mirror/hubmirror.py

@@ -79,6 +79,10 @@ def run(self):
                         cache=self.args.cache_path,
                         timeout=self.args.timeout,
                         force_update=self.args.force_update,
+                        lfs=(
+                            self.args.lfs if hasattr(self.args, "lfs")
+                            else False
+                        )
                     )
                     mirror.download()
                     mirror.create()

hub-mirror/mirror.py

@@ -11,7 +11,7 @@
 class Mirror(object):
     def __init__(
         self, hub, src_name, dst_name,
-        cache='.', timeout='0', force_update=False
+        cache='.', timeout='0', force_update=False, lfs=False
     ):
         self.hub = hub
         self.src_name = src_name
@@ -24,6 +24,7 @@ def __init__(
         else:
             self.timeout = 0
         self.force_update = force_update
+        self.lfs = lfs
 
     @retry(wait=wait_exponential(), reraise=True, stop=stop_after_attempt(3))
     def _clone(self):
@@ -34,12 +35,17 @@ def _clone(self):
             git.cmd.Git.polish_url(self.src_url), self.repo_path,
             kill_after_timeout=self.timeout
         )
-        print("Clone completed: %s" % os.getcwd() + self.repo_path)
+        local_repo = git.Repo(self.repo_path)
+        if self.lfs:
+            local_repo.git.lfs("fetch", "--all", "origin")
+        print("Clone completed: %s" % (os.getcwd() + self.repo_path))
 
     @retry(wait=wait_exponential(), reraise=True, stop=stop_after_attempt(3))
     def _update(self, local_repo):
         try:
             local_repo.git.pull(kill_after_timeout=self.timeout)
+            if self.lfs:
+                local_repo.git.lfs("fetch", "--all", "origin")
         except git.exc.GitCommandError:
             # Cleanup local repo and re-clone
             print('Updating failed, re-clone %s' % self.src_name)
@@ -71,6 +77,7 @@ def _check_empty(self, repo):
     @retry(wait=wait_exponential(), reraise=True, stop=stop_after_attempt(3))
     def push(self, force=False):
         local_repo = git.Repo(self.repo_path)
+        git_cmd = local_repo.git
         if self._check_empty(local_repo):
             print("Empty repo %s, skip pushing." % self.src_url)
             return
@@ -79,7 +86,7 @@ def push(self, force=False):
         try:
             local_repo.create_remote(self.hub.dst_type, self.dst_url)
         except git.exc.GitCommandError:
-            print("Remote exsits, re-create: set %s to %s" % (
+            print("Remote exists, re-create: set %s to %s" % (
                 self.hub.dst_type, self.dst_url))
             local_repo.delete_remote(self.hub.dst_type)
             local_repo.create_remote(self.hub.dst_type, self.dst_url)
@@ -90,7 +97,11 @@ def push(self, force=False):
         if not self.force_update:
             print("(3/3) Pushing...")
             local_repo.git.push(*cmd, kill_after_timeout=self.timeout)
+            if self.lfs:
+                git_cmd.lfs("push", self.hub.dst_type, "--all")
         else:
             print("(3/3) Force pushing...")
+            if self.lfs:
+                git_cmd.lfs("push", self.hub.dst_type, "--all")
             cmd = ['-f'] + cmd
             local_repo.git.push(*cmd, kill_after_timeout=self.timeout)