LFTP syncing files between remote ftp and local gitlab repo

203 Views Asked by At

Whenever I try to get lftp to download files from my remote ftp server to my local gitlab repo I run the pipeline and everything works ok. I even get Removing file then Transferring file. in debug mode it seems to be going to all the correct dir's. however there are no changes what so ever in the gitlab repo regardless. it was working file yesterday.

Anyways here is my code

image: alpine:latest

stages:
  - push
  - force_pull
  - pull


variables:
  op: "1"
  MYTHICLOCALDIR: ./Mythic
  MYTHICREMOTEDIR: ./plugins/MythicMobs 
  LIBLOCALDIR: ./Lib
  LIBREMOTEDIR: ./plugins/MythicLib


pull:
  stage: pull
  script:
    - apk add lftp sudo 
    #- apt-get update -qq && apt-get install -y -qq lftp
    - lftp -c "set ftp:ssl-allow no;set ftp:ssl-protect-data no; open -u $FTPUSER,$FTPPASS $FTPHOST; mirror -v -e $MYTHICREMOTEDIR $MYTHICLOCALDIR --ignore-time --parallel=60/; quit"
    - lftp -c "set ftp:ssl-allow no;set ftp:ssl-protect-data no; open -u $FTPUSER,$FTPPASS $FTPHOST; mirror -v -e $LIBREMOTEDIR $LIBLOCALDIR --ignore-time --parallel=60/; quit"
  only:
    - schedules
  environment:
    name: Mythic_Pull
  parallel:
    matrix:
      - node_version: "12"

push:
  stage: push
  script:
    - apk add lftp sudo 
    #- apt-get update -qq && apt-get install -y -qq lftp
    -  lftp -c "set ftp:ssl-allow no;set ftp:ssl-protect-data no; open -u $FTPUSER,$FTPPASS $FTPHOST; mirror -R -v -e -p $MYTHICLOCALDIR $MYTHICREMOTEDIR --ignore-time --parallel=60 -x .gitkeep/; quit"
    -  lftp -c "set ftp:ssl-allow no;set ftp:ssl-protect-data no; open -u $FTPUSER,$FTPPASS $FTPHOST; mirror -R -v -e -p $LIBLOCALDIR $LIBREMOTEDIR --ignore-time --parallel=60 -x .gitkeep/; quit"
  rules:
    - if: '$op == "2"'
      when: always
  environment:
    name: Mythic_Push
  parallel:
    matrix:
      - node_version: "12"

force_pull:
  stage: force_pull
  script:
    - apk add lftp sudo
    - apk add git sudo
    - git config user.email "$GITLAB_USER_EMAIL"
    - git config user.name "$GITLAB_USER_NAME"
    #- apt-get update -qq && apt-get install -y -qq lftp
    #- lftp -c "set ftp:ssl-allow no;set ftp:ssl-protect-data no; open -u $FTPUSER,$FTPPASS $FTPHOST; mirror -v -e $MYTHICREMOTEDIR $MYTHICLOCALDIR --ignore-time --delete-first --parallel=100/;"
    - lftp -c "set ftp:ssl-allow no;set ftp:ssl-protect-data no; open -u $FTPUSER,$FTPPASS $FTPHOST; mirror -v $LIBREMOTEDIR $LIBLOCALDIR --ignore-time --scan-all-first --transfer-all --log=lftp-log --parallel=60/"
    - git add . 
    - git commit -m "Initial commit"
    - git push origin master
  rules:
    - if: '$op == "1"'
      when: always
  environment:
    name: Mythic_Force_Pull
  parallel:
    matrix:
      - node_version: "12"

for the most part in this example I am using force_pull. I am trying the second one which is why the first one is commented out. All ftp information is stored in gitlab variables and correct. Oh and yes it pushes to the remote ftp server just fine.

0

There are 0 best solutions below