Merge branch 'develop' of frappe/frappe_docker into kubernetes

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,34 @@
+---
+name: Bug report
+about: Report a bug encountered while using the Frappe_docker
+labels: bug
+---
+
+<!--
+Welcome to the frappe_docker issue tracker! Before creating an issue, please heed the following:
+
+1. Is your issue relevant to the frappe_docker or the main Frappe framework? https://github.com/frappe/frappe . if It's the latter, publish the issue there.
+2. Use the search function before creating a new issue. Duplicates will be closed and directed to the original discussion.
+3. When making a bug report, make sure you provide all the required information. The easier it is for maintainers to reproduce, the faster it'll be fixed.
+4. If you think you know what the reason for the bug is, share it with us. Maybe put in a PR 😉
+-->
+
+## Description of the issue
+
+## Context information (for bug reports)
+
+## Steps to reproduce the issue
+
+1.
+2.
+3.
+
+### Observed result
+
+### Expected result
+
+### Stacktrace / full error message if available
+
+```
+(paste here)
+```

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,23 @@
+---
+name: Feature request
+about: Suggest an idea to improve frappe_docker
+labels: enhancement
+---
+
+<!--
+Welcome to the Frappe Framework issue tracker! Before creating an issue, please heed the following:
+1. Use the search function before creating a new issue. Duplicates will be closed and directed to the original discussion.
+2. When making a feature request, make sure to be as verbose as possible. The better you convey your message, the greater the drive to make it happen.
+-->
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/question-about-using-frappe_docker.md

@@ -0,0 +1,12 @@
+---
+name: Question about using Frappe/Frappe Apps
+about: Ask how to do something
+labels: question
+---
+
+<!--
+Welcome to the frappe_docker issue tracker! Before creating an issue, please heed the following:
+
+1. Use the search function before creating a new issue. Duplicates will be closed and directed to the original discussion.
+2. Please write extensively, clearly and in detail.
+-->

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,7 @@
+> Please provide enough information so that others can review your pull request:
+
+<!-- You can skip this if you're fixing a typo or updating existing documentation -->
+
+> Explain the **details** for making this change. What existing problem does the pull request solve?
+
+<!-- Example: When "Adding a function to do X", explain why it is necessary to have a way to do X. -->

.github/workflows/stale.yml

@@ -0,0 +1,19 @@
+name: Mark stale issues and pull requests
+
+on:
+  schedule:
+  - cron: "0 0 * * *"
+
+jobs:
+  stale:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/stale@v1
+      with:
+        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        stale-issue-message: 'Stale issue message'
+        stale-pr-message: 'Stale pull request message'
+        stale-issue-label: 'no-issue-activity'
+        stale-pr-label: 'no-pr-activity'

.travis.yml

@@ -53,60 +53,60 @@ jobs:
     - stage: "Frappe (v12)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py frappe --worker --git-branch 12
+        - ./travis.py frappe --worker --git-version 12
         - ./travis.py frappe --worker --tag v12 --tag-only
         - ./travis.py frappe --worker --tag version-12 --tag-only
     - stage: "Frappe (v12)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py frappe --nginx --git-branch 12
+        - ./travis.py frappe --nginx --git-version 12
         - ./travis.py frappe --nginx --tag v12 --tag-only
         - ./travis.py frappe --nginx --tag version-12 --tag-only
     - stage: "Frappe (v12)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py frappe --socketio --git-branch 12
+        - ./travis.py frappe --socketio --git-version 12
         - ./travis.py frappe --socketio --tag v12 --tag-only
         - ./travis.py frappe --socketio --tag version-12 --tag-only
     - stage: "ERPNext (v12)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py erpnext --worker --git-branch 12
+        - ./travis.py erpnext --worker --git-version 12
         - ./travis.py erpnext --worker --tag v12 --tag-only
         - ./travis.py erpnext --worker --tag version-12 --tag-only
     - stage: "ERPNext (v12)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py erpnext --nginx --git-branch 12
+        - ./travis.py erpnext --nginx --git-version 12
         - ./travis.py erpnext --nginx --tag v12 --tag-only
         - ./travis.py erpnext --nginx --tag version-12 --tag-only
     - stage: "Frappe (v11)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py frappe --worker --git-branch 11
+        - ./travis.py frappe --worker --git-version 11
         - ./travis.py frappe --worker --tag v11 --tag-only
         - ./travis.py frappe  --worker --tag version-11 --tag-only
     - stage: "Frappe (v11)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py erpnext frappe --nginx --git-branch 11
-        - ./travis.py erpnext frappe --nginx --tag v11 --tag-only
-        - ./travis.py erpnext frappe --nginx --tag version-11 --tag-only
+        - ./travis.py frappe --nginx --git-version 11
+        - ./travis.py frappe --nginx --tag v11 --tag-only
+        - ./travis.py frappe --nginx --tag version-11 --tag-only
     - stage: "Frappe (v11)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py frappe --socketio --git-branch 11
+        - ./travis.py frappe --socketio --git-version 11
         - ./travis.py frappe --socketio --tag v11 --tag-only
         - ./travis.py frappe --socketio --tag version-11 --tag-only
     - stage: "ERPNext (v11)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py erpnext --worker --git-branch 11
+        - ./travis.py erpnext --worker --git-version 11
         - ./travis.py erpnext --worker --tag v11 --tag-only
         - ./travis.py erpnext --worker --tag version-11 --tag-only
     - stage: "ERPNext (v11)"
       if: branch = master AND type != pull_request
       script:
-        - ./travis.py erpnext --nginx --git-branch 11
+        - ./travis.py erpnext --nginx --git-version 11
         - ./travis.py erpnext --nginx --tag v11 --tag-only
         - ./travis.py erpnext --nginx --tag version-11 --tag-only

CONTRIBUTING.md

@@ -0,0 +1,25 @@
+# Contribution Guidelines
+## Branches
+
+* *master*:  images on the master branch are built monthly.
+* *develop*: images on this branch are built daily.
+
+# Pull Requests
+
+Please **send all pull request exclusively to the *develop*** branch. 
+When the PR are merged, the merge will trigger the image build automatically.
+
+Please test all PR as extensively as you can, considering that the software can be run in different modes:
+* with docker-compose for production
+* with or without Nginx proxy
+* with VScode for testing environments
+
+Every once in a while (or before monthly release) develop will be merged into master.
+
+## Reducing the number of branching and builds :evergreen_tree: :evergreen_tree: :evergreen_tree: 
+Please be considerate when pushing commits and opening PR for multiple branches, as the process of building images (triggered on push and PR branch push) uses energy and contributes to global warming.
+
+# Documentation
+
+You should place README.md(s) in the relevant directories, explaining what the software in that particular directory does.
+

README.md

@@ -101,6 +101,7 @@ Make sure to replace `<project-name>` with the desired name you wish to set for
 
 Notes:
 
+- New site (first site) needs to be added after starting the services.
 - The local deployment is for testing and REST API development purpose only
 - A complete development environment is available [here](Development/README.md)
 - The site names are limited to patterns matching \*.localhost by default
@@ -140,7 +141,11 @@ docker-compose \
 ```
 
 Make sure to replace `<project-name>` with any desired name you wish to set for the project.
-Note: use `docker-compose-frappe.yml` in case you need only Frappe without ERPNext.
+
+Notes:
+
+- Use `docker-compose-frappe.yml` in case you need only Frappe without ERPNext.
+- New site (first site) needs to be added after starting the services.
 
 ### Docker containers
 
@@ -214,6 +219,9 @@ Environment Variables
 
 - `SITES` is list of sites separated by `:` colon to migrate. e.g. `SITES=site1.domain.com` or `SITES=site1.domain.com:site2.domain.com` By default all sites in bench will be backed up.
 - `WITH_FILES` if set to 1, it will backup user-uploaded files.
+- By default `backup` takes mariadb dump and gzips it. Example file, `20200325_221230-test_localhost-database.sql.gz`
+- If `WITH_FILES` is set then it will also backup public and private files of each site as uncompressed tarball. Example files, `20200325_221230-test_localhost-files.tar` and `20200325_221230-test_localhost-private-files.tar`
+- All the files generated by backup are placed at volume location `sites/{site-name}/private/backups/*`
 
 ```sh
 docker exec -it \
@@ -224,6 +232,36 @@ docker exec -it \
 
 The backup will be available in the `sites` mounted volume.
 
+#### Push backup to s3 compatible storage
+
+Environment Variables
+
+- `BUCKET_NAME`, Required to set bucket created on S3 compatible storage.
+- `ACCESS_KEY_ID`, Required to set access key.
+- `SECRET_ACCESS_KEY`, Required to set secret access key.
+- `ENDPOINT_URL`, Required to set URL of S3 compatible storage.
+- `BUCKET_DIR`, Required to set directory in bucket where sites from this deployment will be backed up.
+- `BACKUP_LIMIT`, Optionally set this to limit number of backups in bucket directory. Defaults to 3.
+
+```sh
+ docker run \
+    -e "BUCKET_NAME=backups" \
+    -e "ACCESS_KEY_ID=access_id_from_provider" \
+    -e "SECRET_ACCESS_KEY=secret_access_from_provider" \
+    -e "ENDPOINT_URL=https://region.storage-provider.com" \
+    -e "BUCKET_DIR=frappe-bench-v12" \
+    -v ./installation/sites:/home/frappe/frappe-bench/sites \
+    --network <project-name>_default \
+    frappe/frappe-worker:v12 push-backup
+```
+
+Note:
+
+- Above example will backup files in bucket called `backup` at location `frappe-bench-v12/site.name.com/DATE_TIME/DATE_TIME-site_name_com-{filetype}.{extension}`,
+- example DATE_TIME: 20200325_042020.
+- example filetype: database, files or private-files
+- example extension: sql.gz or tar
+
 #### Updating and Migrating Sites
 
 Switch to the root of the `frappe_docker` directory before running the following commands:
@@ -251,6 +289,44 @@ docker exec -it \
     <project-name>_erpnext-python_1 docker-entrypoint.sh migrate
 ```
 
+#### Restore backups
+
+Environment Variables
+
+- `MYSQL_ROOT_PASSWORD`, Required to restore mariadb backups.
+- `BUCKET_NAME`, Required to set bucket created on S3 compatible storage.
+- `ACCESS_KEY_ID`, Required to set access key.
+- `SECRET_ACCESS_KEY`, Required to set secret access key.
+- `ENDPOINT_URL`, Required to set URL of S3 compatible storage.
+- `BUCKET_DIR`, Required to set directory in bucket where sites from this deployment will be backed up.
+
+```sh
+docker run \
+    -e "MYSQL_ROOT_PASSWORD=admin" \
+    -e "BUCKET_NAME=backups" \
+    -e "ACCESS_KEY_ID=access_id_from_provider" \
+    -e "SECRET_ACCESS_KEY=secret_access_from_provider" \
+    -e "ENDPOINT_URL=https://region.storage-provider.com" \
+    -e "BUCKET_DIR=frappe-bench-v12" \
+    -v ./installation/sites:/home/frappe/frappe-bench/sites \
+    -v ./backups:/home/frappe/backups \
+    --network <project-name>_default \
+    frappe/frappe-worker:v12 restore-backup
+```
+
+Note:
+
+- Volume must be mounted at location `/home/frappe/backups` for restoring sites
+- If no backup files are found in volume, it will use s3 credentials to pull backups
+- Backup structure for mounted volume or downloaded from s3:
+    - /home/frappe/backups
+        - site1.domain.com
+            - 20200420_162000
+                - 20200420_162000-site1_domain_com-*
+        - site2.domain.com
+            - 20200420_162000
+                - 20200420_162000-site2_domain_com-*
+
 ### Custom apps
 
 To add your own Frappe/ERPNext apps to the image, we'll need to create a custom image with the help of a unique wrapper script

build/common/commands/auto_migrate.py

@@ -0,0 +1,109 @@
+import os
+import json
+import semantic_version
+import git
+
+from migrate import migrate_sites
+from check_connection import get_config
+
+APP_VERSIONS_JSON_FILE = 'app_versions.json'
+APPS_TXT_FILE = 'apps.txt'
+
+def save_version_file(versions):
+    with open(APP_VERSIONS_JSON_FILE, 'w') as f:
+        return json.dump(versions, f, indent=1, sort_keys=True)
+
+def get_apps():
+    apps = []
+    try:
+        with open(APPS_TXT_FILE) as apps_file:
+            for app in apps_file.readlines():
+                if app.strip():
+                    apps.append(app.strip())
+
+    except FileNotFoundError as exception:
+        print(exception)
+        exit(1)
+    except:
+        print(APPS_TXT_FILE+" is not valid")
+        exit(1)
+
+    return apps
+
+def get_container_versions(apps):
+    versions = {}
+    for app in apps:
+        try:
+            version = __import__(app).__version__
+            versions.update({app:version})
+        except:
+            pass
+
+        try:
+            path = os.path.join('..','apps', app)
+            repo = git.Repo(path)
+            commit_hash = repo.head.object.hexsha
+            versions.update({app+'_git_hash':commit_hash})
+        except:
+            pass
+
+    return versions
+
+def get_version_file():
+    versions = None
+    try:
+        with open(APP_VERSIONS_JSON_FILE) as versions_file:
+            versions = json.load(versions_file)
+    except:
+        pass
+    return versions
+
+def main():
+    is_ready = False
+    apps = get_apps()
+
+    container_versions = get_container_versions(apps)
+
+    version_file = get_version_file()
+
+    if not version_file:
+        version_file = container_versions
+        save_version_file(version_file)
+
+    for app in apps:
+        container_version = None
+        file_version = None
+        version_file_hash = None
+        container_hash = None
+
+        repo = git.Repo(os.path.join('..','apps',app))
+        branch = repo.active_branch.name
+
+        if branch == 'develop':
+            version_file_hash = version_file.get(app+'_git_hash')
+            container_hash = container_versions.get(app+'_git_hash')
+            if container_hash and version_file_hash:
+                if container_hash != version_file_hash:
+                    is_ready = True
+                    break
+
+        if version_file.get(app):
+            file_version = semantic_version.Version(version_file.get(app))
+
+        if container_versions.get(app):
+            container_version = semantic_version.Version(container_versions.get(app))
+
+        if file_version and container_version:
+            if container_version > file_version:
+                is_ready = True
+                break
+
+    config = get_config()
+
+    if is_ready and config.get('maintenance_mode') != 1:
+        migrate_sites(maintenance_mode=True)
+        version_file = container_versions
+        save_version_file(version_file)
+
+if __name__ == "__main__":
+    main()

build/common/commands/background.py

@@ -1,7 +1,10 @@
 import frappe
 from frappe.utils.scheduler import start_scheduler
 
+def main():
     print("Starting background scheduler . . .")
     start_scheduler()
-
     exit(0)
+
+if __name__ == "__main__":
+    main()

build/common/commands/backup.py

@@ -20,10 +20,13 @@ def backup(sites, with_files=False):
             print("private files backup taken -", odb.backup_path_private_files, "- on", now())
         frappe.destroy()
 
+def main():
     installed_sites = ":".join(get_sites())
     sites = os.environ.get("SITES", installed_sites).split(":")
     with_files=True if os.environ.get("WITH_FILES") else False
 
     backup(sites, with_files)
-
     exit(0)
+
+if __name__ == "__main__":
+    main()

build/common/commands/check_connection.py

@@ -108,6 +108,13 @@ def check_redis_socketio(retry=10, delay=3, print_attempt=True):
         print("Connection to redis socketio timed out")
         exit(1)
 
+# Get site_config.json
+def get_site_config(site_name):
+    site_config = None
+    with open('{site_name}/site_config.json'.format(site_name=site_name)) as site_config_file:
+        site_config = json.load(site_config_file)
+    return site_config
+
 def main():
     check_mariadb()
     check_redis_queue()

build/common/commands/console.py

@@ -20,6 +20,6 @@ def console(site):
     print("Apps in this namespace:\n{}".format(", ".join(all_apps)))
     IPython.embed(display_banner="", header="")
 
-
+def main():
     site = sys.argv[-1]
     console(site)

build/common/commands/migrate.py

@@ -2,12 +2,7 @@ import os, frappe, compileall, re, json
 
 from frappe.migrate import migrate
 from frappe.utils import get_sites
-
-def get_config():
-    config = None
-    with open('common_site_config.json') as config_file:
-        config = json.load(config_file)
-    return config
+from check_connection import get_config
 
 def save_config(config):
     with open('common_site_config.json', 'w') as f:
@@ -24,9 +19,10 @@ def set_maintenance_mode(enable=True):
         conf.update({ "maintenance_mode": 0, "pause_scheduler": 0 })
         save_config(conf)
 
-
+def migrate_sites(maintenance_mode=False):
     installed_sites = ":".join(get_sites())
     sites = os.environ.get("SITES", installed_sites).split(":")
+    if not maintenance_mode:
         maintenance_mode = True if os.environ.get("MAINTENANCE_MODE") else False
 
     if maintenance_mode:
@@ -44,4 +40,9 @@ for site in sites:
     if maintenance_mode:
         set_maintenance_mode(False)
 
+def main():
+    migrate_sites()
     exit(0)
+
+if __name__ == "__main__":
+    main()

build/common/commands/new.py

@@ -1,7 +1,9 @@
 import os, frappe, json
 
 from frappe.commands.site import _new_site
+from check_connection import get_config, get_site_config
 
+def main():
     site_name = os.environ.get("SITE_NAME", 'site1.localhost')
     mariadb_root_username = os.environ.get("DB_ROOT_USER", 'root')
     mariadb_root_password = os.environ.get("MYSQL_ROOT_PASSWORD", 'admin')
@@ -23,46 +25,35 @@ _new_site(
         reinstall=False,
     )
 
-config = None
-with open('common_site_config.json') as config_file:
-    config = json.load(config_file)
+    config = get_config()
 
-site_config = None
-with open('{site_name}/site_config.json'.format(site_name=site_name)) as site_config_file:
-    site_config = json.load(site_config_file)
+    site_config = get_site_config(site_name)
 
-# update User's host to '%' required to connect from any container
-command = 'mysql -h{db_host} -u{mariadb_root_username} -p{mariadb_root_password} -e '.format(
+    mysql_command = 'mysql -h{db_host} -u{mariadb_root_username} -p{mariadb_root_password} -e '.format(
         db_host=config.get('db_host'),
         mariadb_root_username=mariadb_root_username,
         mariadb_root_password=mariadb_root_password
     )
-command += "\"UPDATE mysql.user SET Host = '%' where User = '{db_name}'; FLUSH PRIVILEGES;\"".format(
+
+    # update User's host to '%' required to connect from any container
+    command = mysql_command + "\"UPDATE mysql.user SET Host = '%' where User = '{db_name}'; FLUSH PRIVILEGES;\"".format(
         db_name=site_config.get('db_name')
     )
     os.system(command)
 
     # Set db password
-command = 'mysql -h{db_host} -u{mariadb_root_username} -p{mariadb_root_password} -e '.format(
-    db_host=config.get('db_host'),
-    mariadb_root_username=mariadb_root_username,
-    mariadb_root_password=mariadb_root_password
-)
-command += "\"SET PASSWORD FOR '{db_name}'@'%' = PASSWORD('{db_password}'); FLUSH PRIVILEGES;\"".format(
+    command = mysql_command + "\"UPDATE mysql.user SET authentication_string = PASSWORD('{db_password}') WHERE User = \'{db_name}\' AND Host = \'%\';\"".format(
         db_name=site_config.get('db_name'),
         db_password=site_config.get('db_password')
     )
     os.system(command)
 
     # Grant permission to database
-command = 'mysql -h{db_host} -u{mariadb_root_username} -p{mariadb_root_password} -e '.format(
-    db_host=config.get('db_host'),
-    mariadb_root_username=mariadb_root_username,
-    mariadb_root_password=mariadb_root_password
-)
-command += "\"GRANT ALL PRIVILEGES ON \`{db_name}\`.* TO '{db_name}'@'%'; FLUSH PRIVILEGES;\"".format(
+    command = mysql_command + "\"GRANT ALL PRIVILEGES ON \`{db_name}\`.* TO '{db_name}'@'%'; FLUSH PRIVILEGES;\"".format(
         db_name=site_config.get('db_name')
     )
     os.system(command)
-
     exit(0)
+
+if __name__ == "__main__":
+    main()

build/common/commands/push_backup.py

@@ -0,0 +1,182 @@
+import os
+import time
+import boto3
+
+import datetime
+from glob import glob
+from frappe.utils import get_sites
+
+DATE_FORMAT = "%Y%m%d_%H%M%S"
+
+def get_file_ext():
+    return {
+        "database": "-database.sql.gz",
+        "private_files": "-private-files.tar",
+        "public_files": "-files.tar"
+    }
+
+def get_backup_details(sitename):
+    backup_details = dict()
+    file_ext = get_file_ext()
+
+    # add trailing slash https://stackoverflow.com/a/15010678
+    site_backup_path = os.path.join(os.getcwd(), sitename, "private", "backups", "")
+
+    if os.path.exists(site_backup_path):
+        for filetype, ext in file_ext.items():
+            site_slug = sitename.replace('.', '_')
+            pattern = site_backup_path + '*-' + site_slug + ext
+            backup_files = list(filter(os.path.isfile, glob(pattern)))
+
+            if len(backup_files) > 0:
+                backup_files.sort(key=lambda file: os.stat(os.path.join(site_backup_path, file)).st_ctime)
+                backup_date = datetime.datetime.strptime(time.ctime(os.path.getmtime(backup_files[0])), "%a %b %d %H:%M:%S %Y")
+                backup_details[filetype] = {
+                    "sitename": sitename,
+                    "file_size_in_bytes": os.stat(backup_files[-1]).st_size,
+                    "file_path": os.path.abspath(backup_files[-1]),
+                    "filename": os.path.basename(backup_files[-1]),
+                    "backup_date": backup_date.date().strftime("%Y-%m-%d %H:%M:%S")
+                }
+
+    return backup_details
+
+def get_s3_config():
+    check_environment_variables()
+    bucket = os.environ.get('BUCKET_NAME')
+
+    conn = boto3.client(
+        's3',
+        aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
+        aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
+        endpoint_url=os.environ.get('ENDPOINT_URL')
+    )
+
+    return conn, bucket
+
+def check_environment_variables():
+    if not 'BUCKET_NAME' in os.environ:
+        print('Variable BUCKET_NAME not set')
+        exit(1)
+
+    if not 'ACCESS_KEY_ID' in os.environ:
+        print('Variable ACCESS_KEY_ID not set')
+        exit(1)
+
+    if not 'SECRET_ACCESS_KEY' in os.environ:
+        print('Variable SECRET_ACCESS_KEY not set')
+        exit(1)
+
+    if not 'ENDPOINT_URL' in os.environ:
+        print('Variable ENDPOINT_URL not set')
+        exit(1)
+
+    if not 'BUCKET_DIR' in os.environ:
+        print('Variable BUCKET_DIR not set')
+        exit(1)
+
+def upload_file_to_s3(filename, folder, conn, bucket):
+
+    destpath = os.path.join(folder, os.path.basename(filename))
+    try:
+        print("Uploading file:", filename)
+        conn.upload_file(filename, bucket, destpath)
+
+    except Exception as e:
+        print("Error uploading: %s" % (e))
+        exit(1)
+
+def delete_old_backups(limit, bucket, site_name):
+    all_backups = list()
+    all_backup_dates = list()
+    backup_limit = int(limit)
+    check_environment_variables()
+    bucket_dir = os.environ.get('BUCKET_DIR')
+    oldest_backup_date = None
+
+    s3 = boto3.resource(
+        's3',
+        aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
+        aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
+        endpoint_url=os.environ.get('ENDPOINT_URL')
+    )
+
+    bucket = s3.Bucket(bucket)
+    objects = bucket.meta.client.list_objects_v2(
+        Bucket=bucket.name,
+        Delimiter='/')
+
+    if objects:
+        for obj in objects.get('CommonPrefixes'):
+            if obj.get('Prefix') == bucket_dir + '/':
+                for backup_obj in bucket.objects.filter(Prefix=obj.get('Prefix')):
+                    try:
+                        # backup_obj.key is bucket_dir/site/date_time/backupfile.extension
+                        bucket_dir, site_slug, date_time, backupfile = backup_obj.key.split('/')
+                        date_time_object = datetime.datetime.strptime(
+                            date_time, DATE_FORMAT
+                        )
+
+                        if site_name in backup_obj.key:
+                            all_backup_dates.append(date_time_object)
+                            all_backups.append(backup_obj.key)
+                    except IndexError as error:
+                        print(error)
+                        exit(1)
+
+    if len(all_backup_dates) > 0:
+        oldest_backup_date = min(all_backup_dates)
+
+    if len(all_backups) / 3 > backup_limit:
+        oldest_backup = None
+        for backup in all_backups:
+            try:
+                # backup is bucket_dir/site/date_time/backupfile.extension
+                backup_dir, site_slug, backup_dt_string, filename = backup.split('/')
+                backup_datetime = datetime.datetime.strptime(
+                    backup_dt_string, DATE_FORMAT
+                )
+                if backup_datetime == oldest_backup_date:
+                    oldest_backup = backup
+
+            except IndexError as error:
+                print(error)
+                exit(1)
+
+            if oldest_backup:
+                for obj in bucket.objects.filter(Prefix=oldest_backup):
+                    # delete all keys that are inside the oldest_backup
+                    if bucket_dir in obj.key:
+                        print('Deleteing ' + obj.key)
+                        s3.Object(bucket.name, obj.key).delete()
+
+def main():
+    details = dict()
+    sites = get_sites()
+    conn, bucket = get_s3_config()
+
+    for site in sites:
+        details = get_backup_details(site)
+        db_file = details.get('database', {}).get('file_path')
+        folder = os.environ.get('BUCKET_DIR') + '/' + site + '/'
+        if db_file:
+            folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(db_file)[:15] + '/'
+            upload_file_to_s3(db_file, folder, conn, bucket)
+
+        public_files = details.get('public_files', {}).get('file_path')
+        if public_files:
+            folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(public_files)[:15] + '/'
+            upload_file_to_s3(public_files, folder, conn, bucket)
+
+        private_files = details.get('private_files', {}).get('file_path')
+        if private_files:
+            folder = os.environ.get('BUCKET_DIR') + '/' + site + '/' + os.path.basename(private_files)[:15] + '/'
+            upload_file_to_s3(private_files, folder, conn, bucket)
+
+        delete_old_backups(os.environ.get('BACKUP_LIMIT', '3'), bucket, site)
+
+    print('push-backup complete')
+    exit(0)
+
+if __name__ == "__main__":
+    main()

build/common/commands/restore_backup.py

@@ -0,0 +1,188 @@
+import os
+import datetime
+import tarfile
+import hashlib
+import frappe
+import boto3
+
+from push_backup import DATE_FORMAT, check_environment_variables
+from frappe.utils import get_sites, random_string
+from frappe.commands.site import _new_site
+from frappe.installer import make_conf, get_conf_params, make_site_dirs
+from check_connection import get_site_config, get_config
+
+def list_directories(path):
+    directories = []
+    for name in os.listdir(path):
+        if os.path.isdir(os.path.join(path, name)):
+            directories.append(name)
+    return directories
+
+def get_backup_dir():
+    return os.path.join(
+        os.path.expanduser('~'),
+        'backups'
+    )
+
+def decompress_db(files_base, site):
+    database_file = files_base + '-database.sql.gz'
+    config = get_config()
+    site_config = get_site_config(site)
+    db_root_user = os.environ.get('DB_ROOT_USER', 'root')
+    command = 'gunzip -c {database_file} > {database_extract}'.format(
+        database_file=database_file,
+        database_extract=database_file.replace('.gz','')
+    )
+
+    print('Extract Database GZip for site {}'.format(site))
+    os.system(command)
+
+def restore_database(files_base, site):
+    db_root_password = os.environ.get('MYSQL_ROOT_PASSWORD')
+    if not db_root_password:
+        print('Variable MYSQL_ROOT_PASSWORD not set')
+        exit(1)
+
+    db_root_user = os.environ.get("DB_ROOT_USER", 'root')
+
+    # restore database
+    database_file = files_base + '-database.sql.gz'
+    decompress_db(files_base, site)
+    config = get_config()
+    site_config = get_site_config(site)
+
+    # mysql command prefix
+    mysql_command = 'mysql -u{db_root_user} -h{db_host} -p{db_password} -e '.format(
+        db_root_user=db_root_user,
+        db_host=config.get('db_host'),
+        db_password=db_root_password
+    )
+
+    # drop db if exists for clean restore
+    drop_database = mysql_command + "\"DROP DATABASE IF EXISTS \`{db_name}\`;\"".format(
+        db_name=site_config.get('db_name')
+    )
+    os.system(drop_database)
+
+    # create db
+    create_database = mysql_command + "\"CREATE DATABASE IF NOT EXISTS \`{db_name}\`;\"".format(
+        db_name=site_config.get('db_name')
+    )
+    os.system(create_database)
+
+    # create user
+    create_user = mysql_command + "\"CREATE USER IF NOT EXISTS \'{db_name}\'@\'%\' IDENTIFIED BY \'{db_password}\'; FLUSH PRIVILEGES;\"".format(
+        db_name=site_config.get('db_name'),
+        db_password=site_config.get('db_password')
+    )
+    os.system(create_user)
+
+    # create user password
+    set_user_password = mysql_command + "\"UPDATE mysql.user SET authentication_string = PASSWORD('{db_password}') WHERE User = \'{db_name}\' AND Host = \'%\';\"".format(
+        db_name=site_config.get('db_name'),
+        db_password=site_config.get('db_password')
+    )
+    os.system(set_user_password)
+
+    # grant db privileges to user
+    grant_privileges = mysql_command + "\"GRANT ALL PRIVILEGES ON \`{db_name}\`.* TO '{db_name}'@'%'; FLUSH PRIVILEGES;\"".format(
+        db_name=site_config.get('db_name')
+    )
+    os.system(grant_privileges)
+
+    command = "mysql -u{db_root_user} -h{db_host} -p{db_password} '{db_name}' < {database_file}".format(
+        db_root_user=db_root_user,
+        db_host=config.get('db_host'),
+        db_password=db_root_password,
+        db_name=site_config.get('db_name'),
+        database_file=database_file.replace('.gz',''),
+    )
+
+    print('Restoring database for site: {}'.format(site))
+    os.system(command)
+
+def restore_files(files_base):
+    public_files = files_base + '-files.tar'
+    # extract tar
+    public_tar = tarfile.open(public_files)
+    print('Extracting {}'.format(public_files))
+    public_tar.extractall()
+
+def restore_private_files(files_base):
+    private_files = files_base + '-private-files.tar'
+    private_tar = tarfile.open(private_files)
+    print('Extracting {}'.format(private_files))
+    private_tar.extractall()
+
+def pull_backup_from_s3():
+    check_environment_variables()
+
+    # https://stackoverflow.com/a/54672690
+    s3 = boto3.resource(
+        's3',
+        aws_access_key_id=os.environ.get('ACCESS_KEY_ID'),
+        aws_secret_access_key=os.environ.get('SECRET_ACCESS_KEY'),
+        endpoint_url=os.environ.get('ENDPOINT_URL')
+    )
+
+    bucket_dir = os.environ.get('BUCKET_DIR')
+    bucket_name = os.environ.get('BUCKET_NAME')
+    bucket = s3.Bucket(bucket_name)
+
+    # Change directory to /home/frappe/backups
+    os.chdir(get_backup_dir())
+
+    for obj in bucket.objects.filter(Prefix = bucket_dir):
+        backup_file = obj.key.replace(os.path.join(bucket_dir,''),'')
+        if not os.path.exists(os.path.dirname(backup_file)):
+            os.makedirs(os.path.dirname(backup_file))
+        print('Downloading {}'.format(backup_file))
+        bucket.download_file(obj.key, backup_file)
+
+    os.chdir(os.path.join(os.path.expanduser('~'), 'frappe-bench', 'sites'))
+
+def main():
+    backup_dir = get_backup_dir()
+
+    if len(list_directories(backup_dir)) == 0:
+        pull_backup_from_s3()
+
+    for site in list_directories(backup_dir):
+        site_slug = site.replace('.','_')
+        backups = [datetime.datetime.strptime(backup, DATE_FORMAT) for backup in list_directories(os.path.join(backup_dir,site))]
+        latest_backup = max(backups).strftime(DATE_FORMAT)
+        files_base = os.path.join(backup_dir, site, latest_backup, '')
+        files_base += latest_backup + '-' + site_slug
+        if site in get_sites():
+            restore_database(files_base, site)
+            restore_private_files(files_base)
+            restore_files(files_base)
+        else:
+            mariadb_root_password = os.environ.get('MYSQL_ROOT_PASSWORD')
+            if not mariadb_root_password:
+                print('Variable MYSQL_ROOT_PASSWORD not set')
+                exit(1)
+            mariadb_root_username = os.environ.get('DB_ROOT_USER', 'root')
+            database_file = files_base + '-database.sql.gz'
+
+            site_config = get_conf_params(
+                db_name='_' + hashlib.sha1(site.encode()).hexdigest()[:16],
+                db_password=random_string(16)
+            )
+
+            frappe.local.site = site
+            frappe.local.sites_path = os.getcwd()
+            frappe.local.site_path = os.getcwd() + '/' + site
+            make_conf(
+                db_name=site_config.get('db_name'),
+                db_password=site_config.get('db_password'),
+            )
+            make_site_dirs()
+            restore_database(files_base, site)
+            restore_private_files(files_base)
+            restore_files(files_base)
+
+    exit(0)
+
+if __name__ == "__main__":
+    main()

build/common/commands/worker.py

@@ -1,7 +1,10 @@
 import os, frappe
 from frappe.utils.background_jobs import start_worker
 
+def main():
     queue = os.environ.get("WORKER_TYPE", "default")
     start_worker(queue, False)
-
     exit(0)
+
+if __name__ == "__main__":
+    main()

build/common/nginx-default.conf.template

@@ -29,9 +29,9 @@ server {
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";
-        proxy_set_header X-Frappe-Site-Name $http_host;
+        proxy_set_header X-Frappe-Site-Name $host;
         proxy_set_header Origin $scheme://$http_host;
-        proxy_set_header Host $host;
+        proxy_set_header Host $http_host;
 
         proxy_pass http://socketio-server;
     }
@@ -52,8 +52,8 @@ server {
     location @webserver {
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_set_header X-Frappe-Site-Name $http_host;
-        proxy_set_header Host $host;
+        proxy_set_header X-Frappe-Site-Name $host;
+        proxy_set_header Host $http_host;
         proxy_set_header X-Use-X-Accel-Redirect True;
         proxy_read_timeout 120;
         proxy_redirect off;

build/common/worker/docker-entrypoint.sh

@@ -76,6 +76,10 @@ if [ "$1" = 'start' ]; then
     export FRAPPE_PORT=8000
   fi
 
+  if [[ ! -z "$AUTO_MIGRATE" ]]; then
+    su frappe -c ". /home/frappe/frappe-bench/env/bin/activate \
+      && python /home/frappe/frappe-bench/commands/auto_migrate.py"
+  fi
 
   if [[ -z "$RUN_AS_ROOT" ]]; then
     su frappe -c ". /home/frappe/frappe-bench/env/bin/activate \
@@ -171,6 +175,18 @@ elif [ "$1" = 'console' ]; then
     python /home/frappe/frappe-bench/commands/console.py "$2"
   fi
 
+elif [ "$1" = 'push-backup' ]; then
+
+  su frappe -c ". /home/frappe/frappe-bench/env/bin/activate \
+    && python /home/frappe/frappe-bench/commands/push_backup.py"
+  exit
+
+elif [ "$1" = 'restore-backup' ]; then
+
+  su frappe -c ". /home/frappe/frappe-bench/env/bin/activate \
+    && python /home/frappe/frappe-bench/commands/restore_backup.py"
+  exit
+
 else
 
   exec su frappe -c "$@"

build/erpnext-nginx/install_app.sh

@@ -27,5 +27,6 @@ mkdir -p /home/frappe/frappe-bench/sites/assets/${APP_NAME}
 cp -R /home/frappe/frappe-bench/apps/${APP_NAME}/${APP_NAME}/public/* /home/frappe/frappe-bench/sites/assets/${APP_NAME}
 
 echo "rsync -a --delete /var/www/html/assets/${APP_NAME} /assets" > /rsync
+chmod +x /rsync
 
 rm /home/frappe/frappe-bench/sites/apps.txt

build/erpnext-nginx/v11.Dockerfile

@@ -8,7 +8,7 @@ FROM frappe/frappe-nginx:v11
 
 COPY --from=0 /home/frappe/frappe-bench/sites/ /var/www/html/
 COPY --from=0 /rsync /rsync
-RUN echo -n "\nerpnext" >> /home/frappe/frappe-bench/sites/apps.txt
+RUN echo -n "\nerpnext" >> /var/www/html/apps.txt
 
 VOLUME [ "/assets" ]
 

build/erpnext-nginx/v12.Dockerfile

@@ -8,7 +8,7 @@ FROM frappe/frappe-nginx:v12
 
 COPY --from=0 /home/frappe/frappe-bench/sites/ /var/www/html/
 COPY --from=0 /rsync /rsync
-RUN echo -n "\nerpnext" >> /home/frappe/frappe-bench/sites/apps.txt
+RUN echo -n "\nerpnext" >> /var/www/html/apps.txt
 
 VOLUME [ "/assets" ]
 

build/frappe-nginx/Dockerfile

@@ -32,7 +32,9 @@ COPY --from=0 /var/www/error_pages /var/www/
 COPY build/common/nginx-default.conf.template /etc/nginx/conf.d/default.conf.template
 COPY build/frappe-nginx/docker-entrypoint.sh /
 
-RUN apt-get update && apt-get install -y rsync && apt-get clean
+RUN apt-get update && apt-get install -y rsync && apt-get clean \
+    && echo "#!/bin/bash" > /rsync \
+    && chmod +x /rsync
 
 VOLUME [ "/assets" ]
 

build/frappe-nginx/v11.Dockerfile

@@ -32,7 +32,9 @@ COPY --from=0 /var/www/error_pages /var/www/
 COPY build/common/nginx-default.conf.template /etc/nginx/conf.d/default.conf.template
 COPY build/frappe-nginx/docker-entrypoint.sh /
 
-RUN apt-get update && apt-get install -y rsync && apt-get clean
+RUN apt-get update && apt-get install -y rsync && apt-get clean \
+    && echo "#!/bin/bash" > /rsync \
+    && chmod +x /rsync
 
 VOLUME [ "/assets" ]
 

build/frappe-nginx/v12.Dockerfile

@@ -32,7 +32,9 @@ COPY --from=0 /var/www/error_pages /var/www/
 COPY build/common/nginx-default.conf.template /etc/nginx/conf.d/default.conf.template
 COPY build/frappe-nginx/docker-entrypoint.sh /
 
-RUN apt-get update && apt-get install -y rsync && apt-get clean
+RUN apt-get update && apt-get install -y rsync && apt-get clean \
+    && echo "#!/bin/bash" > /rsync \
+    && chmod +x /rsync
 
 VOLUME [ "/assets" ]
 

build/frappe-worker/Dockerfile

@@ -21,7 +21,7 @@ RUN install_packages \
 RUN wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.5/wkhtmltox_0.12.5-1.stretch_amd64.deb
 RUN dpkg -i wkhtmltox_0.12.5-1.stretch_amd64.deb && rm wkhtmltox_0.12.5-1.stretch_amd64.deb
 
-RUN mkdir -p apps logs commands
+RUN mkdir -p apps logs commands /home/frappe/backups
 
 RUN virtualenv env \
     && . env/bin/activate \
@@ -40,5 +40,9 @@ COPY build/common/worker/install_app.sh /usr/local/bin/install_app
 
 WORKDIR /home/frappe/frappe-bench/sites
 
+RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites /home/frappe/backups
+
+VOLUME [ "/home/frappe/frappe-bench/sites", "/home/frappe/backups" ]
+
 ENTRYPOINT ["docker-entrypoint.sh"]
 CMD ["start"]

build/frappe-worker/v11.Dockerfile

@@ -18,7 +18,7 @@ RUN install_packages \
 RUN wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.5/wkhtmltox_0.12.5-1.stretch_amd64.deb
 RUN dpkg -i wkhtmltox_0.12.5-1.stretch_amd64.deb && rm wkhtmltox_0.12.5-1.stretch_amd64.deb
 
-RUN mkdir -p apps logs commands
+RUN mkdir -p apps logs commands /home/frappe/backups
 
 RUN virtualenv env \
     && . env/bin/activate \
@@ -37,5 +37,9 @@ COPY build/common/worker/install_app.sh /usr/local/bin/install_app
 
 WORKDIR /home/frappe/frappe-bench/sites
 
+RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites /home/frappe/backups
+
+VOLUME [ "/home/frappe/frappe-bench/sites", "/home/frappe/backups" ]
+
 ENTRYPOINT ["docker-entrypoint.sh"]
 CMD ["start"]

build/frappe-worker/v12.Dockerfile

@@ -21,7 +21,7 @@ RUN install_packages \
 RUN wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.5/wkhtmltox_0.12.5-1.stretch_amd64.deb
 RUN dpkg -i wkhtmltox_0.12.5-1.stretch_amd64.deb && rm wkhtmltox_0.12.5-1.stretch_amd64.deb
 
-RUN mkdir -p apps logs commands
+RUN mkdir -p apps logs commands /home/frappe/backups
 
 RUN virtualenv env \
     && . env/bin/activate \
@@ -40,5 +40,9 @@ COPY build/common/worker/install_app.sh /usr/local/bin/install_app
 
 WORKDIR /home/frappe/frappe-bench/sites
 
+RUN chown -R frappe:frappe /home/frappe/frappe-bench/sites /home/frappe/backups
+
+VOLUME [ "/home/frappe/frappe-bench/sites", "/home/frappe/backups" ]
+
 ENTRYPOINT ["docker-entrypoint.sh"]
 CMD ["start"]

development/README.md

@@ -103,6 +103,16 @@ bench set-config developer_mode 1
 bench clear-cache
 ```
 
+### Start development
+
+Execute following command from the `frappe-bench` directory.
+
+```shell
+bench start
+```
+
+Note: To start bench with debugger refer section for debugging.
+
 ### Fixing MariaDB issues after rebuilding the container
 
 The `bench new-site` command creates a user in MariaDB with container IP as host, for this reason after rebuilding the container there is a chance that you will not be able to access MariaDB correctly with the previous configuration

greetings.yml

@@ -0,0 +1,15 @@
+name: Greetings
+
+on: [pull_request, issues]
+
+jobs:
+  greeting:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/first-interaction@v1
+      with:
+        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        issue-message: |
+          Hello! We're very happy to see your first issue. If your issue is about a problem, go back and check you have copy-pasted all the debug logs you can so we can help you as fast as possible!
+        pr-message: |
+          Hello! Thank you about this PR. Since this is your first PR, please make sure you have described the improvements and your code is well documented.

installation/docker-compose-erpnext.yml

@@ -37,6 +37,7 @@ services:
       - REDIS_QUEUE=redis-queue:6379
       - REDIS_SOCKETIO=redis-socketio:6379
       - SOCKETIO_PORT=9000
+      - AUTO_MIGRATE=1
     volumes:
       - ./sites:/home/frappe/frappe-bench/sites:rw
       - assets-vol:/home/frappe/frappe-bench/sites/assets:rw

installation/docker-compose-frappe.yml

@@ -37,6 +37,7 @@ services:
       - REDIS_QUEUE=redis-queue:6379
       - REDIS_SOCKETIO=redis-socketio:6379
       - SOCKETIO_PORT=9000
+      - AUTO_MIGRATE=1
     volumes:
       - ./sites:/home/frappe/frappe-bench/sites:rw
       - assets-vol:/home/frappe/frappe-bench/sites/assets:rw