ERPNext v16 – Build, Update & Deployment

This document describes the complete update process for the custom ERPNext v16 installation, including:

  • application updates
  • fixture exports
  • translation generation and maintenance
  • Docker image builds
  • custom image tags
  • pushing images to Gitea
  • production deployment
  • database migrations
  • verification
  • rollback

The production image is stored in:

git.prothmann.com/christian/erpnexthp

The image contains ERPNext/Frappe v16 together with the required custom applications.

Current custom applications include:

manufacturing_utilities
eu_einvoice

Legacy applications such as the old standalone:

manufacturing_overview
label_printing

should not be included in new clean builds once their functionality has been migrated into manufacturing_utilities.


1. General workflow

The normal update sequence is:

1. Finish development
2. Export fixtures
3. Update translations
4. Commit changes
5. Build Docker image on Mac
6. Push image to Gitea
7. Back up production and record the rollback target
8. Select the tag, regenerate and validate Compose on hyp1, then pull
9. Start updated containers
10. Run bench migrate
11. Clear cache
12. Verify scheduler/workers
13. Test ERPNext

Do not build the ERPNext image directly on the production server.

The image is built on the Mac and production only pulls the finished image.


2. Development preparation

Before creating a new Docker image, make sure all application changes are committed.

From the bench/app development environment, check:

git status

Review all changed files before continuing.


3. Export fixtures

Fixtures must be exported whenever something managed through ERPNext has changed that belongs in the app.

For manufacturing_utilities:

bench --site development.localhost export-fixtures --app manufacturing_utilities

This includes, among other things, exported Custom HTML Blocks and Custom Fields configured in hooks.py.

For example, the Manufacturing Overview Custom HTML Block is exported through the app fixtures.

After exporting:

git status

Check that the expected fixture files changed.

Then commit them together with the related code.


4. Translation workflow

ERPNext/Frappe v16 uses Gettext .po translation files.

Translations for manufacturing_utilities should be stored in the application repository rather than maintained manually through the ERPNext Translation DocType.

4.1 Generate/update translatable strings

Whenever new translatable strings are added:

bench generate-pot-file --app manufacturing_utilities

Then update the existing language .po files:

bench update-po-files --app manufacturing_utilities

The usual workflow is therefore:

bench generate-pot-file --app manufacturing_utilities
bench update-po-files --app manufacturing_utilities

5. Creating a new language file

This is only required once for a new language.

For German:

bench create-po-file de --app manufacturing_utilities

Afterwards, normally use:

bench update-po-files --app manufacturing_utilities

instead.


6. Editing German translations

The German .po file is stored inside the application locale directory.

Translate entries such as:

msgid "Manufacturing"
msgstr "Fertigung"

Do not modify the msgid.

Only modify the corresponding:

msgstr

7. Important translation rule

Frappe's translation extractor needs literal strings.

Good:

__("Manufacturing");

Good:

_("Manufacturing")

Bad:

__(status);

because the extractor cannot know which value status will contain.

For dynamic status values, expose the possible strings explicitly:

const statusLabels = {
  Draft: __("Draft"),
  "To Transfer": __("To Transfer"),
  "Material Transferred": __("Material Transferred"),
};

The Manufacturing Overview additionally has:

manufacturing_utilities/manufacturing_overview/translations.py

which exposes Custom HTML Block strings to Frappe's translation extractor.

Whenever a new user-facing string is introduced into the Manufacturing Overview, add it there as well.


8. Compile translations

After editing .po files:

bench compile-po-to-mo --app manufacturing_utilities

When working in development, also rebuild the application assets:

bench build --app manufacturing_utilities

Then clear the site cache:

bench --site development.localhost clear-cache

Run the catalog refresh, edit the PO files, then compile and test.

bench generate-pot-file --app manufacturing_utilities
bench update-po-files --app manufacturing_utilities
# Edit and review msgstr values before compiling.
bench compile-po-to-mo --app manufacturing_utilities
bench build --app manufacturing_utilities
bench --site development.localhost clear-cache

Then hard-refresh the browser.


9. Before building a production image

Make sure the repository is clean:

git status

Commit the final changes:

git add -p
# Explicitly stage any new fixture or locale files after review.
git commit -m "Update Manufacturing Utilities"

Push the repository:

git push

The Docker build should always use committed source code.


10. Docker image architecture

Development/build machine:

MacBook Pro M1 Pro
arm64

Production:

Debian
linux/amd64

Therefore the image must be built for:

linux/amd64

The build process uses Docker Buildx.

Conceptually:

docker buildx build \
    --platform linux/amd64 \
    --load \
    ...

The existing build script handles this.


11. Standard image tags

The normal image tag should match the ERPNext version.

Example:

16.55.0

Result:

git.prothmann.com/christian/erpnexthp:16.55.0

Normal build:

./build.sh

Run from the checkout containing the existing build.sh. Its location and current options should be recorded in the release notes after checking the script.

The resulting image should normally use the ERPNext version as its tag.


12. Custom image tags

Use a custom tag whenever testing a special build or when multiple images based on the same ERPNext release need to coexist.

Example:

./build.sh --tag 16.55.0-no-legacy

This creates:

git.prothmann.com/christian/erpnexthp:16.55.0-no-legacy

Useful examples:

16.55.0-test
16.55.0-no-legacy
16.55.0-migration-test
16.55.0-clean

A custom tag should never silently overwrite the normal ERPNext-version tag unless this is explicitly intended.

This makes rollback and parallel testing much easier.


13. Important image dependencies

The image must include the dependencies required for network printing.

Required Debian package:

libcups2-dev

Required Python package:

pycups

This must be baked into the Docker image.

Do not rely on installing these manually inside a running production container because the installation would disappear when the container is recreated.

To verify pycups inside a container:

bench pip show pycups

14. Build the image

Run the build from the Mac.

Normal example:

./build.sh

Custom tag example:

./build.sh --tag 16.55.0-no-legacy

The final image should exist locally as:

git.prothmann.com/christian/erpnexthp:<TAG>

Set IMAGE to the full reference for the tag you just built. Require linux/amd64 in the inspection output.

TAG=16.55.0-no-legacy  # Example. Use your actual build tag.
IMAGE="git.prothmann.com/christian/erpnexthp:$TAG"
docker image inspect --format '{{.Os}}/{{.Architecture}} {{.Id}}' "$IMAGE"

15. Push image to Gitea

Push the finished image:

docker login git.prothmann.com
docker push "$IMAGE"

Example:

docker push git.prothmann.com/christian/erpnexthp:16.55.0-no-legacy

The Gitea container registry is:

git.prothmann.com/christian/erpnexthp

Use the Gitea credential authorized to publish the package. Record the digest printed by docker push, and keep the previously deployed tag and digest available. Production pulls the published image.


16. Release checks before touching hyp1

  • Record the release tag and registry digest, Frappe/ERPNext and custom app refs, build checkout and command, and the app manifest used by the build. Use a fresh tag for each distinct image, including rebuilds against the same ERPNext release.
  • Confirm that the actual build inputs fetch the commits you pushed. Check the image contains frappe, erpnext, manufacturing_utilities and eu_einvoice, with compiled translations and built assets. Verify bench pip show pycups and an actual import cups with the Bench Python environment in the resulting image.
  • On a restored test copy, run the migration and acceptance checks below before a significant version change or app removal. Keep test integrations from sending production mail, print jobs or e-invoices.

Legacy apps

Before deploying an image without manufacturing_overview or label_printing, check list-apps on every site served by this stack. If either app is still installed, use a transitional image that contains it while moving and validating any required data, fixtures and functionality. Rehearse on a restored copy and take a fresh backup before uninstalling in production. Uninstall can delete app-owned DocTypes and data.

On the image that still contains the app, inspect the removal first. Example for one app on the production site:

erp_dc exec backend bench --site erp.prothmann.com uninstall-app --help
erp_dc exec backend bench --site erp.prothmann.com uninstall-app manufacturing_overview --dry-run

After reviewing the dry run and approving the deletions, run the uninstall without --dry-run, migrate, and confirm with list-apps. Repeat separately for label_printing if applicable. Build or deploy the clean image once every affected site is ready. Treat uninstall as a database-changing operation for rollback purposes.

17. hyp1 preflight and backup

SSH to hyp1 with the usual production account. Run the following in Bash, from the deployment directory. Keep your session open or redefine the function after reconnecting.

cd /srv/docker/erpnext-16
docker compose version
docker compose ls
docker compose ps -a

Confirm the running project name from docker compose ls and its configuration files. If the default invocation does not find the expected stack, resolve the project name before continuing. Enter the existing project name when prompted.

read -r -p 'Existing production Compose project name: ' COMPOSE_PROJECT
erp_dc() {
    docker compose --project-name "${COMPOSE_PROJECT:?Set the existing project name}" \
        --project-directory /srv/docker/erpnext-16 --env-file .env \
        -f docker-compose.yml -f docker-compose.override.yml "$@"
}
erp_dc ps -a
erp_dc config --images
erp_dc config --services
erp_dc exec backend bench --site erp.prothmann.com list-apps
erp_dc exec backend bench --site erp.prothmann.com scheduler status

Record the running images and digests (including DB and Redis), service names, Compose project, actual volume and network names, installed apps and scheduler state. Use running container inspection if configured and running images differ. Check disk space and that the external Traefik proxy network exists. Check for extra sites before deployment. If the stack serves multiple sites, plan backup, migration and verification for each.

Schedule a maintenance window. Prevent traffic and other writes, let in-flight requests and background jobs finish, and pause scheduling if it was active. Record which settings you change so you can restore them. For example:

erp_dc exec backend bench --site erp.prothmann.com set-maintenance-mode on
erp_dc exec backend bench --site erp.prothmann.com scheduler pause
erp_dc exec backend bench --site erp.prothmann.com doctor
erp_dc exec backend bench --site erp.prothmann.com ready-for-migration

Check existing worker queues and coordinate integrations that can still write before taking the final backup. Leave services required for draining jobs running until the drain is complete.

umask 077
RECOVERY_DIR="$(mktemp -d "$PWD/recovery-$(date +%Y%m%d-%H%M%S)-XXXXXX")" &&
    cp -p .env docker-compose.yml docker-compose.override.yml "$RECOVERY_DIR/" &&
    git -C /srv/docker/frappe_docker-source rev-parse HEAD > "$RECOVERY_DIR/frappe_docker-revision"

Stop if that block fails. Record the recovery directory, then back up the site:

erp_dc exec backend bench --site erp.prothmann.com backup --with-files
erp_dc exec backend ls -lh sites/erp.prothmann.com/private/backups/

Copy the exact files from this run (database, public and private files, and site configuration) out of the sites volume. For each file, use erp_dc cp backend:sites/erp.prothmann.com/private/backups/ACTUAL_FILENAME "$RECOVERY_DIR/", replacing ACTUAL_FILENAME with the name reported by Bench. Expect dated files, and verify that the database and file archives are readable and nonempty. Verify that the recovery set includes the original site_config.json and its encryption key. Save the original configuration separately if the backup omits it. Preserve sites/common_site_config.json and any referenced secrets or extra environment files with the deployment snapshot. Store an access-restricted copy away from hyp1 and its Docker volumes, with checksums and a recorded restore location. Confirm you can retrieve it before proceeding.

18. Generate and validate production Compose on hyp1

Update the official upstream checkout

cd /srv/docker/frappe_docker-source
git remote -v
git status --short
git branch --show-current

Confirm the checkout is the official frappe/frappe_docker source and the intended branch, with no unexpected local changes. Then update it.

git pull --ff-only && git rev-parse HEAD

If the pull fails, resolve the checkout before generating anything. Record the resulting commit and review changes to the upstream base, MariaDB and Redis overrides since the deployed revision. A source pull can change infrastructure image versions, health checks, mount paths, and startup dependencies. Keep the existing DB and Redis versions or digests for an app-only release. Schedule and test intentional infrastructure upgrades separately.

Select the release and render a candidate

cd /srv/docker/erpnext-16
grep -nE '^(CUSTOM_TAG|CUSTOM_IMAGE|ERPNEXT_VERSION)=' .env

Set CUSTOM_TAG in .env to the exact pushed tag, for example CUSTOM_TAG=16.55.0-no-legacy. Check how the override selects the image. If it hard-codes a tag, update its corresponding image references. Preserve real DB credentials, Traefik routing, external proxy network, and production volume mappings. Check for inherited shell variables that would override values from .env, especially image and tag variables.

Run from /srv/docker/erpnext-16. A temporary file keeps a failed render from truncating the current base.

candidate="$(mktemp "$PWD/docker-compose.generated.XXXXXX.yml")" &&
    DB_PASSWORD=COMPOSE_TEMPLATE_ONLY docker compose \
        --env-file .env \
        -f /srv/docker/frappe_docker-source/compose.yaml \
        -f /srv/docker/frappe_docker-source/overrides/compose.mariadb.yaml \
        -f /srv/docker/frappe_docker-source/overrides/compose.redis.yaml \
        config > "$candidate" &&
    test -s "$candidate"

Stop if rendering fails. COMPOSE_TEMPLATE_ONLY supplies DB_PASSWORD for that render command alone. The local docker-compose.override.yml supplies the real production value. Check the merged result for any surviving placeholder, including DB environment variables and health checks. Correct the override before applying if any remain. Always regenerate after a tag change when the base resolves images from .env.

Validate the candidate with the local override

candidate_dc() {
    docker compose --project-name "${COMPOSE_PROJECT:?Set the existing project name}" \
        --project-directory /srv/docker/erpnext-16 --env-file .env \
        -f "${candidate:?Render the candidate first}" -f docker-compose.override.yml "$@"
}
candidate_dc config --quiet
candidate_dc config --images
candidate_dc config --services

Inspect the full candidate_dc config privately. It can contain real secrets. Compare with the saved deployment and current runtime. Confirm:

  • Every ERPNext service that uses the custom image resolves to the intended Gitea reference, including backend, frontend, websocket, scheduler, workers and the configurator where applicable.
  • MariaDB and Redis image versions, service endpoints and health checks match the release plan. Use the same persistent MariaDB, Redis and sites volumes and verify any bind mount paths. Rendered relative paths may have been resolved against the upstream source checkout.
  • The existing Compose project and external network names, Traefik labels, host routing for erp.prothmann.com, ports and proxy attachment remain correct.
  • The local override supplies the effective production DB settings and no COMPOSE_TEMPLATE_ONLY remains in the merged result. Check existing site DB credentials and encryption keys are preserved.

After the review:

mv -- "$candidate" docker-compose.yml
erp_dc config --quiet
erp_dc config --images
docker login git.prothmann.com
erp_dc pull

Use a Gitea credential authorized to pull the private package. If pull fails, stop before starting the update. Inspect docker image inspect FULL_NEW_IMAGE_REFERENCE using the actual Gitea reference and verify its repository digest against the push record. Recheck MariaDB and Redis digests if their tags are mutable. Keep the old image available.

19. Apply, migrate and verify

In the maintenance window, apply the validated stack using the images just pulled and checked. The explicit pull policy prevents another registry fetch during startup.

erp_dc up -d --pull never
erp_dc ps -a

Confirm backend readiness and check database/Redis health. An exited configurator can be expected if it completed successfully. Check its exit code and logs. Keep production writes blocked while migration runs.

erp_dc exec backend bench --site erp.prothmann.com migrate

If migration fails, keep maintenance active and diagnose before proceeding. Capture the failing patch or fixture and relevant logs. A failed migration can already have changed schema or data. Avoid skipping failing patches as part of routine recovery.

erp_dc exec backend bench --site erp.prothmann.com clear-cache
# When website cache is relevant:
erp_dc exec backend bench --site erp.prothmann.com clear-website-cache
erp_dc exec backend bench --site erp.prothmann.com list-apps
erp_dc ps -a
erp_dc logs --since=15m --tail=100

Confirm running image identities against the release record and check for DB, Redis, permission, asset, fixture, websocket and queue errors. Use erp_dc config --services to find actual service names and narrow logs as needed. Confirm scheduler and worker processes are up. Restore maintenance mode and scheduler pause to their recorded settings after migration and infrastructure checks pass. For a site that was originally live with an unpaused scheduler:

erp_dc exec backend bench --site erp.prothmann.com set-maintenance-mode off
erp_dc exec backend bench --site erp.prothmann.com scheduler resume
erp_dc exec backend bench --site erp.prothmann.com scheduler status
erp_dc exec backend bench --site erp.prothmann.com doctor

If scheduling is unexpectedly disabled, diagnose its settings. Use bench --site erp.prothmann.com enable-scheduler inside the backend only when the site is intended to schedule jobs and its scheduler is disabled. Verify a known safe background job completes and the expected workers are online. If traffic was restricted externally, reopen it once smoke tests pass.

Acceptance checklist

  • HTTPS, login, Desk, static assets and realtime features work through Traefik.
  • Manufacturing Utilities workspace and Manufacturing Overview load.
  • PRO / Work Order and Sales Order links work. Planned End Date sorting is correct.
  • Subcontracting cards, draft Purchase Orders and submitted subcontracting orders appear.
  • German translations appear after selecting German and hard-refreshing. Check a newly translated Manufacturing Overview label.
  • Label Composer opens, preview renders, and an authorized test label prints on the network printer.
  • E-invoice functionality opens and passes a non-sending smoke test.
  • Scheduler is in its intended state, workers process jobs, and logs show no new recurring errors.
  • Release record has the deployed digest, upstream Compose revision, backup location, and verification result.

20. Rollback and recovery

Keep writes blocked while deciding how to recover. Capture service status, logs and the failing release details. If the failure happened before any schema, data or infrastructure change and the old image is compatible with current state, restore the saved pair of Compose files and saved .env from the same release. Include any other changed deployment files and preserve file permissions. Verify the previous config and image digests before starting it.

cd /srv/docker/erpnext-16
# Set RECOVERY_DIR to the verified pre-release snapshot if reconnecting.
cp -p "${RECOVERY_DIR:?Set the recovery directory}/.env" \
    "$RECOVERY_DIR/docker-compose.yml" \
    "$RECOVERY_DIR/docker-compose.override.yml" .
erp_dc config --quiet
erp_dc config --images
# Pull only if needed, and verify the previous digest before applying.
erp_dc up -d --pull never
erp_dc ps -a

A prior image cannot undo migrate, app uninstall, data patches, or MariaDB on-disk upgrades. If the old image is incompatible with the modified state, preserve a copy of the failed state and establish what writes occurred since the backup. Restore the verified pre-update SQL and both file archives together with the matching site configuration/encryption key, using the old compatible application and database versions. Confirm the exact bench --site erp.prothmann.com restore --help on that release and rehearse with the saved files on a separate recovery target before overwriting production. For an incompatible MariaDB storage upgrade, recover onto compatible storage rather than starting an older server on the upgraded data directory. Recheck queues and pending integrations before resuming them. Run the acceptance checklist after recovery.

Never run docker compose down -v during this workflow. Preserve DB, Redis and sites volumes, public and private files, DB credentials, encryption keys, and Traefik networks. Keep the previous image and recovery artifacts until the release has passed verification.

Release checklist

  1. Export fixtures when changed, refresh Gettext catalogs, translate, compile, test and push the source.
  2. Build on the Mac for linux/amd64, verify apps and printing dependencies, push the unique Gitea tag and record its digest.
  3. On hyp1 identify the current project, block writes, drain jobs and verify the backup and rollback snapshot.
  4. Update the official source checkout, select the new tag, render the base with COMPOSE_TEMPLATE_ONLY, then validate it with the production override.
  5. Install the validated base, pull, compare digests, run up -d, migrate and clear caches.
  6. Verify scheduler, workers and application behavior. Restore normal traffic and record the outcome.

References

S
Description
No description provided
Readme
62 KiB
0 Stars 1 Watchers 0 Forks
Languages
Shell 100%