Added DATEV app

This commit is contained in:
Christian Anetzberger committed 2026-11-14 18:49:11 +01:00
1 parent 50c2e024a3
commit 21befef8d0
2 files changed
+35 -665

No files matched your search

+31 -665
View File
@@ -1,700 +1,66 @@
# ERPNext v16 – Build, Update & Deployment
# Deploy ERPNext v16
This document describes the complete update process for the custom ERPNext v16 installation, including:
## 1. Prepare the app
- application updates
- fixture exports
- translation generation and maintenance
- Docker image builds
- custom image tags
- pushing images to Gitea
- production deployment
- database migrations
- verification
- rollback
The production image is stored in:
```text
git.prothmann.com/christian/erpnexthp
```
The image contains ERPNext/Frappe v16 together with the required custom applications.
Current custom applications include:
```text
manufacturing_utilities
eu_einvoice
```
Legacy applications such as the old standalone:
```text
manufacturing_overview
label_printing
```
should not be included in new clean builds once their functionality has been migrated into `manufacturing_utilities`.
---
## 1. General workflow
The normal update sequence is:
```text
1. Finish development
2. Export fixtures
3. Update translations
4. Commit changes
5. Build Docker image on Mac
6. Push image to Gitea
7. Back up production and record the rollback target
8. Select the tag, regenerate and validate Compose on hyp1, then pull
9. Start updated containers
10. Run bench migrate
11. Clear cache
12. Verify scheduler/workers
13. Test ERPNext
```
Do not build the ERPNext image directly on the production server.
The image is built on the Mac and production only pulls the finished image.
---
## 2. Development preparation
Before creating a new Docker image, make sure all application changes are committed.
From the bench/app development environment, check:
```bash
git status
```
Review all changed files before continuing.
---
## 3. Export fixtures
Fixtures must be exported whenever something managed through ERPNext has changed that belongs in the app.
For `manufacturing_utilities`:
From the development bench, export fixtures and update translations.
```bash
bench --site development.localhost export-fixtures --app manufacturing_utilities
```
This includes, among other things, exported Custom HTML Blocks and Custom Fields configured in `hooks.py`.
For example, the Manufacturing Overview Custom HTML Block is exported through the app fixtures.
After exporting:
```bash
git status
```
Check that the expected fixture files changed.
Then commit them together with the related code.
---
## 4. Translation workflow
ERPNext/Frappe v16 uses Gettext `.po` translation files.
Translations for `manufacturing_utilities` should be stored in the application repository rather than maintained manually through the ERPNext Translation DocType.
### 4.1 Generate/update translatable strings
Whenever new translatable strings are added:
```bash
bench generate-pot-file --app manufacturing_utilities
```
Then update the existing language `.po` files:
```bash
bench update-po-files --app manufacturing_utilities
```
The usual workflow is therefore:
```bash
bench generate-pot-file --app manufacturing_utilities
bench update-po-files --app manufacturing_utilities
```
---
## 5. Creating a new language file
This is only required once for a new language.
For German:
```bash
bench create-po-file de --app manufacturing_utilities
```
Afterwards, normally use:
```bash
bench update-po-files --app manufacturing_utilities
```
instead.
---
## 6. Editing German translations
The German `.po` file is stored inside the application locale directory.
Translate entries such as:
```po
msgid "Manufacturing"
msgstr "Fertigung"
```
Do not modify the `msgid`.
Only modify the corresponding:
```text
msgstr
```
---
## 7. Important translation rule
Frappe's translation extractor needs literal strings.
Good:
```javascript
__("Manufacturing");
```
Good:
```python
_("Manufacturing")
```
Bad:
```javascript
__(status);
```
because the extractor cannot know which value `status` will contain.
For dynamic status values, expose the possible strings explicitly:
```javascript
const statusLabels = {
Draft: __("Draft"),
"To Transfer": __("To Transfer"),
"Material Transferred": __("Material Transferred"),
};
```
The Manufacturing Overview additionally has:
```text
manufacturing_utilities/manufacturing_overview/translations.py
```
which exposes Custom HTML Block strings to Frappe's translation extractor.
Whenever a new user-facing string is introduced into the Manufacturing Overview, add it there as well.
---
## 8. Compile translations
After editing `.po` files:
Edit the `msgstr` entries in the app’s `locale/de.po`, then compile.
```bash
bench compile-po-to-mo --app manufacturing_utilities
```
When working in development, also rebuild the application assets:
```bash
bench build --app manufacturing_utilities
```
Then clear the site cache:
```bash
bench --site development.localhost clear-cache
```
Run the catalog refresh, edit the PO files, then compile and test.
```bash
bench generate-pot-file --app manufacturing_utilities
bench update-po-files --app manufacturing_utilities
# Edit and review msgstr values before compiling.
bench compile-po-to-mo --app manufacturing_utilities
bench build --app manufacturing_utilities
bench --site development.localhost clear-cache
```
Then hard-refresh the browser.
---
## 9. Before building a production image
Make sure the repository is clean:
```bash
git status
```
Commit the final changes:
```bash
git add -p
# Explicitly stage any new fixture or locale files after review.
git commit -m "Update Manufacturing Utilities"
```
Push the repository:
From the app repo, stage the app changes, fixtures, and translation sources, then commit and push.
```bash
git commit -m "Prepare ERPNext update"
git push
```
The Docker build should always use committed source code.
## 2. Build and push on the Mac
---
## 10. Docker image architecture
Development/build machine:
```text
MacBook Pro M1 Pro
arm64
```
Production:
```text
Debian
linux/amd64
```
Therefore the image must be built for:
```text
linux/amd64
```
The build process uses Docker Buildx.
Conceptually:
Run the existing build script for `linux/amd64`. Keep `libcups2-dev` and `pycups` baked into the image.
```bash
docker buildx build \
--platform linux/amd64 \
--load \
...
./build.sh # Default ERPNext version tag
# ./build.sh --tag 16.55.0-custom # Optional custom tag
docker push git.prothmann.com/christian/erpnexthp:TAG
```
The existing build script handles this.
Replace `TAG` with the tag produced by the build.
---
## 3. Deploy on hyp1
## 11. Standard image tags
The normal image tag should match the ERPNext version.
Example:
```text
16.55.0
```
Result:
```text
git.prothmann.com/christian/erpnexthp:16.55.0
```
Normal build:
Update `frappe_docker` and regenerate the base Compose file.
```bash
./build.sh
git -C /srv/docker/frappe_docker-source pull
cd /srv/docker/erpnext-16-prod
DB_PASSWORD=COMPOSE_TEMPLATE_ONLY docker compose --env-file .env \
-f /srv/docker/frappe_docker-source/compose.yaml \
-f /srv/docker/frappe_docker-source/overrides/compose.mariadb.yaml \
-f /srv/docker/frappe_docker-source/overrides/compose.redis.yaml \
config > docker-compose.yml.tmp && mv docker-compose.yml.tmp docker-compose.yml
```
Run from the checkout containing the existing `build.sh`. Its location and current options should be recorded in the release notes after checking the script.
The resulting image should normally use the ERPNext version as its tag.
---
## 12. Custom image tags
Use a custom tag whenever testing a special build or when multiple images based on the same ERPNext release need to coexist.
Example:
Record the previous image tag and set `CUSTOM_TAG=TAG` in `.env` to the tag you pushed. The local override supplies the production DB password. Validate the merged config before pulling.
```bash
./build.sh --tag 16.55.0-no-legacy
docker compose -f docker-compose.yml -f docker-compose.override.yml config --quiet
docker compose pull
docker compose up -d
docker compose exec backend bench --site erp.prothmann.com migrate
docker compose exec backend bench --site erp.prothmann.com clear-cache
docker compose ps
```
This creates:
Confirm the services are running, then log in at `erp.prothmann.com` and open Manufacturing Overview.
```text
git.prothmann.com/christian/erpnexthp:16.55.0-no-legacy
```
Useful examples:
```text
16.55.0-test
16.55.0-no-legacy
16.55.0-migration-test
16.55.0-clean
```
A custom tag should never silently overwrite the normal ERPNext-version tag unless this is explicitly intended.
This makes rollback and parallel testing much easier.
---
## 13. Important image dependencies
The image must include the dependencies required for network printing.
Required Debian package:
```text
libcups2-dev
```
Required Python package:
```text
pycups
```
This must be baked into the Docker image.
Do not rely on installing these manually inside a running production container because the installation would disappear when the container is recreated.
To verify pycups inside a container:
```bash
bench pip show pycups
```
---
## 14. Build the image
Run the build from the Mac.
Normal example:
```bash
./build.sh
```
Custom tag example:
```bash
./build.sh --tag 16.55.0-no-legacy
```
The final image should exist locally as:
```text
git.prothmann.com/christian/erpnexthp:<TAG>
```
Set `IMAGE` to the full reference for the tag you just built. Require `linux/amd64` in the inspection output.
```bash
TAG=16.55.0-no-legacy # Example. Use your actual build tag.
IMAGE="git.prothmann.com/christian/erpnexthp:$TAG"
docker image inspect --format '{{.Os}}/{{.Architecture}} {{.Id}}' "$IMAGE"
```
---
## 15. Push image to Gitea
Push the finished image:
```bash
docker login git.prothmann.com
docker push "$IMAGE"
```
Example:
```bash
docker push git.prothmann.com/christian/erpnexthp:16.55.0-no-legacy
```
The Gitea container registry is:
```text
git.prothmann.com/christian/erpnexthp
```
Use the Gitea credential authorized to publish the package. Record the digest printed by `docker push`, and keep the previously deployed tag and digest available. Production pulls the published image.
---
## 16. Release checks before touching hyp1
- Record the release tag and registry digest, Frappe/ERPNext and custom app refs, build checkout and command, and the app manifest used by the build. Use a fresh tag for each distinct image, including rebuilds against the same ERPNext release.
- Confirm that the actual build inputs fetch the commits you pushed. Check the image contains `frappe`, `erpnext`, `manufacturing_utilities` and `eu_einvoice`, with compiled translations and built assets. Verify `bench pip show pycups` and an actual `import cups` with the Bench Python environment in the resulting image.
- On a restored test copy, run the migration and acceptance checks below before a significant version change or app removal. Keep test integrations from sending production mail, print jobs or e-invoices.
### Legacy apps
Before deploying an image without `manufacturing_overview` or `label_printing`, check `list-apps` on every site served by this stack. If either app is still installed, use a transitional image that contains it while moving and validating any required data, fixtures and functionality. Rehearse on a restored copy and take a fresh backup before uninstalling in production. Uninstall can delete app-owned DocTypes and data.
On the image that still contains the app, inspect the removal first. Example for one app on the production site:
```bash
erp_dc exec backend bench --site erp.prothmann.com uninstall-app --help
erp_dc exec backend bench --site erp.prothmann.com uninstall-app manufacturing_overview --dry-run
```
After reviewing the dry run and approving the deletions, run the uninstall without `--dry-run`, migrate, and confirm with `list-apps`. Repeat separately for `label_printing` if applicable. Build or deploy the clean image once every affected site is ready. Treat uninstall as a database-changing operation for rollback purposes.
## 17. hyp1 preflight and backup
SSH to `hyp1` with the usual production account. Run the following in Bash, from the deployment directory. Keep your session open or redefine the function after reconnecting.
```bash
cd /srv/docker/erpnext-16
docker compose version
docker compose ls
docker compose ps -a
```
Confirm the running project name from `docker compose ls` and its configuration files. If the default invocation does not find the expected stack, resolve the project name before continuing. Enter the existing project name when prompted.
```bash
read -r -p 'Existing production Compose project name: ' COMPOSE_PROJECT
erp_dc() {
docker compose --project-name "${COMPOSE_PROJECT:?Set the existing project name}" \
--project-directory /srv/docker/erpnext-16 --env-file .env \
-f docker-compose.yml -f docker-compose.override.yml "$@"
}
erp_dc ps -a
erp_dc config --images
erp_dc config --services
erp_dc exec backend bench --site erp.prothmann.com list-apps
erp_dc exec backend bench --site erp.prothmann.com scheduler status
```
Record the running images and digests (including DB and Redis), service names, Compose project, actual volume and network names, installed apps and scheduler state. Use running container inspection if configured and running images differ. Check disk space and that the external Traefik proxy network exists. Check for extra sites before deployment. If the stack serves multiple sites, plan backup, migration and verification for each.
Schedule a maintenance window. Prevent traffic and other writes, let in-flight requests and background jobs finish, and pause scheduling if it was active. Record which settings you change so you can restore them. For example:
```bash
erp_dc exec backend bench --site erp.prothmann.com set-maintenance-mode on
erp_dc exec backend bench --site erp.prothmann.com scheduler pause
erp_dc exec backend bench --site erp.prothmann.com doctor
erp_dc exec backend bench --site erp.prothmann.com ready-for-migration
```
Check existing worker queues and coordinate integrations that can still write before taking the final backup. Leave services required for draining jobs running until the drain is complete.
```bash
umask 077
RECOVERY_DIR="$(mktemp -d "$PWD/recovery-$(date +%Y%m%d-%H%M%S)-XXXXXX")" &&
cp -p .env docker-compose.yml docker-compose.override.yml "$RECOVERY_DIR/" &&
git -C /srv/docker/frappe_docker-source rev-parse HEAD > "$RECOVERY_DIR/frappe_docker-revision"
```
Stop if that block fails. Record the recovery directory, then back up the site:
```bash
erp_dc exec backend bench --site erp.prothmann.com backup --with-files
erp_dc exec backend ls -lh sites/erp.prothmann.com/private/backups/
```
Copy the exact files from this run (database, public and private files, and site configuration) out of the sites volume. For each file, use `erp_dc cp backend:sites/erp.prothmann.com/private/backups/ACTUAL_FILENAME "$RECOVERY_DIR/"`, replacing `ACTUAL_FILENAME` with the name reported by Bench. Expect dated files, and verify that the database and file archives are readable and nonempty. Verify that the recovery set includes the original `site_config.json` and its encryption key. Save the original configuration separately if the backup omits it. Preserve `sites/common_site_config.json` and any referenced secrets or extra environment files with the deployment snapshot. Store an access-restricted copy away from hyp1 and its Docker volumes, with checksums and a recorded restore location. Confirm you can retrieve it before proceeding.
## 18. Generate and validate production Compose on hyp1
### Update the official upstream checkout
```bash
cd /srv/docker/frappe_docker-source
git remote -v
git status --short
git branch --show-current
```
Confirm the checkout is the official `frappe/frappe_docker` source and the intended branch, with no unexpected local changes. Then update it.
```bash
git pull --ff-only && git rev-parse HEAD
```
If the pull fails, resolve the checkout before generating anything. Record the resulting commit and review changes to the upstream base, MariaDB and Redis overrides since the deployed revision. A source pull can change infrastructure image versions, health checks, mount paths, and startup dependencies. Keep the existing DB and Redis versions or digests for an app-only release. Schedule and test intentional infrastructure upgrades separately.
### Select the release and render a candidate
```bash
cd /srv/docker/erpnext-16
grep -nE '^(CUSTOM_TAG|CUSTOM_IMAGE|ERPNEXT_VERSION)=' .env
```
Set `CUSTOM_TAG` in `.env` to the exact pushed tag, for example `CUSTOM_TAG=16.55.0-no-legacy`. Check how the override selects the image. If it hard-codes a tag, update its corresponding image references. Preserve real DB credentials, Traefik routing, external proxy network, and production volume mappings. Check for inherited shell variables that would override values from `.env`, especially image and tag variables.
Run from `/srv/docker/erpnext-16`. A temporary file keeps a failed render from truncating the current base.
```bash
candidate="$(mktemp "$PWD/docker-compose.generated.XXXXXX.yml")" &&
DB_PASSWORD=COMPOSE_TEMPLATE_ONLY docker compose \
--env-file .env \
-f /srv/docker/frappe_docker-source/compose.yaml \
-f /srv/docker/frappe_docker-source/overrides/compose.mariadb.yaml \
-f /srv/docker/frappe_docker-source/overrides/compose.redis.yaml \
config > "$candidate" &&
test -s "$candidate"
```
Stop if rendering fails. `COMPOSE_TEMPLATE_ONLY` supplies `DB_PASSWORD` for that render command alone. The local `docker-compose.override.yml` supplies the real production value. Check the merged result for any surviving placeholder, including DB environment variables and health checks. Correct the override before applying if any remain. Always regenerate after a tag change when the base resolves images from `.env`.
### Validate the candidate with the local override
```bash
candidate_dc() {
docker compose --project-name "${COMPOSE_PROJECT:?Set the existing project name}" \
--project-directory /srv/docker/erpnext-16 --env-file .env \
-f "${candidate:?Render the candidate first}" -f docker-compose.override.yml "$@"
}
candidate_dc config --quiet
candidate_dc config --images
candidate_dc config --services
```
Inspect the full `candidate_dc config` privately. It can contain real secrets. Compare with the saved deployment and current runtime. Confirm:
- Every ERPNext service that uses the custom image resolves to the intended Gitea reference, including backend, frontend, websocket, scheduler, workers and the configurator where applicable.
- MariaDB and Redis image versions, service endpoints and health checks match the release plan. Use the same persistent MariaDB, Redis and sites volumes and verify any bind mount paths. Rendered relative paths may have been resolved against the upstream source checkout.
- The existing Compose project and external network names, Traefik labels, host routing for `erp.prothmann.com`, ports and proxy attachment remain correct.
- The local override supplies the effective production DB settings and no `COMPOSE_TEMPLATE_ONLY` remains in the merged result. Check existing site DB credentials and encryption keys are preserved.
After the review:
```bash
mv -- "$candidate" docker-compose.yml
erp_dc config --quiet
erp_dc config --images
docker login git.prothmann.com
erp_dc pull
```
Use a Gitea credential authorized to pull the private package. If pull fails, stop before starting the update. Inspect `docker image inspect FULL_NEW_IMAGE_REFERENCE` using the actual Gitea reference and verify its repository digest against the push record. Recheck MariaDB and Redis digests if their tags are mutable. Keep the old image available.
## 19. Apply, migrate and verify
In the maintenance window, apply the validated stack using the images just pulled and checked. The explicit pull policy prevents another registry fetch during startup.
```bash
erp_dc up -d --pull never
erp_dc ps -a
```
Confirm backend readiness and check database/Redis health. An exited configurator can be expected if it completed successfully. Check its exit code and logs. Keep production writes blocked while migration runs.
```bash
erp_dc exec backend bench --site erp.prothmann.com migrate
```
If migration fails, keep maintenance active and diagnose before proceeding. Capture the failing patch or fixture and relevant logs. A failed migration can already have changed schema or data. Avoid skipping failing patches as part of routine recovery.
```bash
erp_dc exec backend bench --site erp.prothmann.com clear-cache
# When website cache is relevant:
erp_dc exec backend bench --site erp.prothmann.com clear-website-cache
erp_dc exec backend bench --site erp.prothmann.com list-apps
erp_dc ps -a
erp_dc logs --since=15m --tail=100
```
Confirm running image identities against the release record and check for DB, Redis, permission, asset, fixture, websocket and queue errors. Use `erp_dc config --services` to find actual service names and narrow logs as needed. Confirm scheduler and worker processes are up. Restore maintenance mode and scheduler pause to their recorded settings after migration and infrastructure checks pass. For a site that was originally live with an unpaused scheduler:
```bash
erp_dc exec backend bench --site erp.prothmann.com set-maintenance-mode off
erp_dc exec backend bench --site erp.prothmann.com scheduler resume
erp_dc exec backend bench --site erp.prothmann.com scheduler status
erp_dc exec backend bench --site erp.prothmann.com doctor
```
If scheduling is unexpectedly disabled, diagnose its settings. Use `bench --site erp.prothmann.com enable-scheduler` inside the backend only when the site is intended to schedule jobs and its scheduler is disabled. Verify a known safe background job completes and the expected workers are online. If traffic was restricted externally, reopen it once smoke tests pass.
### Acceptance checklist
- [ ] HTTPS, login, Desk, static assets and realtime features work through Traefik.
- [ ] Manufacturing Utilities workspace and Manufacturing Overview load.
- [ ] PRO / Work Order and Sales Order links work. Planned End Date sorting is correct.
- [ ] Subcontracting cards, draft Purchase Orders and submitted subcontracting orders appear.
- [ ] German translations appear after selecting German and hard-refreshing. Check a newly translated Manufacturing Overview label.
- [ ] Label Composer opens, preview renders, and an authorized test label prints on the network printer.
- [ ] E-invoice functionality opens and passes a non-sending smoke test.
- [ ] Scheduler is in its intended state, workers process jobs, and logs show no new recurring errors.
- [ ] Release record has the deployed digest, upstream Compose revision, backup location, and verification result.
## 20. Rollback and recovery
Keep writes blocked while deciding how to recover. Capture service status, logs and the failing release details. If the failure happened before any schema, data or infrastructure change and the old image is compatible with current state, restore the saved **pair** of Compose files and saved `.env` from the same release. Include any other changed deployment files and preserve file permissions. Verify the previous config and image digests before starting it.
```bash
cd /srv/docker/erpnext-16
# Set RECOVERY_DIR to the verified pre-release snapshot if reconnecting.
cp -p "${RECOVERY_DIR:?Set the recovery directory}/.env" \
"$RECOVERY_DIR/docker-compose.yml" \
"$RECOVERY_DIR/docker-compose.override.yml" .
erp_dc config --quiet
erp_dc config --images
# Pull only if needed, and verify the previous digest before applying.
erp_dc up -d --pull never
erp_dc ps -a
```
A prior image cannot undo `migrate`, app uninstall, data patches, or MariaDB on-disk upgrades. If the old image is incompatible with the modified state, preserve a copy of the failed state and establish what writes occurred since the backup. Restore the verified pre-update SQL and both file archives together with the matching site configuration/encryption key, using the old compatible application and database versions. Confirm the exact `bench --site erp.prothmann.com restore --help` on that release and rehearse with the saved files on a separate recovery target before overwriting production. For an incompatible MariaDB storage upgrade, recover onto compatible storage rather than starting an older server on the upgraded data directory. Recheck queues and pending integrations before resuming them. Run the acceptance checklist after recovery.
**Never run `docker compose down -v` during this workflow.** Preserve DB, Redis and sites volumes, public and private files, DB credentials, encryption keys, and Traefik networks. Keep the previous image and recovery artifacts until the release has passed verification.
## Release checklist
1. Export fixtures when changed, refresh Gettext catalogs, translate, compile, test and push the source.
2. Build on the Mac for `linux/amd64`, verify apps and printing dependencies, push the unique Gitea tag and record its digest.
3. On hyp1 identify the current project, block writes, drain jobs and verify the backup and rollback snapshot.
4. Update the official source checkout, select the new tag, render the base with `COMPOSE_TEMPLATE_ONLY`, then validate it with the production override.
5. Install the validated base, pull, compare digests, run `up -d`, migrate and clear caches.
6. Verify scheduler, workers and application behavior. Restore normal traffic and record the outcome.
## References
- [Frappe v16 Gettext command definitions](https://github.com/frappe/frappe/blob/version-16/frappe/commands/gettext.py)
- [Frappe v16 scheduler and worker commands](https://github.com/frappe/frappe/blob/version-16/frappe/commands/scheduler.py)
- [Official frappe_docker Compose overrides](https://github.com/frappe/frappe_docker/blob/main/docs/02-setup/05-overrides.md)
- [Docker Compose config reference](https://docs.docker.com/reference/cli/docker/compose/config/)
**Rollback.** Restore the previous `CUSTOM_TAG` and repeat `docker compose pull` and `docker compose up -d`. If migrations made incompatible database changes, restore the pre-deployment backup as well.