GitLab and GitHub

All the development process is happening inside GitLab server https://git.bgerp.org There we do branches for perform changes, run automatic tests and publish updates.

To the external GitHub https://github.com/Pingvin235/bgerp are published only state of master branch after change’s acceptance. The repository is enough for exploration purposes and Custom development.

Changes

Each change is performed in a separated GIT branch. For development of the project we use Standard GIT Workflow. Similar one is used, for example, for GNU/Linux development.

The main idea is that merging is only happening 'downstream', i.e. from the main branch to secondary branches, in order to get to the actual state. Main branch has linear structure as all changes are incorporated via 'patches'. Secondary branches can be then deleted, as all the aggregated information from secondary branches will be present in the main branch.

master is a main project branch and is used for builds.

  • Each change should be related to a BGERP process with PROCESS_ID, used for all information exchange for a given change.

  • For each change create a separate GIT branch based on master. Branch’s name should start from p<PROCESS_ID>, then use "-" as delimiters. Example: p11788-link-filter-title

  • When developing, you can use any commit strategy within a branch: intermediate commits, reverts and resets. We would recommend to push intermediate commits, using GIT repository as a backup copy.

  • Main branch can be periodically merged to the change branch in order to synchronize with the actual state.

  • Branch has to add build/change.<PROCESS_ID>.txt file, create it by running ./gradlew touchChanges. This file has to describe new functions, fixes or other changes - each item on a separate row. Format is identical to changes.txt, which accumulates all build/change.<PROCESS_ID>.txt information when release update gets published.

  • All the necessary documentation changes are done in the branch after the code changes.

  • Request branch acceptance to master once development and testing completes for a given change by moving BGERP process into Acceptance status.

  • Process gets closed after check/acceptance, and change gets merged to master as a single commit with a comment starting with the change ID.

GIT Commands

Be sure that your local GIT repo is properly configured.

All the following GIT commands have to be executed in the project’s directory.

Creation a change branch

git checkout master && git pull origin master && git checkout -b p12345-my-change-branch

Commit and push changes

git checkout p12345-my-change-branch && git add . && git commit -am "My changes" && git push

At the first push for a branch you will be asked to set an upstream branch using the command.

git push --set-upstream origin p12345-my-change-branch

Show current workdirs' state

For showing the current state of GIT working directory use:

git status

Displays text-rendered GIT graph:

git log --oneline --decorate --all --graph

Update a change branch

git checkout p12345-my-change-branch && git pull --rebase && git fetch origin master:master && git merge master

Cleanup

Clean all links to non-existent/deleted branches.

git remote prune origin

Clean GIT repo from unreferenced objects after.

git reflog expire --expire-unreachable=now --all &&
git gc --prune=now

Clean unneeded GIT LFS files.

git lfs prune

Troubleshooting

Error: Encountered 7 file(s) that should have been pointers, but weren’t

Fixing commands:

git rm --cached -r . && git reset --hard

Source

Acceptance

When change is being accepted, 'change' branch is compared with the master. So merge the latest state of master to the 'change' branch before passing a change to acceptance.

branch compare

GIT commands to be used for merging change from pXXXXX-short-change-description branch to master:

git checkout pXXXXX-short-change-description && git pull
git commit --allow-empty -m "MERGED" && git push
git checkout master && git pull
git merge --squash pXXXXX-short-change-description

Commit with current GIT user.

git commit -am "pXXXXX Some change description."

Or for preserving the author in GitHub. Mapping internal to external mails is available in file .gitlab-ci-publish-author.sh

git commit -am "pXXXXX Some change description." --author="Developer Name <internal@gitlab.email>"

And finally, push.

git push
Perform a separate GIT Push after each acceptance in order to correctly publish in the open repository.

After each acceptance to the master branch CI performs test-and-publish-change-master and test-integration-master jobs, which together prepare data for running Demo System.

Long-lived branches

Long-lived branches only get changes which are not altering product functionality, for example: localization and documentation fixes, code formatting, tests. BGERP process for a change does not get closed , and acceptance is performed multiple times upon completion of specific change(s).

Documentation

For building of the manual used Open Source tool PzdcDoc. The source AsciiDoctor and resource files are placed in srcx/doc directory. The examples of documentation’s format may be found here.

Plugin

Plugin documentation should have the following structure:

  • About - common information about;

  • Setup - how to configure;

  • Usage - using instructions with screenshots;

  • Development - info for developers.

As examples of following the structure see plugins: GIT, Backup.

Images

Images like screenshots in documentation must be stored in PNG format with .png extension. This extension is configured to be stored in GIT LFS. Use width attribute for limiting size rather than resizing that can sometimes increase a file size.

Change

Documentation changes are recommended to be done at the end of branch development, using changes file as preliminary notes. CI job test-and-publish-change builds documentation automatically for each commit or it might be built locally.

Due the strict references and snippets checking, it is quite possible to have broken state of documentation even without changes in .adoc files. Something like the following:

2020-05-25 12:17:39,149 INFO DocGenerator [main] Processing: srcx/doc/project.adoc
2020-05-25 12:17:39,844 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't start from: 'message.se', line number: 205, content: newProcess.setDescription(message.getSubject());
2020-05-25 12:17:39,855 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't end on: ');', line number: 71, content: if (message == null)
2020-05-25 12:17:39,859 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't start from: 'pu', line number: 241, content:
2020-05-25 12:17:39,860 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't end on: '}', line number: 253, content:
2020-05-25 12:17:39,911 ERROR Snippet [main] Snippet '../../webapps/WEB-INF/jspf/user/search/search.jsp' doesn't start from: '<div', line number: 1, content: <%@ page contentType="text/html; charset=UTF-8"%>
2020-05-25 12:17:39,911 ERROR Snippet [main] Snippet '../../webapps/WEB-INF/jspf/user/search/search.jsp' doesn't end on: '/div>', line number: 134, content: <%@ include file="/WEB-INF/jspf/shell_title.jsp"%>
2

For such cases here is the fixing algorithm. First, find the failing line in .adoc file:

snippet fix 1

After that, using branch comparison, find the new rows and change them in the .adoc:

snippet fix 2

Hints:

  • If documentation was already corrected in the current branch, you can create a mock branch on the last working state.

  • Use line numbers for searching over failing snippets.

Place human readable changes description with screens (possible link images from a main article) in 0 changes file.

You can postpone the complete work and only store screenshots to the _res directory there.

Release

Publication of the product manual runs automatically from long-lived branch p11862-documentation. In the same branch may be made changes for documentation only of the latest release that also will be published.

GitLab CI

.gitlab-ci.yml file has configuration for running certain jobs automatically upon each GIT commit. Different jobs are executed within different GIT branches.

Jobs

Please find below description of CI jobs.

test-and-publish-change

The job is executed in every change branch.

Launches Unit tests and documentation build and validity check. If everything is went without error then publishing update with documentation from the branch to https://bgerp.org/change/<PROCESS_ID>;

All the published change updates have the same version, equal to the next release.

test-and-publish-change-master

The job is executed in master branch.

Does the same as test-and-publish-change but for master branch. As there is no real PROCESS_ID available, it is taken as 0 published update. The artificial change 0 can be used for updating to the latest state of master. To this state is periodically reset Demo System.

test-integration

Runs integration test with DB in container.

test-integration-master

The job is executed in master branch.

Runs integration test. Upon successful tests execution, written in DB dump gets extracted for Demo System.

publish-source

The job is executed in master branch.

Publishes actual source code from master into an open repository https://github.com/pingvin235/bgerp

publish-doc

The job is executed in long-lived branch p11862-documentation.

The branch contains the documentation sources for the latest release. Launches documentation build and validity check, if no errors are found, publishes on https://bgerp.org/version/3.0/doc.

Runners

Runners are responsible for executing jobs. The project already has some runners available, but you can register additional ones to speed up process.

The list of runners is available here: https://git.bgerp.org/admin/runners There also can be taken registration token.

gitlab runner token

A runner can be also added are here: https://git.bgerp.org/bgerp/bgerp/-/settings/ci_cd under Runners section.

That can be done on every system with installed Docker.

# delete existing runner if exists
docker pull gitlab/gitlab-runner:latest && docker stop gitlab-runner && docker rm gitlab-runner
# create and start runner container
docker run -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest
# register runner, use executor: 'docker', URL: https://git.bgerp.org/
docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

Build and publish

Project is configured in Gradle format (configuration file: build.gradle)

For building and publishing, apart from Java you will need console environment with available scp, ssh and rsync commands.

Here and below all commands are shown for WSL environment, *NIX will not need bash -c prefix.

Documentation

bash -c "./gradlew clean doc"

Resulting HTML files will be present in target/doc. Internal link validation is performed automatically.

Change

This task is automatically run by CI.

Build and publish on https://bgerp.org/change update package with a change.

All the updates packages are copies to Web directory: https://bgerp.org/change/PROCESS_ID The changes file has also copied, and all documentation links there starting from https://bgerp.org/version/VERSION_NUMBER/doc are automatically replaced to the https://bgerp.org/change/PROCESS_ID/doc.

Users have a capability to update to the change using a PROCESS_ID as an identifier.

Multiple update publications are possible until all bugs/caveats are found - after that that change accepted into a main branch and a new build gets published

Release

Public SSH key or the developer might be added for cdn@bgerp.org and www@bgerp.org. Before you publish a change update, make sure that ssh cdn@bgerp.org and ssh www@bgerp.org sessions work for you.

Script files.sh uses rename command, it must be installed.

When pushing Docker images from a system first you would need to perform login:

docker login --username bgerp

And input an Access Token for Docker Hub.

Build is performed from a master branch and can include many accepted changes.

Be sure that Integration tests are successfully done on master to provide data for Demo System.
Stop all running IDEs with opened project directory.

Perform the following:

./gradlew clean resetProperties updateLibProperties buildUpdateLib updateProperties buildUpdate buildDist
buildUpdateLib and updateLibProperties tasks check existence of file build/changes.lib.txt, marking changes in Java libraries.

Build the bgerp/base Docker Image, which should be rebuilt in case of updating MySQL or Java versions in it:

bash build/docker/base/files.sh && docker build build/docker/base -t bgerp/base

Build the bgerp/bgerp Docker image:

bash build/docker/bgerp/files.sh && docker build build/docker/bgerp -t bgerp/bgerp

Test the created image with in local Docker:

docker run -d --name bgerp --restart unless-stopped -p 9088:9088 bgerp/bgerp && docker logs --follow bgerp

Build the bgerp/devd Docker image:

docker build build/docker/devd -t bgerp/devd

Then apply file changes and commit them:

./gradlew changesFile changesRss changesDoc publishCommit

Copy the built artifacts to the server and check the results on https://bgerp.org and https://hub.docker.com/r/bgerp/bgerp/tags

bash -c "./gradlew publishRelease"
docker push bgerp/base && docker push bgerp/bgerp && docker push bgerp/devd

Check the release commit and push it:

git push

Merge the latest state of the master on documentation branch:

git checkout p11862-documentation && git pull --rebase && git merge master && git push

Pick documentation changes back to master. Normally there shouldn’t be anyone.

git checkout master && git pull --rebase && git merge --squash p11862-documentation && git commit -am "p11862 Documentation."

Add news to Web sites about release with a link to the release changes.

In case of bugfix releases already existing changes document may be renamed, now news is needed.

Tests

Unit

JUnit framework is used (srcx/test directory) Used to test specific algorithms, test do not depend on each other and do not work with DB.

Test are launched locally using the following command:

bash -c "./gradlew clean test"

Integration

TestNG framework is used (srcx/itest directory)

Integration test performs initialization an empty DB and filling after it with configuration. Tests form a dependencies graph which defined order and execution parallelism.

./gradlew itest

By default the tests uses MySQL instance with credentials taken from bgerp.properties file. For local run use Docker DB Instance for that.

Optimization

Additionally to the recommended MySQL configuration, check and set in my.ini:

innodb_file_per_table=0

That will significantly increase table creation speed. After the first successful run, the structure of a DB will remain persistent and re-created faster by this way.

Creation of dump for Windows:

echo DROP DATABASE IF EXISTS bgerp; > ./dump.sql &&
echo CREATE DATABASE bgerp DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci; >> ./dump.sql &&
echo USE bgerp; >> ./dump.sql &&
mysqldump -uDB_USER -pDB_PSWD bgerp --add-drop-database --no-data >> ./dump.sql &&
type build\bgerp\db_init_end.sql >> ./dump.sql

For *NIX:

echo "DROP DATABASE IF EXISTS bgerp;" > ./dump.sql &&
echo "CREATE DATABASE bgerp DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;" >> ./dump.sql &&
echo "USE bgerp;" >> ./dump.sql &&
mysqldump -uDB_USER -pDB_PSWD bgerp --add-drop-database --no-data >> ./dump.sql &&
cat build/bgerp/db_init_end.sql >> ./dump.sql

Running the tests after:

mysql -uDB_USER -pDB_PSWD < ./dump.sql &&
gradlew integrationTest -Pdb.user=DB_USER -Pdb.pswd=DB_PSWD -Pskip.dbReset=true