GitLab and GitHub

All the development process is happening inside GitLab server https://git.bgerp.org There we do branches for perform changes, run automatic tests and publish updates.

To the external GitHub https://github.com/Pingvin235/bgerp are published only state of master branch after change’s acceptance. The repository is enough for exploration purposes and Custom development.

Changes

Each change is performed in a separated GIT branch. For development of the project we use Standard GIT Workflow. Similar one is used, for example, for GNU/Linux development.

The main idea is that merging is only happening 'downstream', i.e. from the main branch to secondary branches, in order to get to the actual state. Main branch has linear structure as all changes are incorporated via 'patches'. Secondary branches can be then deleted, as all the aggregated information from secondary branches will be present in the main branch.

master is a main project branch and is used for builds.

  • Each change should be related to a BGERP process with PROCESS_ID, used for all information exchange for a given change.

  • For each change create a separate GIT branch based on master. Branch’s name should start from p<PROCESS_ID>, then use "-" as delimiters. Example: p11788-link-filter-title

  • When developing, you can use any commit strategy within a branch: intermediate commits, reverts and resets. We would recommend to push intermediate commits, using GIT repository as a backup copy.

  • Main branch can be periodically merged to the change branch in order to synchronize with the actual state.

  • Branch has to add build/change.<PROCESS_ID>.txt file, create it by running ./gradlew touchChanges. This file has to describe new functions, fixes or other changes - each item on a separate row. Format is identical to changes.txt, which accumulates all build/change.<PROCESS_ID>.txt information when release update gets published.

  • All the necessary documentation changes are done in the branch after the code changes.

  • Request branch acceptance to master once development and testing completes for a given change by moving BGERP process into Acceptance status.

  • Process gets closed after check/acceptance, and change gets merged to master as a single commit with a comment starting with the change ID.

GIT Commands

Be sure that your local GIT repo is properly configured.

All the following GIT commands have to be executed in the project’s directory.

Creation a change branch

git checkout master && git pull origin master && git checkout -b p12345-my-change-branch

Commit and push changes

git checkout p12345-my-change-branch && git add . && git commit -am "My changes" && git push

At the first push for a branch you will be asked to set an upstream branch using the command.

git push --set-upstream origin p12345-my-change-branch

Show current workdirs' state

For showing the current state of GIT working directory use:

git status

Displays text-rendered GIT graph:

git log --oneline --decorate --all --graph

Update a change branch

git checkout p12345-my-change-branch && git pull --rebase && git fetch origin master:master && git merge master

Clean all links to non-existent/deleted branches

git remote prune origin

Acceptance

When change is being accepted, 'change' branch is compared with the master. So merge the latest state of master to the 'change' branch before passing a change to acceptance.

branch compare

GIT commands to be used for merging change from pXXXXX-short-change-description branch to master:

git checkout pXXXXX-short-change-description && git pull
git commit --allow-empty -m "MERGED" && git push
git checkout master && git pull
git merge --squash pXXXXX-short-change-description

Commit with current GIT user.

git commit -am "pXXXXX Some change description."

Or for preserving the author in GitHub. Mapping internal to external mails is available in file .gitlab-ci-publish-author.sh

git commit -am "pXXXXX Some change description." --author="Developer Name <internal@gitlab.email>"

And finally, push.

git push
Perform a separate GIT Push after each acceptance in order to correctly publish in the open repository.

Release is published periodically from the master branch.

Long-lived branches

Long-lived branches only get changes which are not altering product functionality, for example: localization and documentation fixes, code formatting, tests. BGERP process for a change does not get closed , and acceptance is performed multiple times upon completion of specific change(s).

Documentation

For building of the manual used Open Source tool PzdcDoc. The source AsciiDoctor and resource files are placed in srcx/doc directory. The samples of documentation’s format may be found here.

Change

Documentation changes are recommended to be done at the end of branch development, using changes file as preliminary notes. CI job test-and-publish-update builds documentation automatically for each commit or it might be built locally.

Due the strict references and snippets checking, it is quite possible to have broken state of documentation even without changes in .adoc files. Something like the following:

2020-05-25 12:17:39,149 INFO DocGenerator [main] Processing: srcx/doc/project.adoc
2020-05-25 12:17:39,844 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't start from: 'message.se', line number: 205, content: newProcess.setDescription(message.getSubject());
2020-05-25 12:17:39,855 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't end on: ');', line number: 71, content: if (message == null)
2020-05-25 12:17:39,859 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't start from: 'pu', line number: 241, content:
2020-05-25 12:17:39,860 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't end on: '}', line number: 253, content:
2020-05-25 12:17:39,911 ERROR Snippet [main] Snippet '../../webapps/WEB-INF/jspf/user/search/search.jsp' doesn't start from: '<div', line number: 1, content: <%@ page contentType="text/html; charset=UTF-8"%>
2020-05-25 12:17:39,911 ERROR Snippet [main] Snippet '../../webapps/WEB-INF/jspf/user/search/search.jsp' doesn't end on: '/div>', line number: 134, content: <%@ include file="/WEB-INF/jspf/shell_title.jsp"%>
2

For such cases here is the fixing algorithm. First, find the failing line in .adoc file:

snippet fix 1

After that, using branch comparison, find the new rows and change them in the .adoc:

snippet fix 2

Hints:

  • If documentation was already corrected in the current branch, you can create a mock branch on the last working state.

  • Use line numbers for searching over failing snippets.

Release

Publication of the product manual runs automatically from long-lived branch p11862-documentation. In the same branch may be made changes for documentation only of the latest release that also will be published.

GitLab CI

.gitlab-ci.yml file has configuration for running certain jobs automatically upon each GIT commit. Different jobs are executed within different GIT branches. Please find below description of CI jobs.

test-and-publish-update

The job is executed in every change branch.

All the published change updates have the same version, equal to the next release.

test-and-publish-update-master

The job is executed in master branch.

Does the same as test-and-publish-update but for master branch. As there is no real PROCESS_ID available, it is taken as 00000 published update. The artificial change 00000 can be used for updating to the latest state of master. To this state is periodically reset https://demo.bgerp.org instance.

test-integration

The job is executed in master branch.

Runs integration test. Upon successful tests execution, written in DB dump gets extracted for https://demo.bgerp.org

publish-source

The job is executed in master branch.

Publishes actual source code from master into an open repository https://github.com/pingvin235/bgerp

publish-doc

The job is executed in long-lived branch p11862-documentation.

The branch contains the documentation sources for the latest release. Launches documentation build and validity check, if no errors are found, publishes on https://bgerp.org/doc/3.0/manual.

Build and publish

Project is configured in Gradle format (configuration file: build.gradle)

For building and publishing, apart from Java you will need console environment with available ant, ssh and rsync packages.

Here and below all commands are shown for WSL environment, *NIX will not need bash -c prefix.

Documentation

bash -c "./gradlew clean doc"

Resulting HTML files will be present in target/doc. Internal link validation is performed automatically.

Change update

This task is automatically run by CI.

Build and publish on https://bgerp.org/update update package with a change.

All the updates packages are copies to Web directory: https://bgerp.org/update/PROCESS_ID The changes file has also copied, and all documentation links there starting from https://bgerp.org/doc are automatically replaced to the https://bgerp.org/update/PROCESS_ID/doc.

Users have a capability to update to the change using a PROCESS_ID as an identifier.

Multiple update publications are possible until all bugs/caveats are found - after that that change accepted into a main branch and a new build gets published

Release update

Public SSH key or the developer might be added for cdn@bgerp.org and www@bgerp.org. Before you publish a change update, make sure that ssh cdn@bgerp.org and ssh www@bgerp.org sessions work for you.

Build is performed from a master branch and can include many accepted changes.

Be sure that Integration tests are successfully done on master to provide data for Demo DB.

Perform the following:

bash -c "./gradlew clean buildClean buildUpdateLib buildUpdate"
buildUpdateLib task checks existence of file build/changes.lib.txt, marking changes in Java libraries.

Then:

bash -c "./gradlew patchChanges rss publishBuild publishCommit"

Check the release commit and make:

git push

Merge the latest state of the master on documentation branch:

git checkout p11862-documentation && git pull --rebase && git merge master && git push

Docker Image

docker login --username bgerp

Input access token.

Go in directory build/docker

bash files.sh &&
docker build . -t bgerp/bgerp &&
docker push bgerp/bgerp

Tests

Unit

JUnit framework is used (`srcx/test' directory) Used to test specific algorithms, test do not depend on each other and do not work with DB.

Test are launched locally using the following command:

bash -c "./gradlew clean test"

Integration

TestNG framework is used (`srcx/itest' directory)

Integration test performs initialization an empty DB and filling after it with configuration. Tests form a dependencies graph which defined order and execution parallelism.

./gradlew itest

By default the tests uses MySQL instance with credentials taken from bgerp.properties file. For local run use Docker DB Instance for that.

Optimization

Additionally to the recommended MySQL configuration, check and set in my.ini:

innodb_file_per_table=0

That will significantly increase table creation speed. After the first successful run, the structure of a DB will remain persistent and re-created faster by this way.

Creation of dump for Windows:

echo DROP DATABASE IF EXISTS bgerp; > ./dump.sql &&
echo CREATE DATABASE bgerp DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci; >> ./dump.sql &&
echo USE bgerp; >> ./dump.sql &&
mysqldump -uDB_USER -pDB_PSWD bgerp --add-drop-database --no-data >> ./dump.sql &&
type build\bgerp\db_init_end.sql >> ./dump.sql

For *NIX:

echo "DROP DATABASE IF EXISTS bgerp;" > ./dump.sql &&
echo "CREATE DATABASE bgerp DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;" >> ./dump.sql &&
echo "USE bgerp;" >> ./dump.sql &&
mysqldump -uDB_USER -pDB_PSWD bgerp --add-drop-database --no-data >> ./dump.sql &&
cat build/bgerp/db_init_end.sql >> ./dump.sql

Running the tests after:

mysql -uDB_USER -pDB_PSWD < ./dump.sql &&
gradlew integrationTest -Pdb.user=DB_USER -Pdb.pswd=DB_PSWD -Pskip.dbReset=true