# Development WorkflowS

## GitLab and GitHub

All the development process is happening inside GitLab server https://git.bgerp.org There we do branches for perform changes, run automatic tests and publish updates.

To the external GitHub https://github.com/Pingvin235/bgerp are published only state of master branch after change’s acceptance. The repository is enough for exploration purposes and Custom development.

## Changes

Each change is performed in a separated GIT branch. For development of the project we use Standard GIT Workflow. Similar one is used, for example, for GNU/Linux development.

The main idea is that merging is only happening 'downstream', i.e. from the main branch to secondary branches, in order to get to the actual state. Main branch has linear structure as all changes are incorporated via 'patches'. Secondary branches can be then deleted, as all the aggregated information from secondary branches will be present in the main branch.

master is a main project branch and is used for builds.

- Each change should be related to a BGERP process with PROCESS_ID, used for all information exchange for a given change.

- For each change create a separate GIT branch based on master. Branch’s name should start from p<PROCESS_ID>, then use "-" as delimiters. Example: p11788-link-filter-title

- When developing, you can use any commit strategy within a branch: intermediate commits, reverts and resets. We would recommend to push intermediate commits, using GIT repository as a backup copy.

- Main branch can be periodically merged to the change branch in order to synchronize with the actual state.

- Branch has to add build/change.<PROCESS_ID>.txt file, create it by running `./gradlew touchChanges`. This file has to describe new functions, fixes or other changes - each item on a separate row. Format is identical to changes.txt, which accumulates all build/change.<PROCESS_ID>.txt information when release update gets published.

- All the necessary documentation changes are done in the branch after the code changes.

- Request branch acceptance to master once development and testing completes for a given change by moving BGERP process into Acceptance status.

- Process gets closed after check/acceptance, and change gets merged to master as a single commit with a comment starting with the change ID.

### GIT Commands

| Be sure that your local GIT repo is properly configured.

All the following GIT commands have to be executed in the project’s directory.

#### Creation a change branch

    git checkout master && git pull origin master && git checkout -b p12345-my-change-branch

#### Commit and push changes

    git checkout p12345-my-change-branch && git add . && git commit -am "My changes" && git push

At the first push for a branch you will be asked to set an upstream branch using the command.

    git push --set-upstream origin p12345-my-change-branch

#### Show current workdirs' state

For showing the current state of GIT working directory use:

    git status

Displays text-rendered GIT graph:

    git log --oneline --decorate --all --graph

#### Update a change branch

    git checkout p12345-my-change-branch && git pull --rebase && git fetch origin master:master && git merge master

#### Clean Branches

| Be sure, that all needed change branches are pushed

Stale branches should be deleted first on GitLab: https://git.bgerp.org/bgerp/bgerp/-/branches

Afterwards, run the script for any local repository:

    bash git-clean-branches.sh

#### Troubleshooting

Error: Encountered 7 file(s) that should have been pointers, but weren’t

Fixing commands:

    git rm --cached -r . && git reset --hard

Source

### Acceptance

When change is being accepted, 'change' branch is compared with the master. So merge the latest state of master to the 'change' branch before passing a change to acceptance.

GIT commands to be used for merging change from pXXXXX-short-change-description branch to master:

    git checkout pXXXXX-short-change-description && git pull
    git commit --allow-empty -m "MERGED" && git push
    git checkout master && git pull
    git merge --squash pXXXXX-short-change-description

Commit with current GIT user.

    git commit -am "pXXXXX Some change description."

Or for preserving the author in GitHub. Mapping internal to external mails is available in file `.gitlab-ci-publish-author.sh`

    git commit -am "pXXXXX Some change description." --author="Developer Name <internal@gitlab.email>"

And finally, push.

    git push

| Perform a separate GIT Push after each acceptance in order to correctly publish in the open repository.

After each acceptance to the master branch CI performs test-and-publish-change-master and test-integration-master jobs, which together prepare data for running Demo System.

### Long-lived branches

Long-lived branches only get changes which are not altering product functionality, for example: localization and documentation fixes, code formatting, tests. BGERP process for a change does not get closed , and acceptance is performed multiple times upon completion of specific change(s).

### Documentation

For building of the manual used Open Source tool PzdcDoc. The source AsciiDoctor and resource files are placed in `srcx/doc` directory. The examples of documentation’s format may be found here.

#### Plugin

Plugin documentation should have the following structure:

- About - common information about;

- Setup - how to configure;

- Usage - using instructions with screenshots;

- Development - info for developers.

As examples of following the structure see plugins: GIT, Backup.

#### Images

Images like screenshots in documentation must be stored in PNG format with .png extension. This extension is configured to be stored in GIT LFS. Use width attribute for limiting size rather than resizing that can sometimes increase a file size.

#### Change

Documentation changes are recommended to be done at the end of branch development, using changes file as preliminary notes. CI job test-and-publish-change builds documentation automatically for each commit or it might be built locally.

Due the strict references and snippets checking, it is quite possible to have broken state of documentation even without changes in `.adoc` files. Something like the following:

    2020-05-25 12:17:39,149 INFO DocGenerator [main] Processing: srcx/doc/project.adoc
    2020-05-25 12:17:39,844 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't start from: 'message.se', line number: 205, content: newProcess.setDescription(message.getSubject());
    2020-05-25 12:17:39,855 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't end on: ');', line number: 71, content: if (message == null)
    2020-05-25 12:17:39,859 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't start from: 'pu', line number: 241, content:
    2020-05-25 12:17:39,860 ERROR Snippet [main] Snippet '../../src/ru/bgcrm/struts/action/MessageAction.java' doesn't end on: '}', line number: 253, content:
    2020-05-25 12:17:39,911 ERROR Snippet [main] Snippet '../../webapps/WEB-INF/jspf/user/search/search.jsp' doesn't start from: '<div', line number: 1, content: <%@ page contentType="text/html; charset=UTF-8"%>
    2020-05-25 12:17:39,911 ERROR Snippet [main] Snippet '../../webapps/WEB-INF/jspf/user/search/search.jsp' doesn't end on: '/div>', line number: 134, content: <%@ include file="/WEB-INF/jspf/shell_title.jsp"%>
    2

For such cases here is the fixing algorithm. First, find the failing line in `.adoc` file:

After that, using branch comparison, find the new rows and change them in the `.adoc`:

Hints:

- If documentation was already corrected in the current branch, you can create a mock branch on the last working state.

- Use line numbers for searching over failing snippets.

Place human readable changes description with screens (possible link images from a main article) in 0 changes file.

| You can postpone the complete work and only store screenshots to the `_res` directory there.

#### Release

Publication of the product manual runs automatically from long-lived branch p11862-documentation. In the same branch may be made changes for documentation only of the latest release that also will be published.

## GitLab CI

.gitlab-ci.yml file has configuration for running certain jobs automatically upon each GIT commit. Different jobs are executed within different GIT branches.

### Jobs

Please find below description of CI jobs.

#### test-and-publish-change

The job is executed in every change branch.

Launches Unit tests and documentation build and validity check. If everything is went without error then publishing update with documentation from the branch to https://bgerp.org/change/<PROCESS_ID>;

| All the published change updates have the same version, equal to the next release.

#### test-and-publish-change-master

The job is executed in master branch.

Does the same as test-and-publish-change but for master branch. As there is no real PROCESS_ID available, it is taken as 0 published update. The artificial change 0 can be used for updating to the latest state of master. To this state is periodically reset Demo System.

#### test-integration

Runs integration test with DB in container.

#### test-integration-master

The job is executed in master branch.

Runs integration test. Upon successful tests execution, written in DB dump gets extracted for Demo System.

#### publish-source

The job is executed in master branch.

Publishes actual source code from `master` into an open repository https://github.com/pingvin235/bgerp

#### publish-doc

The job is executed in long-lived branch p11862-documentation.

The branch contains the documentation sources for the latest release. Launches documentation build and validity check, if no errors are found, publishes on https://bgerp.org/version/3.0/doc.

### Runners

Runners are responsible for executing jobs. The project already has some runners available, but you can register additional ones to speed up process.

The list of runners is available here: https://git.bgerp.org/admin/runners There also can be taken registration token.

A runner can be also added are here: https://git.bgerp.org/bgerp/bgerp/-/settings/ci_cd under Runners section.

That can be done on every system with installed Docker.

    # delete existing runner if exists
    docker pull gitlab/gitlab-runner:latest && docker stop gitlab-runner && docker rm gitlab-runner
    # create and start runner container
    docker run -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock gitlab/gitlab-runner:latest
    # register runner, use executor: 'docker', URL: https://git.bgerp.org/
    docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

## Build & Publish

Project is configured in Gradle format (configuration file: build.gradle)

For building and publishing, apart from Java you will need console environment with available scp, ssh and rsync commands.

| Here and below all commands are shown for WSL environment, *NIX will not need `bash -c` prefix.

### Documentation

    bash -c "./gradlew clean doc"

Resulting HTML files will be present in target/doc. Internal link validation is performed automatically.

### Change

| This task is automatically run by CI.

Build and publish on https://bgerp.org/change update package with a change.

All the updates packages are copies to Web directory: `https://bgerp.org/change/PROCESS_ID` The changes file has also copied, and all documentation links there starting from `https://bgerp.org/change/0/doc` are automatically replaced to the `https://bgerp.org/change/PROCESS_ID/doc`.

Users have a capability to update to the change using a PROCESS_ID as an identifier.

Multiple update publications are possible until all bugs/caveats are found - after that that change accepted into a main branch and a new build gets published

### Release

| Public SSH key or the developer might be added for cdn@cdn.bgerp.org and www@bgerp.org. Before you publish a change update, make sure that `ssh cdn@cdn.bgerp.org` and `ssh www@bgerp.org` sessions work for you. Script `files.sh` uses rename command, it must be installed. When pushing Docker images from a system first you would need to perform login:
    docker login --username bgerp
And input an Access Token for Docker Hub.

Build is performed from a master branch and can include many accepted changes.

| Be sure that Integration tests are successfully done on master to provide data for Demo System.

| Stop all running IDEs with opened project directory.

Checkout the master branch and merge there possible latest documentation changes:

    git checkout master && git merge --squash p15055-documentation-master && git commit -am "p15055 Documentation Master" && git push

Reset the properties files:

    ./gradlew resetProperties

Set the new stable release version YYMM in build.number property of `build/update/update.properties` file.

Perform the following:

    ./gradlew clean updateLibProperties buildUpdateLib updateProperties buildUpdate buildDist

| buildUpdateLib and updateLibProperties tasks check existence of file `build/changes.lib.txt`, marking changes in Java libraries.

Build the bgerp/base Docker Image, which should be rebuilt in case of updating MariaDB or Java versions in it:

    bash build/docker/base/files.sh && docker build build/docker/base -t bgerp/base

Build the bgerp/bgerp Docker image:

    bash build/docker/bgerp/files.sh && docker build build/docker/bgerp -t bgerp/bgerp

Test the created image with in local Docker:

    docker run --rm -it -e MASTER=no -p 9088:9088 --name bgerp bgerp/bgerp

Build the bgerp/devd Docker image:

    docker build build/docker/devd -t bgerp/devd

Then apply file changes and commit them:

    ./gradlew changesFile changesRss changesDoc publishCommit

Copy the built artifacts to the server and check the results on https://bgerp.org and https://hub.docker.com/r/bgerp/bgerp/tags

    bash -c "./gradlew publishRelease"
    docker push bgerp/base && docker push bgerp/bgerp && docker push bgerp/devd

Check the release commit and push it:

    git push

Replace <YYMM> to the release number and create stable code and documentation branches:

    git checkout -b stable-<YYMM> && git push --set-upstream origin stable-<YYMM>
    git checkout p11862-documentation && git checkout -b p11862-documentation-<YYMM> && git push --set-upstream origin p11862-documentation-<YYMM>

Open https://git.bgerp.org/bgerp/bgerp/-/settings/repository#branch-rules and allow to push force branches p11862-documentation and p15055-documentation-master Reset the documentation branches:

    git checkout p11862-documentation && git reset --hard master && git push --force
    git checkout p15055-documentation-master && git reset --hard master && git push --force

Disallow to push force branches p11862-documentation and p15055-documentation-master

Add news to Web sites about release with a link to the release changes.

### Release Fix

| Stop all running IDEs with opened project directory.

    ./gradlew publishReleaseFix && git push

## Tests

### Unit

JUnit framework is used (`srcx/test` directory) Used to test specific algorithms, test do not depend on each other and do not work with DB.

Test are launched locally using the following command:

    bash -c "./gradlew clean test"

### Integration

TestNG framework is used (`srcx/itest` directory)

Integration test performs initialization an empty DB and filling after it with configuration. Tests form a dependencies graph which defined order and execution parallelism.

    ./gradlew itest

By default the tests uses MariaDB instance with credentials taken from `bgerp.properties` file. For local run use Docker DB Instance for that.