//Cloudogu EcoSystem Docs

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

[v3.2.0] - 2021-09-17

Added

  • Mechanism for automated release (#378)

v3.1.0 - 2021-08-17

Added

  • logging initializer which does not use the syslog hook, so it can be used in environments where syslogd is not present

v3.0.0 - 2021-07-20

Breaking Change Ahead!

A breaking change occurred, as we changed the handling of the internal dogu format.

Changed

  • Export createV2Copy function; #369
  • Add exported function to get v1 dogu copy from v2; #371

v2.32.0 - 2021-06-29

Changed

  • Changed the dogu.json format for expressing more complex dependencies. The old format is still accepted by the cesapp. However, this may change in the future #367

v2.31.3 - 2021-06-17

Fixed

  • Handling of docker errors #365

v2.31.2 - 2021-05-26

Fixed

  • Missing log information if the cesapp is used as module (#360)

v2.31.1 - 2021-05-17

Fixed

  • Docker log output: improved readability and avoidance of duplicate output (#362)

v2.31.0 - 2021-04-08

Added

  • A new sub-command to list all defined retention policies and their description (#357)

    • cesapp remove-backup list-policies
    • For more information see list-policies

v2.30.0 - 2021-03-25

Added

  • Stream console output and log lines up to log level "debug" into /var/log/cesapp (#355)

Changed

  • Log lines in /var/log/cesapp receive now also the cesapp's process ID in order to distinguish different cesapp calls (#355)
  • Replace deprecated Docker term library with encouraged Moby library

Fixed

  • These configuration files receive now proper file permissions 0644

    • /etc/rsyslogd/10-cesapp.conf
    • /etc/logrotate.d/cesapp

v2.29.1 - 2021-03-16

Fixed

  • An upgrade via cesapp upgrade <dogu> has been continued, even if the version to be installed has already been installed. The upgrade is now finished directly if the version to be installed is already installed. (#351)
  • During the recursive installation of a dogu (cesapp install --recursive <dogu>), no service accounts could be created because the dependent dogus were only installed but not started. If a service account is to be created, the corresponding dogu is started from now on and waited until it is healthy. After the service account has been created, the started dogu is stopped again. (#353)

v2.29.0 - 2021-03-08

Changed

  • Switch from go-logging to logrus logging framework (#307)

    • Logging to stdout (with specified log level; default: INFO)
    • Logging to /var/log/cesapp.log via syslog(always DEBUG)
  • Update the dependency of restic to version 0.12.0 to fix S3 incompatibilities(with unsupported restic 0.11.0) (#348)

v2.28.0 - 2021-02-11

Changed

  • Upgrade docker lib (to v20.10.2) and everything depending on it
  • If a dogu which is listed in the dogus section of a blueprint is already installed in another namespace, the namespace will not be switched automatically in blueprint-upgrade. (#343) This feature is blocked unless the user provides the --allow-namespace-switch-flag. Example: cesapp upgrade blueprint --allow-namespace-switch blueprint.json
  • Improved the manpage output of the cesapp upgrade command.
  • Refactored the switch-namespace command to use the upgrade functionality (#346)

Added

  • doc files for upgrade functionallity

Fixed

  • Fixed bug where purge was not possible when two equal docker images with different tag exists

v2.27.0 - 2021-01-29

Changed

  • Added the version from which a dogu or package is being upgraded to blueprint output (#340)

Fixed

  • Fix blueprint defect where packages are never upgraded, only installed (#342)

v2.26.0 - 2021-01-27

Added

  • Added command to switch the namespace of a dogu (see docs/switchnamespaceen.md) (#336)
  • Documentation for the cesapp purge command (#339)
  • Documentation for the cesapp switch-namespace command (#336)

Changed

  • Moved DoguNamespaceSwitcher from package cli to package remote to expose it (#338)
  • Updated documentation for the cesapp list-remote command

v2.25.0 - 2021-01-19

Fixed

  • remove accidentally introduced mandatory argument for cesapp list-remote (#344)

Changed

  • the dogu list produced by cesapp list is now sorted alphabetically
  • cesapp list-remote prints now a syntax help if used with wrong parameters (#344)

Removed

  • remove the cesapp list-remote switches --table and --latest, introduced by v2.24.3

v2.24.3 - 2021-01-12

Heads-up:

This released accidentally introduced a mandatory argument for cesapp list-remote which leads to an error when called without this argument. Instead, you may want to install cesapp v2.24.4 (or later) which removes this issue (#344).

Changed

  • Make the print limit output, which added to cesapp for issue #329, optional. This resolves a bug, where it was not possible to update the cesapp when the configuration.json was modified (332)

v2.24.2 - 2021-01-11

Added

  • cesapp listremote as an alias for cesapp list-remote (#329)
  • Configuration option to enable insecure connections to TLS servers. Can be set via the configuration.json of the cesapp (#331):

    "remote": {
    "endpoint": "https://dogu.cloudogu.com/api/v1/",
    "authenticationEndpoint": "https://instance.cloudogu.com",
    "cacheDir": "/tmp/ces/cache/remote",
    "insecure": true
    },

Changed

  • Reworked cesapp list-remote/listremote to show the latest versions for all or a specific dogu (#329)

Specification for cesapp list-remote/listremote:

NAME:
   cesapp listremote - list all available dogu versions for a dogu from remote server

USAGE:
   cesapp listremote [command options] [arguments...]

OPTIONS:
   --available, --all  prints available versions for all dogus (default: false)
   --latest, --table   prints only the latest version. Formatted as table (default: false)
   --limit value       only print out the last x versions. -1 = default limit | 0 = infinite versions (default: -1)
   --help, -h          show help (default: false)

The default limit of versions to show can be configured in the configuration.json:

 "output": {
     "listremote":{
       "defaultVersionPrintLimit": 3
     }
   }

v2.24.1 - 2020-12-18

Fixed

  • Fixes backup errors in conjunction with Google S3 buckets via restic's S3 backend (#327)
  • Downgrade debian dependency to restic 0.10.0

    • if Debian denies a restic downgrade the package downgrade must be done manually: sudo apt-get install restic=0.10.0

v2.24.0 - 2020-12-17

Fixed

  • Useful error message is shown if wrong backup encryption key is provided; #323
  • Detection of uninitialized backup repository improved
  • integration tests use mock command executor now

Changed

  • Upgrade to makefiles 4.3.0

v2.23.0 - 2020-12-16

Changed

  • service accounts are now deleted on dogu purge (#319)
  • logs are now deleted on dogu purge (#319)

Added

  • flag '--keep-service-accounts' which will prevent that service accounts are removed on purge (#319)
  • flag '--keep-logs' which will prevent that logs are removed on purge (#319)

v2.22.0 - 2020-12-15

Added

  • Validator to verify float-based values between 0 - 100%. Validator Descriptor ID: FLOAT_PERCENTAGE_HUNDRED (#321)

v2.21.0 - 2020-11-20

Added

:warning: Possible performance decrease when kernel capabilities are changed! More see below.

This release adds container memory and swap limitation support (#317) by leveraging Docker's resource limitations. If fully configured (see below), Docker will kill containers that require more memory and swap than configured. The respective dogu will then be restarted automatically.

Dogu containers can be restricted in their memory usage by setting dogu configuration keys either by

  • using the edit-config command (only dogus that already implement the respective keys)
  • setting a dogu's registry keys:

    • /config/mydogu/container_config/memory_limit
    • /config/mydogu/container_config/swap_limit
Configuring memory and swap limits

Memory values must be suffixed with the units b (bytes), k (kibibytes), m (mebibytes), or g (gibibytes).

# limit memory to 5,242,880 bytes
etcdctl set /config/mydogu/container_config/memory_limit 5242880b
# limit memory to 5,242,880 bytes but less verbosely
etcdctl set /config/mydogu/container_config/memory_limit 5120k
# limit memory to 5,242,880 bytes but less verbosely
etcdctl set /config/mydogu/container_config/memory_limit 5m
# limit memory to 2,147,483,648 bytes
etcdctl set /config/mydogu/container_config/memory_limit 2g  

Setting a desired swap memory value works like setting the memory limit:

# limit container to swap up to 5,242,880 bytes to the host machine
etcdctl set /config/mydogu/container_config/swap_limit 5m  
The host's swap limit capability

The memory swap limitation strongly depends on the host's kernel capabilities. By default, Ubuntu machines have no memory swap limit capabilities enabled. Disabled swap limiting will allow a container to unlimitedly swap memory pages to the host machine. This might render the desired memory limit as useless because the container will swap memory pages until the swap storage is exhausted.

The host's swap limit capability can be enabled like this:

  1. modify /etc/default/grub as root with an editor of your choice

    1. add to the line GRUB_CMDLINE_LINUX_DEFAULT the following values: cgroup_enable=memory swapaccount=1
  2. apply the changes with sudo update-bootloader –refresh
  3. restart the CES host

Enabling swap limit capabilities will impact the host machines performance, namely:

  • 1 % of memory overhead and
  • 10 % of overall performance degradation Additional information about this behaviour can be found in Docker's documentation.

Changed

  • Refactor container provider creation for better testability

v2.20.0 - 2020-10-05

Changed

  • The original blueprint and the blueprint mask are now also historized (#315)

Fixed

  • Fixed error where the error message was not shown when pull from dogu registry failed

v2.19.0 - 2020-09-23

Added

  • Add blueprint mask capability for cesapp upgrade blueprint command; #312
  • Add end-to-end test capability in Jenkinsfile

v2.18.0 - 2020-09-21

Changed

  • The command cesapp login now also executes a login to docker-registry

v2.17.0 - 2020-09-09

Added

  • Add functionality to remove etcd keys via blueprint upgrade (#306) You can do this by adding the keys to the "registryConfigAbsent"-Array. Top-level entries like "_global" will be skipped for deletion. Example:

    "registryConfigAbsent": [
      "_global/key1",
      "dogu/key2"
    ]

Fixed

  • When using cesapp recreate for a single dogu, the container is now stopped properly (#309)

v2.16.0 - 2020-09-01

Added

  • add a new cesapp command for re-creating dogu containers (#304)

Recreating dogu containers comes in handy when the system entered a defective state (f. ex. due to manual container removal or resource exhaustion). Another benefit is that containers receive fresh container settings. You can create a new container for a single dogu container or for all installed dogus, just like so:

cesapp recreate jenkins
cesapp recreate --all

Changed

  • supply another way to provide extra Docker hosts (#304)
  • always recreate dogu container during Blueprint Upgrade (#304)

The new recreate command leverages a slightly changed functionality: supplying a container with extra hosts (which corresponds with Docker --add-host option).

Please note that there is already a way of providing extra hosts, namely by adding entries cesapp's configuration.json. Anyhow, for restricted systems it is now possible to add extra hosts by assigning an IP address to a hostname within the registry. The following example would add an extra host once a container is recreated:

etcdctl set /config/_global/containers/additional_hosts/myhostname 172.123.45.87

The two sources (via configuration.json and registry) are joined into a single list. Please note that "overwriting" hosts from one source with a host from another source is not possible and will lead to an error.

This comes also into account when the administrator executes a Blueprint Upgrade. The Blueprint mechanism now automatically recreates containers of already installed dogus. Any additional extra hosts (even if supplied with a blueprint JSON) are automatically supplied to the new container instance.

v2.15.0 - 2020-08-21

Changed

  • Reworked mechanism to set and remove the critical process key (#222, #299)
  • The critical process key is now removed by the etcd after a timeout
  • Updated cli-library
  • Updated etcd-version used in jenkinsfile
  • increase timeout for backend requests (5s -> 10s)

v2.14.1 - 2020-08-19

Fixed

  • remove dogus from versions file format string for index url schema (#300)
  • this fix corresponds with ces-mirror > 1.5.1

v2.14.0 - 2020-07-14

Fixed

  • Integration tests
  • allow skipping optional values when using the edit-config command (#297)

Changed

  • Prevent building of dogus with invalid version #293
  • Prevent pushing of dogus with different image #295

v2.13.0 - 2020-07-01

Added

  • Resolved Issue 291 which adds the fields default and validation to configuration fields inside dogu.json

Changed

  • Use sonar branch plugin in sonarQube analysis

v2.12.0 - 2020-05-12

Attention! To use cesapp as a dependency in other projects, use github/cloudogu/cesapp/v2!

Changed

  • Switch cesapp go module to v2 to make it compatible to go modules standard

v2.11.0 - 2020-04-22

Added

  • Support for local backup-type
  • New CLI switch for restore --keep-existing-backup-config

When restoring a previously made backup it is now possible to keep the CES instance's existing backup configuration. This is esp. interesting when the backup configuration has changed between the time when the backup was created and the current configuration. With cesapp restore --id <your backup id> --all --keep-existing-backup-config, the restore process saves the existing backup configuration, and proceeds to restore the backup as usual. After all subsystems are restored, the previous configuration from the CES host overwrites the one from the backup. After all is done, the system is ready to backup out-of-the-box. (#283)

Fixed

This version fixes a defect in which interaction with the backup repository may lead to a state of being indefinitely unresponsive. This can only happen with the backup type SFTP under the premise that no key was given. This release configures the SFTP connection in a way that the SFTP host rather fails with an error message instead of a password prompt. (#279)

Changed

  • When applying a minimal blueprint file (consisting only of ID and cesapp properties) dogus are no longer stopped and restarted in order to reduce downtime. (#285)

v2.10.0 - 2020-04-15

Added

  • Recursive dependency check before dogu upgrade
  • --all flag for healthy command

v2.9.0 - 2020-04-03

Added

  • add information about last cesapp version during an upgrade process

v2.8.0 - 2020-04-02

Added

  • Pre-Backup step: A bash script that will be automatically executed before each backup. To use, set the registry key config/backup/pre_backup_script to value /path/to/script.sh.
  • Post-Backup step: A bash script that will be automatically executed after each successful backup. To use, set the registry key config/backup/post_backup_script to value /path/to/script.sh.

v2.7.0 - 2020-03-23

Added

  • Pre-Restore step: A bash script that will be automatically executed before each restore. To use, set the registry key config/backup/pre_restore_script to value /path/to/script.sh.
  • Post-Restore step: A bash script that will be automatically executed after each successful restore. To use, set the registry key config/backup/post_restore_script to value /path/to/script.sh.

v2.6.0 - 2020-03-12

Added

  • Health checks before dogu and blueprint upgrade

v2.5.1 - 2020-03-06

Fixed

  • Fix sending of previous id during re-register of an instance

v2.5.0 - 2020-03-04

Added

  • instance register command: Allows to (re-) register a ces instance

    • AuthenticationEndpoint config value

v2.4.2 - 2020-02-24

Fixed

  • Blueprint upgrading: When during a Blueprint upgrade the cesapp is about to be updated the maintenance mode was not deactivated. This release fixes this behaviour and deactivates the maintenance mode during the (implicit) second cesapp call.## v2.4.1 - 2020-02-13
  • Please note that the maintenance mode will not be deactivated if you have set a custom maintenance mode message. (#260)
  • In the same scope the error handling was made more robust.

v2.4.1 - 2020-02-13

Fixed

  • Etcd permission issue after restore (#256)
  • Swallowing of errors messages during backup & restore

v2.4.0 - 2020-01-21

Added

  • The configuration.json is now validated at program start

Removed

  • Broken bash autocomplete

Changed

  • Use go modules instead of glide
  • Updated makefiles
  • Fix static-analysis and SonarQube integration

Fixed

  • Blueprint log output for package actions are less misleading (#250)
  • Tolerate restarting dogus during health check (#252)
  • Detect critical health check results without errors

v2.3.0 - 2020-01-16

Changed

  • Docker-Registry can be accessed anonymous
  • Dogu-Registry can be accessed anonymous

v2.2.0 - 2020-01-09

Changed

  • Forward Standard Out/Err streams of restic commands to simplify debugging (#242)

Removed

  • Omit some steps of the complete backup. The steps included dogu specific data which is not required at the moment and were error prone.

v2.1.0 - 2019-12-10

Added

  • Upgrading a Dogu via Blueprint should clean up the cache and remove the unused Docker image; #215

Fixed

  • More places that regex on dogu names match only on parts of dogu names; #232
  • Fix CI pipeline to properly attach unit test results; #234

v2.0.1 - 2019-12-04

Fixed

  • Fixes a defective self-update behaviour during cesapp upgrade blueprint in which the cesapp upgrades to a newer version (#229)

    • This defect affects versions from v1.4.0 up to v2.0.0. You have to manually upgrade the cesapp version.

v2.0.0 - 2019-11-28

Breaking change ahead. Please see the 'removed' section of this release.

Changed

  • Behaviour of cesapp blueprint upgrade

    • It is ensured that registry config changes take place before the dogu is installed or upgraded. To prevent unintended side effects the changes happen after stopping the dogus.
    • The new coordinated way of stopping and starting of dogus should result in a faster blueprint upgrade process since far less stops and starts are needed.
    • The installation of dogus is now more reliable since the process waits for service account providers.
    • Due to the nature of pulling Docker images first and purging some afterwards a slightly higher disk usage has to be accounted for during the time of the blueprint upgrade. In turn the number of Docker image layers that need to be transferred are averagely reduced.

Removed

  • The change above made checking the Dogu registry for availability during cesapp upgrade (both versions dogu and blueprint obsolete.

    • command switch --registryCheckTimeout is removed in favor of pulling dogu images to the host up front
    • Please note that calling cesapp upgrade with this switch will now no longer be accepted.

v1.8.1 - 2019-11-27

Fixed

  • Fix regex to match the dogu name, including a leading "/"; #227

v1.8.0 - 2019-11-27

Added

  • criticalprocessrunning key added to global registry; #222

    • used to ensure that only one critical process can run at a time
    • registry key has json value with a process name
    • the key is added whenever a critical process is started
    • the key is removed after the process finished regardless of potential errors

Changed

  • the backup and restore process now use criticalprocessrunning key

v1.7.1 - 2019-11-25

Fixed

  • Fix regex to match the complete dogu name; #225

v1.7.0 - 2019-11-22

Added

  • default sorting of newest backup start time first
  • export check whether a backup was successful
  • new field for BlueprintID in Backup struct
  • basic backup id validation
  • export time parsing helpers which parse a time in RFC3339 format

v1.6.1 - 2019-11-13

Added

  • New command parameter cesapp login --password-file allows to supply the instance token as file instead of a parameter #188

Changed

  • cesapp login #188

    • Fixes a defect where cesapp login shows now a proper error message if the login was not successfully instead of mistakenly using the cache.
    • As a security measure, cesapp login allows to leave away one or both credential parameter. The credentials are prompted on the CLI then .
  • cesapp restore #219

    • Fixes a rare defect where restoring backups no longer stops the restore process when mistakenly a cross-blueprint upgrade was detected

v0.0.1 - v1.6.0 / previous versions

Up till version v1.6.0 there was no change log. Please see the release page