//Cloudogu EcoSystem Docs

Upgrade the CES with Blueprints

CES specifications are recorded in a so-called Blueprint.

A blueprint is a document written in JSON. It describes in a declarative way the target state of an EcoSystem regarding:

  • its installed dogus
  • its other packages that are necessary or useful for operation.

Structure of a Blueprint

In this section, top-level properties of a blueprint are explained.

Example Blueprint

The properties of the properties listed here are explained in the following section "Structure of a Blueprint".

{
  "blueprintApi": "v1",
  "blueprintId": "blueprint-id",
  "cesappVersion": "4.2.3",
  "dogus": [
    {
      "name": "official/redmine",
      "version": "4.2.3-10",
      "targetState": "present"
    },
    {
      "name": "official/postfix",
      "version": "3.2.4-1"
    }
  ],
  "packages": [
    {
      "name": "backup-watcher",
      "version": "1.0.0"
    }
  ],
  "registryConfig": {
    "nexus": {
      "successfulInitialConfiguration": "true"
    },
    "_global": {
      "domain": "ces.local"
    }
  },
  "registryConfigEncrypted": {
    "nexus": {
      "admin_password": "nAtK3WSvdlYzeNmQhBkCeaTEGPMCHTcJimdtkncE...=="
    }
  }
}

Property blueprintApi

This property defines the version of the blueprint. For each version, different possible implementations are that indicate the functionality of the cesapp that interprets this blueprint. Thus also in future with progress of the cesapp still previous blueprints can be evaluated and used.

This property must not be empty.

Currently the value v1 is allowed.

Property blueprintId

This property is used for unique identification across different blueprints, but is only descriptive. descriptive character. The framework in which the uniqueness is sufficient is determined by the organization that uses these blueprints for itself. uses these Blueprints for itself. It is appropriate to use a versioning scheme.

Since blueprints aim to track system states, blueprints should not be modified after they have been applied at least once to one or more EcoSystem instances. Instead, a successive identifier should be chosen for a new Blueprint.

Example of sequential Blueprint IDs for five different Blueprints:

"1.0.0 RC2"
"1.0.0"
"2.0.0 RC1"
"2.0.1 RC1"
"2.0.1"

This property must not be empty.

Property cesappVersion

This property indicates the exact version of the cesapp, i.e. the tool that significantly orchestrates the CES. If version differs from the version of the cesapp previously installed in the CES, then the version defined in the Blueprint is installed and then the upgrade process is continued.

The version of the cesapp can correspond directly with functionalities or bugfix statuses and must fit into the overall overall picture of the Dogus. As with all other version specifications in a Blueprint, only a fixed version specification is required and no minimum version or version intervals are possible to ensure auditable traceability.

This property must not be empty.

Property 'dogus

This property defines a set of dogus to be installed or modified if they:

  • are not yet present in the EcoSystem or
  • are present in the EcoSystem, but not in the correct version.

This property and its values are optional, the order of the entries is not relevant. If a Blueprint is applied to on an existing EcoSystem where certain dogus are already installed but not named, then the handling of these dogus is then the treatment of these dogus is ignored and they remain as they are.

Dogu-Property within dogus

If present, individual dogu entries must or can contain these properties:

  • name

    • consists of dogu namespace and dogu name
    • example: "official/redmine"
    • Mandatory entry
  • version

    • the version of the dogu to install (depending on the dogu)
    • example: "3.15.2-2
    • Conditional mandatory if "targetState": "present" is used to install this part.
  • targetState

    • Indicates whether the dogu is allowed to be present in the target state or not.

      • present: The dogu must be present in the specified version, or an existing version will be updated. updated
      • absent: The dogu must not be present, or an existing version will be uninstalled without replacement.
    • Optional specification. If this property is not present, then present is assumed by default.

Dogus must not be named more than once in the blueprint. Not even if they come from different namespaces.

packages property

This property defines a set of system packages to be installed or modified if they:

  • do not yet exist on the EcoSystem host system, or
  • are installed on the EcoSystem host system, but not in the correct version.

These property as well as their values are optional, the order of the entries is not relevant. If a Blueprint is applied to on an existing EcoSystem where certain packages are already installed but not named, then the handling of these packages is then the handling of these packages is ignored and they remain as they are.

Package-Properties within packages

If present, individual package entries must or can contain these properties:

  • name

    • the name of the package
    • example: "backup-watcher"
    • mandatory entry
  • version

    • the version of the package to install
    • example: "1.0.0
    • Conditional mandatory if "targetState": "present" is used to install this part.
  • targetState

    • Indicates whether the package is allowed to be present in the target state or not.

      • present: The package must be present in the specified version, or an existing version will be updated
      • absent`: the package must not be present, or an existing version will be uninstalled without replacement.
    • Optional specification. If this property is not present, then present is assumed by default.

Because of the special importance of cesapp in the CES context, it must not be defined in the packages section of the blueprint, but only in the cesappVersion property.

Maintenance mode behavior during a blueprint upgrade

As soon as a Blueprint upgrade is started, the CES maintenance mode is activated to prevent Dogus from being used during the upgrade. during the process. When the process is complete, the maintenance mode is deactivated and the CES can be used as usual. can be used as usual. If the maintenance mode is already activated in the CES before the start of the blueprint upgrade it will be maintained during the entire process and will not be deactivated at the end.

Execution

If the blueprint described above is executed using:

cesapp upgrade blueprint Blueprint.json

All dogus listed there will be installed in the specified version. Downgrades are generally not supported, so errors would occur if the dogus are already installed in a higher version.

Set and remove registry keys

Setting etcd values using Blueprint

Values can be set in etcd using the Blueprint upgrade. The structure of the corresponding area of blueprint.json corresponds to the structure of setup.json.

Global and dogu-specific etcd values can be set. Encrypted etcd values have to be set according to the same scheme under the section registryConfigEncrypted.

Setting the keys is the last step in the blueprint upgrade. First all global values and then all dogu-specific etcd values are set.

Some dogus may need to be restarted for the registry changes to take effect. This will be pointed out when running the blueprint upgrade.

It is not possible to store global etcd values in encrypted form. The same applies to etcd values of a Dogus which is is not installed. If it is nevertheless attempted, the blueprint upgrade will fail.

The encrypted values are stored encrypted in the blueprint history.

Example: blueprint.json to set etcd values
    {
        "blueprintApi": "v1",
        "blueprintId": "BPID",
        "cesappVersion": { "1.4.0",
        "registryConfig": {
          "doguName": {
            "doguConfig" : "doguConfigValue",
            "doguConfigGroup" : {
              "doguConfig2" : "doguConfigValue",
              "doguConfig3" : "doguConfigValue2"
            }
          },
          "_global": {
            "globalConfigKey": "globalConfigValue"
          }
        },
        "registryConfigEncrypted": {
          "backup": {
            "key1" : { "val1",
            "key2" : {
              "v1" : "val2",
              "v2" : "val3"
            }
          }
        },
        "registryConfigAbsent" : [
          "_global/dir",
          "_global/dir2/key1",
          "_global/dir2/key2",
          "cas/dir1/key1",
          "cas/dir1/key2",
          "cas/dir2",
          "cas/key2"
        ]
    }
Property registryConfig

This section corresponds to the structure of the registryConfig property of setup.json. In nested JSON objects the values and structure of the config area can be edited in the etcd. It is to be noted that here the values are stored in plain text in both the blueprint and the etcd.

Property registryConfigEncrypted

This section corresponds to the already explained registryConfig section. The difference, however, is that the values are encrypted before being stored in etcd using the public keys associated with the dogus. Before the The values are also replaced with their encrypted counterparts before the blueprint is historized.

Property registryConfigAbsent

This section performs the function of deleting registry keys. The specified keys will be deleted from the registry before setting new keys in the registry. The section differs from the registryConfig and registryConfigEncrypted. Here all keys or alternatively directories are specified with their full path are specified. The nesting of keys is specified here with a /. Toplevel entries cannot be deleted. be deleted. So e.g. the entry "_global" or "cas" would lead to an error.

Example:

Setting keys:

{
  "registryConfig": {
    "cas": {
      "keys": {
        "key1": "value1",
        "key2": "value2"
      }
    }
  }
}

Delete these keys:

{
  "registryConfigAbsent": [
    "cas/keys/key1",
    "cas/keys/key2"
  ]
}

Or alternatively:

{
  "registryConfigAbsent": [
    "cas/keys"
  ]
}

Blueprint upgrade customizations via Blueprint Mask

A customized upgrade of a CES instance using Blueprint and Blueprint Mask can be done via

cesapp upgrade blueprint --blueprint-mask blueprintmask.json blueprint.json

where blueprint.json contains the general information which is adjusted by the information in blueprintmask.json.

Structure of a Blueprint Mask

A Blueprint Mask is created like a Blueprint as a JSON formatted text file. In it, various properties are described. Example:

{
  "blueprintMaskApi": "v1",
  "blueprintMaskId": "ExampleMask",
  "dogus": [
    {
      "name": "official/redmine",
      "targetState": "absent"
    },
    {
      "name": "postfix",
      "targetState": "absent"
    }
  ]
}
Property blueprintMaskApi

This property defines the version of the blueprint mask. Depending on the version, different implementations are conceivable that indicate the functionality of the cesapp that interprets this Blueprint Mask. In this way, previous Blueprint Masks can still be evaluated and applied in the future as the cesapp progresses.

This property must not be empty.

Currently the value v1 is allowed.

Property blueprintMaskId

This property is used for unique identification across different Blueprint Masks, but is descriptive only. The framework in which the uniqueness is sufficient is determined by the organization that uses these Blueprints for itself. It is a good idea to use a versioning scheme.

This property must not be missing or empty.

Property dogus

This property contains a JSON array consisting of dogu objects. These Dogu objects are identified by their name (including namespace; e.g. "name": "official/scm") and the desired installation state (present or absent; e.g. "targetState": "present"). It is also possible to specify the name without namespace, since the namespace is mandatory in blueprint.json anyway.

Restrictions

In general, a Blueprint Mask does not modify the Blueprint JSON file, but only the information used in the upgrade process.

The Blueprint Mask can only modify objects that are in the Blueprint to which the mask is applied. For example, no new Dogu objects can be added via the mask.

The targetState of a Dogu definition in the Mask is not validated by the cesapp. Any targetState that is not absent is interpreted as present. Even a missing targetState is interpreted as "present".

Namespace change through the blueprint mechanism

The blueprint mechanism can be used to change the namespace of a dogu. An upgrade can also take place at the same time by simply specifying a newer version for the dogu to be installed.

Example: official/myDogu is installed in version 1.0.0. In the blueprint.json to be executed myDogu is in the version 1.0.1 and in the namespace premium. A namespace change (including upgrade) is therefore performed when executing the of the blueprint.

The namespace change must be explicitly allowed via the blueprint command. This is done via the allow-namespace-switch flag. Example: cesapp upgrade blueprint --allow-namespace-switch myBlueprint.json. Installing an older version is generally not possible.

Blueprint history

The information about all performed Blueprint upgrades is stored in the etcd. Basically, the following values are historized:

  1. the time at which the last successful Blueprint upgrade was started (overwritten with each Blueprint upgrade).
  2. the time at which a Blueprint upgrade was completed.
  3. the effective blueprint.json. If a blueprint mask has been used, the effects of this are visible here.

For this purpose, in case the blueprint mask has been used during a blueprint upgrade, the following values are historized:

  1. the blueprintMask.json
  2. the blueprintOriginal.json. This is the blueprint file, without the effects of the blueprint mask.

Note: All timestamps are in the format RFC3339.

Structure etcd

The values listed above can be found under these keys in etcd:

  1. /blueprint/current.
  2. /blueprint/<starttime>/endTime
  3. /blueprint/<starttime>/blueprint.json
  4. if necessary /blueprint/<startTime>/blueprintOriginal.json
  5. if necessary /blueprint/<starttime>/blueprintMask.json

Sequence

At the start of the blueprint upgrade, the start time is recorded. After completion of the blueprint upgrade, the blueprint.json, as well as the blueprintMask.json and the blueprintOriginal.json, if applicable, including the start time, are stored directly in etcd.

If the blueprint upgrade runs successfully, the end time as well as the start time of this blueprint upgrade is subsequently stored in etcd.

Special features

If a blueprint.json contains one or more encrypted entries, it is not stored in its original form in the blueprint history.

The values to be encrypted are encrypted with the public keys of the corresponding dogus. In the blueprint.json the values to be encrypted are replaced by the encrypted ones and the changed version is stored in the Blueprint history.

If necessary, information (formatting or values that are not recognized by the cesapp) of the blueprint.json can be lost here, since this is transferred from the cesapp to the class structure of the Blueprint and a .json file is then created from it again.

Example: Querying the values in etcd

To get information about the last performed Blueprint upgrade, first the start time of the current Blueprint upgrade must be queried with etcdctl get /blueprint/current. The value could look like this: 2019-09-27T09:14:42+02:00.

The blueprint.json of the last blueprint upgrade is retrieved by the command etcdctl get blueprint/2019-09-27T09:14:42+02:00/blueprint.json.

The end time can be determined by the query etcdctl get blueprint/2019-09-27T09:14:42+02:00/endTime.

The BlueprintId, of the last successfully executed Blueprint upgrade, can be retrieved via the shell using the following command: etcdctl get /blueprint/$(etcdctl get /blueprint/current)/blueprint.json | jq -r ".blueprintId".