rasmus/mq-container
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
Rasmus Lauritsen
2024-10-28 23:04:48 +01:00
commit 1ee55157f1
911 changed files with 325331 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
docs/building.md Executable file
View File

@@ -0,0 +1,57 @@
# Building a container image
## Prerequisites
You need to have the following tools installed:
* [Docker](https://www.docker.com/) 20.10 or later, or [Podman](https://podman.io) 4.4 or later.
* [GNU make](https://www.gnu.org/software/make/)
## Building Images
To build an IBM MQ image, navigate to the appropriate section:
- [Building a production image](#building-a-production-image)
- [Building a developer image](#building-a-developer-image)
## Building a production image
### MQ Continuous Delivery (CD)
The procedure below is for building the 9.4.1 release, on `amd64`, `ppc64le` and `s390x` architectures.
1. Create a `downloads` directory in the root of this repository
2. Identify the correct eImage part number for your architecture from https://www.ibm.com/support/pages/downloading-ibm-mq-94 and download
3. Ensure the `tar.gz` file is in the `downloads` directory
4. Run `make build-advancedserver`
If you have an MQ archive file with a different file name, you can specify a particular file (which must be in the `downloads` directory). You should also specify the MQ version, so that the resulting image is tagged correctly, for example:
```bash
MQ_ARCHIVE=mq-1.2.3.4.tar.gz MQ_VERSION=1.2.3.4 make build-advancedserver
```
### MQ LTS
The procedure below is for building the 9.4.0 release, on `amd64`, `ppc64le` and `s390x` architectures.
1. Create a `downloads` directory in the root of this repository
2. Identify the correct eImage part number for your architecture from https://www.ibm.com/support/pages/downloading-ibm-mq-94 and download
3. Ensure the `tar.gz` file is in the `downloads` directory
4. Run `make build-advancedserver`
If you have an MQ archive file with a different file name, you can specify a particular file (which must be in the `downloads` directory). You should also specify the MQ version, so that the resulting image is tagged correctly, for example:
```bash
MQ_ARCHIVE=mq-1.2.3.4.tar.gz MQ_VERSION=1.2.3.4 make build-advancedserver
```
## Building a developer image
Run `make build-devserver`, which will download the latest version of MQ Advanced for Developers. This is available on the `amd64` and `arm64` (Apple Silicon) architectures.
You can use the environment variable `MQ_ARCHIVE_DEV` to specify an alternative local file to install from (which must be in the `downloads` directory).
## Installed components
This image includes the core MQ server, Java, language packs, GSKit, and web server. This is configured in the `mq-redux` build stage in `Dockerfile-server`.

75
docs/developer-config.md Normal file
View File

@@ -0,0 +1,75 @@
# Default developer configuration
If you build this image with MQ Advanced for Developers, then an optional set of configuration can be applied automatically. This configures your Queue Manager with a set of default objects that you can use to quickly get started developing with IBM MQ. If you do not want the default objects to be created you can set the `MQ_DEV` environment variable to `false`.
## Using Secrets to set passwords for app & admin users
Secrets must be used to set the passwords for `admin` and `app` user. For setting password for user `admin`, `mqAdminPassword` secret must be created and for user `app`, `mqAppPassword` secret must be created.
### Example usage with podman:
Create podman secrets with secret names as “mqAdminPassword” & "mqAppPassword":
- `printf "passw0rd" | podman secret create mqAdminPassword -`
- `printf "passw0rd" | podman secret create mqAppPassword -`
Run container referencing mounted secrets:
- `podman run --secret mqAdminPassword,type=mount,mode=0777 --secret mqAppPassword,type=mount,mode=0777 --env LICENSE=accept --env MQ_QMGR_NAME=QM1 --publish 1414:1414 --publish 9443:9443 --detach --name QM1 icr.io/ibm-messaging/mq:latest`
### Example usage with docker:
Docker secrets are only available via Docker Swarm services, hence to create a secret using docker, Docker Swarm must be used.
Create docker secrets with secret names as “mqAdminPassword” & "mqAppPassword":
- `printf "passw0rd" | docker secret create mqAdminPassword –`
- `printf "passw0rd" | docker secret create mqAppPassword –`
Run container referencing mounted secret:
- `docker service create --secret mqAdminPassword --secret mqAppPassword --env LICENSE=accept --env MQ_QMGR_NAME=QM8 --publish 1414:1414 --publish 9443:9443 --detach --name QM8 icr.io/ibm-messaging/mq`
## Environment variables
From IBM MQ v9.4.0.0, environment variables `MQ_ADMIN_PASSWORD` and `MQ_APP_PASSWORD` are deprecated. Secrets as detailed in the previous section must be used to set the passwords for `admin` and `app` user.
The MQ Developer Defaults supports some customization options, these are all controlled using environment variables:
* **MQ_DEV** - Set this to `false` to stop the default objects being created.
* **MQ_ADMIN_PASSWORD** - Specify the password of the `admin` user. Must be at least 8 characters long.
* **MQ_APP_PASSWORD** - Specify the password of the `app` user. If set, this will cause the `DEV.APP.SVRCONN` channel to become secured and only allow connections that supply a valid userid and password. Must be at least 8 characters long.
## Details of the default configuration
The following users are created:
* User **admin** for administration. This user is created only if the password is set. Secrets must be used to set the password.
* User **app** for messaging (in a group called `mqclient`). This user is created only if the password is set. Secrets must be used to set the password.
Users in `mqclient` group have been given access connect to all queues and topics starting with `DEV.**` and have `put`, `get`, `pub`, `sub`, `browse` and `inq` permissions.
The following queues and topics are created:
* DEV.QUEUE.1
* DEV.QUEUE.2
* DEV.QUEUE.3
* DEV.DEAD.LETTER.QUEUE - configured as the Queue Manager's Dead Letter Queue.
* DEV.BASE.TOPIC - uses a topic string of `dev/`.
Two channels are created, one for administration, the other for normal messaging:
* DEV.ADMIN.SVRCONN - configured to only allow the admin user to connect into it. The `admin` user can be used with the password configured via secret.
* DEV.APP.SVRCONN - does not allow administrative users to connect. Only the `app` user can connect. The password would be as configured by the secret.
## Web Console
By default the MQ Advanced for Developers image will start the IBM MQ Web Console that allows you to administer your Queue Manager running on your container. When the web console has been started, you can access it by opening a web browser and navigating to `https://<Container IP>:9443/ibmmq/console`. Where `<Container IP>` is replaced by the IP address of your running container.
When you navigate to this page you may be presented with a security exception warning. This happens because, by default, the web console creates a self-signed certificate to use for the HTTPS operations. This certificate is not trusted by your browser and has an incorrect distinguished name.
If you choose to accept the security warning, you will be presented with the login menu for the IBM MQ Web Console. The login for the console is:
* **User:** admin
* **Password:** The password for the `admin` user must be specified using a secret, as described above.
If you do not wish the web console to run, you can disable it by setting the environment variable `MQ_ENABLE_EMBEDDED_WEB_SERVER` to `false`.

52
docs/internals.md Normal file
View File

@@ -0,0 +1,52 @@
# Internals
This page documents internal code details and design decisions.
The resulting Docker image contains the following:
* Base linux distribution - this provides standard Linux libraries (such as "glibc") and utilities (such as "ls" and "grep") required by MQ
* MQ installation (under `/opt/mqm`)
* Three additional programs, to enable running in a containerized environment:
- `runmqserver` - The main process, which creates and runs a queue manager
- `runmqdevserver` - The main process for MQ Advanced for Developers
- `chkmqhealthy` - Checks the health of the queue manager. This can be used by (say) a Kubernetes liveness probe.
- `chkmqready` - Checks if the queue manager is ready for work. This can be used by (say) a Kubernetes readiness probe.
- `chkmqstarted` - Checks if the queue manager has successfully started. This can be used by (say) a Kubernetes startup probe.
## runmqserver
The `runmqserver` command has the following responsibilities:
* Checks license acceptance
* Sets up `/var/mqm`
- MQ data directory needs to be set up at container creation time. This is done using the `crtmqdir` utility, which was introduced in MQ V9.0.3
- It assumes that a storage volume for data is mounted under `/mnt/mqm`. It creates a sub-directory for the MQ data, so `/var/mqm` is a symlink which resolves to `/mnt/mqm/data`. The reason for this is that it's not always possible to change the ownership of an NFS mount point directly (`/var/mqm` needs to be owned by "mqm"), but you can change the ownership of a sub-directory.
* Acts like a daemon
- Handles UNIX signals, like SIGTERM
- Works as PID 1, so is responsible for [reaping zombie processes](https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/)
* Creating and starting a queue manager
* Configuring the queue manager, by running any MQSC scripts found under `/etc/mqm`
* Starts the MQ web server (if enabled)
* Starting Prometheus metrics generation for the queue manager (if enabled)
* Indicates to the `chkmqready` command that configuration is complete, and that normal readiness checking can happen. This is done by writing a file into `/run/runmqserver`
In addition, for MQ Advanced for Developers only, the web server is started.
## runmqdevserver
The `runmqdevserver` command is added to the MQ Advanced for Developers image only. It does the following, before invoking `runmqserver`:
1. Sets passwords based on supplied environment variables
2. Generates MQSC files to put in `/etc/mqm`, based on a template, which is updated with values based on supplied environment variables.
3. If requested, it creates TLS key stores under `/run/runmqserver`, and configures MQ and the web server to use them
## Prometheus metrics
[Prometheus](https://prometheus.io) metrics are generated for the queue manager as follows:
1. A connection is established with the queue manager
2. Metrics are discovered by subscribing to topics that provide meta-data on metric classes, types and elements
3. Subscriptions are then created for each topic that provides this metric data
4. Metrics are initialised using Prometheus names mapped from their element descriptions
5. The metrics are then registered with the Prometheus registry as Prometheus Gauges
6. Publications are processed on a periodic basis to retrieve the metric data
7. An HTTP server is setup to listen for requests from Prometheus on `/metrics` port `9157`
8. Prometheus requests are handled by updating the Prometheus Gauges with the latest metric data
9. These updated Prometheus Gauges are then collected by the Prometheus registry

28
docs/pluggable-connauth.md Normal file
View File

@@ -0,0 +1,28 @@
### Queue Manager Connection Authentication using secrets
Prior to IBM MQ v9.4.0.0, passwords could be supplied through MQ_ADMIN_PASSWORD and MQ_APP_PASSWORD environment variables. From IBM MQ v9.4.0.0, supplying passwords through environment variables is deprecated and not recommended. IBM MQ v9.4.0.0 provides a new authentication mode to allow developers using mq-container developer image to authenticate users. In this new authentication mode passwords for `app` and `admin` users are supplied through secrets securely mounted into the file system. Secret with name `mqAppPassword` and `mqAdminPassword` can now be used to supply password for users `app` and `admin`. This is in addition to the existing methods of users authentication, described in [User authentication and authorization for IBM MQ in containers] (https://www.ibm.com/docs/en/ibm-mq/latest?topic=containers-user-authentication-authorization-mq-in)
**Please note:**
1. This new feature is enabled only when environment variable `MQ_CONNAUTH_USE_HTP=true` is set while starting the MQ Container.
2. When enabled, the `AuthType` value of the ConnectionAuthentication (`CONNAUTH`) is ignored and secrets are used. However,
the MQ authority records created using (`setmqaut` or `AUTHREC`) will be in effect while using the secrets.
3. Channel Authentication records (`CHLAUTH`) will be in effect while using the secrets.
4. This is developer only feature and not recommended for use in Production.
### Using Secrets
1. `mqAppPassword` and `mqAdminPassword` secrets passed to the container are mounted under /run/secrets directory. These secrets are used for authentication of `app` or `admin` users. It must be noted that `app` and `admin` user do not have any default password.
2. The `app` user is authorized to access `DEV.*` objects of the queue manager.
#### Next Steps:
Use an administrative tool or your application to connect to queue manager using the passwords that are set as secrets for user `app` and `admin`.
**Please note**: When an authentication request is made with a userid other than `app` or `admin`, then the authentication process is delegated to queue manager to handle. This will then use `IDPWOS` or `LDAP` modes for further processing.
#### Troubleshooting
A log file named `mqsimpleauth.log` is generated under `/var/mqm/errors` directory path of the container. This file will contain all the failed connection authentication requests. Additional information is logged to this file if the environment variable `DEBUG` is set to `true`.
**Please note**: This log file will be wiped when the queue manager is next started.

20
docs/security.md Normal file
View File

@@ -0,0 +1,20 @@
# Security
## Container runtime
### User
The MQ server image is run using with UID 1001, though this can be any UID, with a fixed GID of 0 (root).
### Capabilities
The MQ Advanced image requires no Linux capabilities, so you can drop any capabilities which are added by default. For example, in Docker you could do the following:
```sh
docker run \
--cap-drop=ALL \
--env LICENSE=accept \
--env MQ_QMGR_NAME=QM1 \
--detach \
ibm-mqadvanced-server:9.4.1.0-amd64
```

56
docs/testing.md Normal file
View File

@@ -0,0 +1,56 @@
# Testing
## Prerequisites
You need to ensure you have the following tools installed:
* [Docker](https://www.docker.com/) 19.03 or higher (API version 1.40)
* [GNU make](https://www.gnu.org/software/make/)
* [Go](https://golang.org/) - only needed for running the tests
## Running the tests
There are two main sets of tests:
1. Unit tests, which are run during a build
2. Docker tests, which test a complete Docker image, using the Docker API
### Running the Docker tests
The Docker tests can be run locally on a machine with Docker. For example:
```
make test-devserver
make test-advancedserver
```
You can specify the image to use directly by using the `MQ_IMAGE_ADVANCEDSERVER` or `MQ_IMAGE_DEVSERVER` variables, for example:
```
MQ_IMAGE_ADVANCEDSERVER=ibm-mqadvanced-server:9.4.1.0-amd64 make test-advancedserver
```
You can pass parameters to `go test` with an environment variable. For example, to run the "TestGoldenPath" test, run the following command:
```
TEST_OPTS_DOCKER="-run TestGoldenPath" make test-advancedserver
```
You can enable additional tracing of container engine commands by setting `TEST_LOG_CONTAINER_COMMANDS` to `true`:
```
TEST_LOG_CONTAINER_COMMANDS="true" make test-advancedserver
```
You can also use the same environment variables you specified when [building](./building), for example, the following will try and test an image called `ibm-mqadvanced-server:9.2.0.0-amd64`:
```
MQ_VERSION=9.2.0.0 make test-advancedserver
```
### Running the Docker tests with code coverage
You can produce code coverage results from the Docker tests by running the following:
```
make build-advancedserver-cover
make test-advancedserver-cover
```
In order to generate code coverage metrics from the Docker tests, the build step creates a new Docker image with an instrumented version of the code. Each test is then run individually, producing a coverage report each under `test/docker/coverage/`. These individual reports are then combined. The combined report is written to the `coverage` directory.

17
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,17 @@
# Troubleshooting
## AMQ7017: Log not available
If you see this message in the container logs, it means that the directory being used for the container's volume doesn't use a filesystem supported by IBM MQ. To solve this, you need to make sure the container's `/mnt/mqm` volume is put on a supported filesystem. The best way to do this is to use [Docker volumes](https://docs.docker.com/storage/volumes/), instead of bind-mounted directories.
## Container command not found or does not exist
This message also appears as "System error: no such file or directory" in some versions of Docker. This can happen using a Docker client on Windows, and is related to line-ending characters. When you clone the Git repository on Windows, Git is often configured to convert any UNIX-style LF line-endings to Windows-style CRLF line-endings. Files with these line-endings end up in the built Docker image, and cause the container to fail at start-up. One solution to this problem is to stop Git from converting the line-ending characters, with the following command:
```
git config --global core.autocrlf input
```
## Old Linux kernel versions
MQ works best if you have a Linux kernel version of V3.16 or higher (run `uname -r` to check).
If you have an older version, you might need to add the [`--ipc host`](https://docs.docker.com/engine/reference/run/#ipc-settings-ipc) option when you run an MQ container. The reason for this is that IBM MQ uses shared memory, and on Linux kernels prior to V3.16, containers are usually limited to 32 MB of shared memory. In a [change](https://git.kernel.org/cgit/linux/kernel/git/mhocko/mm.git/commit/include/uapi/linux/shm.h?id=060028bac94bf60a65415d1d55a359c3a17d5c31
) to Linux kernel V3.16, the hard-coded limit is greatly increased. This kernel version is available in Ubuntu 14.04.2 onwards, Fedora V20 onwards, and boot2docker V1.2 onwards. Some Linux distributions, like Red Hat Enterprise Linux, patch older kernel versions, so you might find that the patch has been applied already, even if you see a lower kernel version number. If you are using a host with an older kernel version, then you can still run MQ, but you have to give it access to the host's IPC namespace using the [`--ipc host`](https://docs.docker.com/engine/reference/run/#ipc-settings-ipc) option on `docker run`. Note that this reduces the security isolation of your container.

146
docs/usage.md Normal file
View File

@@ -0,0 +1,146 @@
# Usage
In order to use the image, it is necessary to accept the terms of the IBM MQ license. This is achieved by specifying the environment variable `LICENSE` equal to `accept` when running the image. You can also view the license terms by setting this variable to `view`. Failure to set the variable will result in the termination of the container with a usage statement. You can view the license in a different language by also setting the `LANG` environment variable.
> **Note**: You can use `podman` instead of `docker` in any of the examples on this page.
## Running with the default configuration
You can run a queue manager with the default configuration and a listener on port 1414 using the following command. For example, the following command creates and starts a queue manager called `QM1`, and maps port 1414 on the host to the MQ listener on port 1414 inside the container, as well as port 9443 on the host to the web console on port 9443 inside the container:
```sh
docker run \
--env LICENSE=accept \
--env MQ_QMGR_NAME=QM1 \
--publish 1414:1414 \
--publish 9443:9443 \
--detach \
icr.io/ibm-messaging/mq
```
## Running with the default configuration and a volume
The above example will not persist any configuration data or messages across container runs. In order to do this, you need to use a [volume](https://docs.docker.com/storage/volumes/). For example, you can create a volume with the following command:
```sh
docker volume create qm1data
```
You can then run a queue manager using this volume as follows:
```sh
docker run \
--env LICENSE=accept \
--env MQ_QMGR_NAME=QM1 \
--publish 1414:1414 \
--publish 9443:9443 \
--detach \
--volume qm1data:/mnt/mqm \
icr.io/ibm-messaging/mq
```
The Docker image always uses `/mnt/mqm` for MQ data, which is correctly linked for you under `/var/mqm` at runtime. This is to handle problems with file permissions on some platforms.
## Running with the default configuration and Prometheus metrics enabled
You can run a queue manager with [Prometheus](https://prometheus.io) metrics enabled. The following command will generate Prometheus metrics for your queue manager on `/metrics` port `9157`:
```sh
docker run \
--env LICENSE=accept \
--env MQ_QMGR_NAME=QM1 \
--env MQ_ENABLE_METRICS=true \
--publish 1414:1414 \
--publish 9443:9443 \
--publish 9157:9157 \
--detach \
icr.io/ibm-messaging/mq
```
## Customizing the queue manager configuration
You can customize the configuration in several ways:
1. For getting started, you can use the [default developer configuration](developer-config.md), which is available out-of-the-box for the MQ Advanced for Developers image
2. By creating your own image and adding your own MQSC file into the `/etc/mqm` directory on the image. This file will be run when your queue manager is created.
3. By using [remote MQ administration](https://www.ibm.com/docs/en/ibm-mq/9.4?topic=administering-working-remote-mq-objects), via an MQ command server, the MQ HTTP APIs, or using a tool such as the MQ web console or MQ Explorer.
Note that a listener is always created on port 1414 inside the container. This port can be mapped to any port on the Docker host.
The following is an *example* `Dockerfile` for creating your own pre-configured image, which adds a custom MQ configuration file:
```dockerfile
FROM icr.io/ibm-messaging/mq
USER 1001
COPY 20-config.mqsc /etc/mqm/
```
Here is an example corresponding `20-config.mqsc` script, which creates two local queues:
```mqsc
DEFINE QLOCAL(MY.QUEUE.1) REPLACE
DEFINE QLOCAL(MY.QUEUE.2) REPLACE
```
The file `20-config.mqsc` should be saved into the same directory as the `Dockerfile`.
## Running MQ commands
It is recommended that you configure MQ in your own custom image. However, you may need to run MQ commands directly inside the process space of the container. To run a command against a running queue manager, you can use `docker exec`, for example:
```sh
docker exec \
--tty \
--interactive \
${CONTAINER_ID} \
dspmq
```
Using this technique, you can have full control over all aspects of the MQ installation. Note that if you use this technique to make changes to the filesystem, then those changes would be lost if you re-created your container unless you make those changes in volumes.
## Supplying TLS certificates
If you wish to supply TLS Certificates that the queue manager and MQ Console should use for TLS operations then you must supply a PKCS#1 or unencrypted PKCS#8 PEM files for both the certificates and private keys in the following directories:
* `/etc/mqm/pki/keys/<Label>` - for certificates with public and private keys
* `/etc/mqm/pki/trust/<index>` - for certificates with only the public key
For example, if you have an identity certificate you wish to add with the label `mykey` and 2 certificates you wish to add as trusted then you would need to add the files into the following locations where files ending in `.key` contain private keys and `.crt` contain certificates:
- `/etc/mqm/pki/keys/mykey/tls.key`
- `/etc/mqm/pki/keys/mykey/tls.crt`
- `/etc/mqm/pki/keys/mykey/ca.crt`
- `/etc/mqm/pki/trust/0/tls.crt`
- `/etc/mqm/pki/trust/1/tls.crt`
This can be achieved by either mounting the directories or files into the container when you run it or by baking the files into the correct location in the image.
If you supply multiple identity certificates then the first label alphabetically will be chosen as the certificate to be used by the MQ Console and the default certificate for the queue manager. If you wish to use a different certificate on the queue manager then you can change the certificate to use at runtime by executing the MQSC command `ALTER QMGR CERTLABL('<newlabel>')`
It must be noted that queue manager certificate with a Subject Distinguished Name (DN) same as it's Issuer certificate (CA) is not supported. Certificates must have a unique Subject Distinguished Name.
## Running with a read-only root filesystem
Starting with version 9.3.4.0, you can run MQ container with a read-only root filesystem. In order to do this, you need to mount three [volumes](https://docs.docker.com/storage/volumes/) into the MQ container, one for queue manager data, one for `run` directory that will contain files used for queue manager configuration and one for `tmp` directory that will be used for collecting diagnostic data. You also need specify `--read-only` parameter while starting the container. Following describes the steps to run MQ container with a read-only root filesystem.
```sh
docker volume create qm1data
```
```sh
docker volume create run
```
```sh
docker volume create tmp
```
You can then run a queue manager with a read-only root filesystem as follows:
```sh
docker run \
--env LICENSE=accept \
--env MQ\_QMGR\_NAME=QM1 \
--mount type=volume,source=run,destination=/run \
--mount type=volume,source=tmp,destination=/tmp \
--mount type=volume,source=qm1data,destination=/mnt/mqm \
--read-only \
--publish 1414:1414 \
--detach \
icr.io/ibm-messaging/mq
```