utms-agent/README.md

# UTMS Agent

UTMS Agent is a Java 8 background service that connects to an MQTT broker, consumes terminal messages, stores incoming data in PostgreSQL, and publishes download/profile messages back to terminals. It is designed to run as a standard Linux `systemd` service without Tanuki or YAJWS wrappers.

## Runtime Overview

The application entrypoint is `id.iptek.utms.agent.Main`.

Startup flow:

1. Load `conf/mqtt-agents.cfg`.
2. Initialize Log4j from `conf/mqtt-agents-log4j.cfg`.
3. Initialize PostgreSQL access through `DBConn`.
4. Initialize a global MQTT client for scheduled task publishing.
5. Initialize a MinIO client for presigned download URLs.
6. Load terminal ID cache asynchronously.
7. Start queue handlers for heartbeat, diagnostic, and device init messages.
8. Start configured MQTT consumer workers.
9. Start Quartz scheduler for pending download tasks.

Shutdown flow:

1. Stop Quartz scheduler so no new publish jobs start.
2. Drain queue handlers gracefully for up to 30 seconds.
3. Force interrupt queue work if the graceful timeout is exceeded.
4. Stop MQTT workers.
5. Disconnect the global MQTT client.
6. Stop the executor service.

## Project Layout

```text
src/id/iptek/utms/agent/
  Main.java                         Application lifecycle and service entrypoint
  AppConfig.java                    Properties config loader
  db/                               Database connection and utility classes
  dao/                              PostgreSQL data access objects
  model/                            Message, task, app, profile, and terminal models
  queue/                            In-memory delay queues and queue handlers
  scheduler/job/                    Quartz jobs for download task scheduling/publishing
  worker/                           MQTT consumer workers
  util/                             Shared utility classes

conf/
  mqtt-agents.cfg                   Main runtime configuration
  mqtt-agents-log4j.cfg             File-only rolling log configuration

lib/                                Runtime dependency jars
utms-agent.service                  Example systemd unit
build.xml                           NetBeans/Ant project file
```

Ignored generated/runtime folders:

```text
build/
dist/
logs/
```

## Main Components

### Main

`Main.java` owns the service lifecycle. It accepts one required argument:

```bash
java -jar utms-agent.jar /opt/utms-agent/conf/mqtt-agents.cfg
```

It also accepts the old wrapper-style argument form for compatibility:

```bash
java -cp ... id.iptek.utms.agent.Main id.iptek.utms.agent.Main /opt/utms-agent/conf/mqtt-agents.cfg
```

### MQTT Workers

MQTT workers are created dynamically by `WorkerFactory` from the `mqtt.consumers` property.

Configured consumers:

```properties
mqtt.consumers = heartbeat_c, diagnostic_c, device_init_c, download_ack_c
```

Each consumer has its own:

- topic
- thread count
- client ID prefix
- MQTT credentials
- keepalive
- clean session setting
- reconnect setting
- worker class
- queue mode where applicable

Worker classes:

| Worker | Purpose |
| --- | --- |
| `HeartbeatWorker` | Consumes `HEARTBEAT` messages and queues them for persistence. |
| `DiagnosticWorker` | Consumes `DIAGNOSTIC` messages and queues diagnostic details. |
| `DeviceInitWorker` | Consumes device init requests and queues profile response work. |
| `DownloadTaskAckWorker` | Consumes download ACK messages and sends confirm ACKs when needed. |

The base `Worker` class uses Eclipse Paho MQTT. It supports automatic reconnect and has a reconnect delay when the broker is unavailable.

### Queues

`DelayMessageQueue<T>` is the internal queue abstraction.

Supported modes:

| Mode | Behavior |
| --- | --- |
| `SINGLE` | Each message is handled separately. |
| `BATCH` | Messages are collected until capacity is reached or the flush interval expires. |

The queue shutdown is graceful. During service stop, queues are asked to drain before workers are disconnected. If queue work does not finish within 30 seconds, remaining work is forced to stop.

### Scheduler

Quartz runs `DownloadTaskSchedulerJob` every `scheduler.periode.minutes`.

The scheduler:

1. Finds pending download tasks.
2. Schedules `DownloadTaskPublisherJob`.
3. Publishes task messages over MQTT.
4. Regenerates expired MinIO presigned URLs before publishing.
5. Updates last broadcast timestamps in the database.

### MinIO URLs

`DownloadTaskPublisherJob` uses MinIO presigned URLs for app and icon downloads.

Relevant config:

```properties
fileserver.endpoint = https://...
fileserver.bucket = application-bucket
fileserver.icon.path = icons
fileserver.icon.expiry = 7
fileserver.icon.timeunit = DAYS
fileserver.app.path = apps
fileserver.app.expiry = 7
fileserver.app.timeunit = DAYS
```

If a stored URL expiry is null or in the past, the publisher generates a fresh URL before sending the MQTT message.

## Configuration

Main config file:

```text
conf/mqtt-agents.cfg
```

Important sections:

| Section | Keys |
| --- | --- |
| Logging | `log4j.file` |
| Database | `db.driver`, `db.url`, `db.user`, `db.password` |
| MQTT global publisher | `mqtt.broker.url`, `mqtt.clientid_prefix`, `mqtt.user`, `mqtt.password`, `mqtt.keepalive`, `mqtt.maxinflight`, `mqtt.cleansession`, `mqtt.autoreconnect` |
| Consumers | `mqtt.consumers`, `mqtt.consumer.<name>.*` |
| Pending task query | `pendingtask.query.limit`, `pendingtask.periode.minutes` |
| MinIO | `fileserver.*` |
| Scheduler | `scheduler.periode.minutes` |

`mqtt.keepalive` and `mqtt.consumer.<name>.keepalive` are in seconds because Eclipse Paho expects seconds.

Do not commit production credentials. Replace database, MQTT, and MinIO secrets before deploying.

## Logging

Logging is file-only. Console logging is disabled.

Current Log4j output:

```text
logs/utms-agent.log
```

Rotation:

```properties
log4j.appender.FILE.MaxFileSize=100MB
log4j.appender.FILE.MaxBackupIndex=10
```

The Linux service user must be able to write to the `logs/` directory under the service working directory.

## Build

This is a NetBeans/Ant Java project. If Ant is installed:

```bash
ant clean jar
```

The expected runtime artifact is:

```text
dist/utms-agent.jar
```

The jar is run with the dependency jars in `dist/lib`.

## Manual Build Fallback

If Ant is not available, compile with Java 8 and package a jar with:

```bash
javac -cp "lib/*;lib/Mqtt/*;lib/Quartz/*;lib/Minio-Client/*" -d build/classes <all source files>
jar cfm dist/utms-agent.jar build/MANIFEST.MF -C build/classes .
```

On Linux, use `:` instead of `;` in the classpath.

## Run Locally

```bash
java -jar dist/utms-agent.jar conf/mqtt-agents.cfg
```

The process writes logs to:

```text
logs/utms-agent.log
```

## Linux systemd

Example unit file:

```ini
[Unit]
Description=UTMS Agent
After=network.target

[Service]
Type=simple
User=tomcat
Group=tomcat
WorkingDirectory=/opt/utms-agent
ExecStart=/usr/bin/java -jar /opt/utms-agent/utms-agent.jar /opt/utms-agent/conf/mqtt-agents.cfg
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
```

Install example:

```bash
sudo mkdir -p /opt/utms-agent
sudo cp dist/utms-agent.jar /opt/utms-agent/
sudo cp -r dist/lib /opt/utms-agent/
sudo cp -r conf /opt/utms-agent/
sudo mkdir -p /opt/utms-agent/logs
sudo chown -R tomcat:tomcat /opt/utms-agent
sudo cp utms-agent.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable utms-agent
sudo systemctl start utms-agent
```

Useful commands:

```bash
sudo systemctl status utms-agent
sudo systemctl stop utms-agent
tail -f /opt/utms-agent/logs/utms-agent.log
```

## Operational Notes

- Tanuki and YAJWS wrappers are no longer used.
- The app does not log to console.
- MQTT worker keepalive values are seconds, not milliseconds.
- Queue shutdown is graceful for up to 30 seconds, then forced.
- MinIO presigned URLs are regenerated before publish if expired.
- `build/`, `dist/`, and `logs/` are ignored by Git.