📦 Ditos Manifest Service — Backend Guide

A friendly, in-depth tour of the Java backend — written so a newcomer can understand what the project does, how the code is organized, and why each piece exists. Read it top-to-bottom the first time; later, jump straight to the subsystem you care about.

1. What is this project? (the 30-second version)
2. The technology stack
3. The big picture
4. How the code is organized
5. The request lifecycle (this is the heart of it)
6. Shared building blocks (common/)
7. Authentication & authorization deep-dive (auth/)
8. The Unicommerce integration (unicommerce/)
9. Subsystem tour
10. The “two-gate” pattern (important!)
11. The scheduler (ManifestScheduler)
12. Configuration & secrets
13. Running it locally
14. Cross-cutting conventions (the “house style”)
15. Mini-glossary (for newcomers)
16. Suggested reading path for a new developer

1. What is this project? (the 30-second version)

This is a single-client integration backend that sits between a retailer’s systems and Unicommerce (a warehouse / order-management SaaS).

It does five practical jobs:

#	Job	Plain-English description
1	Shipping manifests	Pull “what shipped today” data out of Unicommerce on a schedule and store it.
2	Catalog sync	Push product item types and categories into Unicommerce.
3	Inventory	Push stock adjustments to Unicommerce, and bulk-import full inventory snapshots back.
4	Mall feeds	Let external “malls” (landlords/partners) securely pull a store’s sales transactions.
5	Admin console	A web UI (React) for staff to manage users, credentials, outlets, toggles, and view logs.

Everything is exposed as a REST API. A separate React frontend talks to it.

2. The technology stack

Layer	Choice	Why it matters
Language	Java 25	Modern Java (records, pattern matching, text blocks).
Framework	Spring Boot 4.0.5	Web, Security, Data JPA, Validation, Actuator.
Database	PostgreSQL	Relational store; also used for fast bulk `COPY` loads.
Migrations	Flyway	Versioned SQL files own the schema (`V1__...sql`, `V2__...sql`, …).
Auth	JWT (jjwt) + Spring Security	Stateless tokens for the admin UI.
Secrets	AWS Secrets Manager	DB credentials loaded at boot in prod.
Analytics source	Google BigQuery	Mall transactions are read from BigQuery.
API docs	springdoc / Swagger UI	Auto-generated at `/swagger-ui.html`.
Boilerplate	Lombok	`@Getter`, `@RequiredArgsConstructor`, etc.

Mental model: Spring Boot wires everything together. You write small focused classes (controllers, services, repositories) and annotate them; Spring creates and connects them at startup.

3. The big picture

                    ┌─────────────────────────────────────────────┐
                    │              React Admin UI                  │
                    │        (browser, JWT in Authorization)       │
                    └───────────────────────┬─────────────────────┘
                                            │  HTTPS /api/**
        External malls ──X-API-Key──┐       │       ┌──── Client POS ── X-Secret-Token
                                    ▼       ▼       ▼
                    ┌─────────────────────────────────────────────┐
                    │            Spring Boot backend               │
                    │  ┌────────────────────────────────────────┐ │
                    │  │  Servlet filter chain (auth gates)      │ │
                    │  └────────────────────────────────────────┘ │
                    │  Controllers → Services → Repositories       │
                    └───────┬───────────────────────┬─────────────┘
                            │                       │
                ┌───────────▼─────────┐   ┌─────────▼──────────┐   ┌──────────────┐
                │     PostgreSQL      │   │    Unicommerce     │   │   BigQuery   │
                │  (own schema, JPA)  │   │   (REST + OAuth)   │   │  (read-only) │
                └─────────────────────┘   └────────────────────┘   └──────────────┘

There are three “front doors”, each with its own way of proving who you are:

Door	URL prefix	Who uses it	How it authenticates
🛠️ Admin	most `/api/**`	staff via the React UI	JWT (`Authorization: Bearer …`)
🤝 Client	`/api/client/**`	the retailer’s POS / systems	`X-Secret-Token` header + IP allowlist
🏬 Mall	`/api/mall/**`	external partners	`X-API-Key` header + per-mall IP allowlist + rate limit

4. How the code is organized

Everything lives under com.ditos. Each business area is its own package — a “vertical slice” holding its own controller, service, entity, repository, and DTOs together.

com.ditos
├── DitosManifestApplication.java   ← main() entry point
├── common/        🧱 shared building blocks (ApiResponse, exceptions, BaseEntity, AES, error handler)
├── config/        ⚙️  security, CORS, OpenAPI, AWS Secrets, IP allowlist
├── security/      🛡️  cross-cutting filters (trace id, login rate limit)
├── auth/          🔑 users, login, JWT, password rules, first-boot admin
├── client/        🤝 /api/client/** gate (token + IP) for the retailer
├── mall/          🏬 /api/mall/** partner feeds (token + IP + rate limit + XML)
├── unicommerce/   🔌 OAuth token lifecycle + two HTTP clients to Unicommerce
├── manifest/      🚚 shipping-manifest fetch, storage, scheduler
├── catalog/       🏷️  push item types & categories to Unicommerce (with call logs)
├── inventory/     📊 stock adjustments + bulk snapshot import pipeline
├── bigquery/      🔎 read transactions/analytics from BigQuery
├── outlet/        🏪 store/facility master data
└── operations/    🔘 feature toggles (turn jobs on/off without redeploy)

Where do I start reading? Pick a feature, open its package, and read in this order: Controller (the HTTP shape) → Service (the logic) → Repository/Entity (the data).

5. The request lifecycle (this is the heart of it)

Every HTTP request walks through a chain of servlet filters before it reaches a controller. Think of filters as a series of gates; each can let the request pass or stop it cold.

HTTP request
   │
   ▼
[1] RequestTraceFilter        → stamps a traceId (X-Trace-Id) for log correlation
   │
[2] RateLimitFilter           → throttles POST /api/auth/login per IP (anti brute-force)
   │
   ├──── Spring Security chain ─────────────────────────────────────────┐
   │  [3] JwtAuthenticationFilter   → if Bearer token valid, set the user │
   │  [4] IpAllowlistFilter         → optional global IP allowlist        │
   │  [5] ClientApiAuthFilter       → guards /api/client/** (token + IP)  │
   │  [6] MallAuthFilter            → guards /api/mall/** (token+IP+rate)  │
   │  [7] MustChangePasswordFilter  → blocks all but pwd-change if forced  │
   │  [8] authorizeHttpRequests     → "is this path public or protected?" │
   └────────────────────────────────────────────────────────────────────┘
   │
   ▼
Controller → Service → Repository → (DB / Unicommerce / BigQuery)
   │
   ▼
Response wrapped in ApiResponse { success, message, data, timestamp }

Defined in config/SecurityConfig.java. Key facts:

Stateless: SessionCreationPolicy.STATELESS — no server-side sessions; the JWT is the session.
Public paths (PUBLIC[]): /api/auth/login, /api/client/**, /api/mall/**, /api/manifest/** (⚠️ temporary public test trigger), health, and Swagger. “Public to Spring Security” means JWT is not required — the client/mall filters still guard their paths.
Security headers: frame deny, content-type-options, HSTS (1 year), referrer policy.
CORS: driven by app.cors.allowed-origins (* in dev, locked-down list in prod).

6. Shared building blocks (`common/`)

These are used everywhere, so learn them once.

6.1 `ApiResponse<T>` — the uniform envelope

Every JSON response has the same shape:

{ "success": true, "message": "optional note", "data": { ... }, "timestamp": "2026-..." }

Helpers: ApiResponse.ok(data), ApiResponse.ok("msg", data), ApiResponse.error("msg"). Null fields are dropped, so error bodies stay small.

6.2 `ApiException` + `GlobalExceptionHandler` — one place for errors

Throw new ApiException(HttpStatus.NOT_FOUND, "No such outlet") anywhere in a service. GlobalExceptionHandler (a @RestControllerAdvice) catches it — and validation errors, bad credentials, 404s, etc. — and turns each into the right HTTP status + an ApiResponse.error(...). The catch-all logs the stack trace and returns a generic 500 so internals never leak.

6.3 `BaseEntity` — audit + soft-delete for free

Most entities extend BaseEntity, which adds: id, createdBy/createdDate, updatedBy/lastModifiedDate, version (optimistic locking), and soft-delete via is_deleted + @SQLRestriction("is_deleted = false") — so “deleted” rows are simply filtered out of every query. active is a separate enable/disable flag.

6.4 `AesStringConverter` — encrypt sensitive columns at rest

Annotate a field with @Convert(converter = AesStringConverter.class) and JPA transparently AES-GCM-encrypts it on write and decrypts on read. The 256-bit key is derived (SHA-256) from the APP_ENCRYPTION_KEY env var. Used for things like stored third-party credentials.

APP_ENCRYPTION_KEY must be a real OS env var — there’s a loud dev fallback you must never ship.

7. Authentication & authorization deep-dive (`auth/`)

This is the admin/staff login system.

7.1 The players

POST /api/auth/login { username, password }
   → AuthService.login()
       • find user, check active, check BCrypt password
       • JwtService.issue(user)  → signs a token with claims:
              subject=username, role, mustChangePassword, exp(+8h)
   ← { token, username, role, mustChangePassword }

On later requests the browser sends Authorization: Bearer <token>. JwtAuthenticationFilter parses it, and if valid puts a ROLE_<role> authority into the SecurityContext. Method-level rules like @PreAuthorize("hasRole('PLATFORM_ADMIN')") then apply.

7.3 First-boot admin (no seeded password in SQL!)

AuthBootstrap runs once when the user table is empty: it creates admin with a random 16-char password printed once to the logs under a FIRST-BOOT: initial admin created banner, with mustChangePassword = true.

If you ever can’t log in locally, this banner in the backend startup log is where the password is.

7.4 Forced password change

The JWT carries mustChangePassword. While that’s true, MustChangePasswordFilter blocks every endpoint with 403 except /api/auth/change-password, /me, /logout. This makes the rule authoritative even if someone bypasses the UI.

8. The Unicommerce integration (`unicommerce/`)

Most features ultimately call Unicommerce. All the auth plumbing lives here so feature code stays simple.

8.1 Two HTTP clients, one host

UnicommerceClientConfig defines two RestClient beans — neither hard-codes a base URL (callers pass absolute URLs):

Bean	Purpose	Auth	Read timeout
`unicommerceAuthRestClient`	the OAuth token endpoint	none (it fetches the token)	30s
`unicommerceApiRestClient`	all authenticated API calls	auto-injects `Authorization: Bearer …`	60s

They resolve to the same host because UnicommerceTokenService.getApiBaseUrl() derives scheme://host[:port] from the stored credentials’ token URL.

8.2 Token lifecycle (`UnicommerceTokenService`)

Caches a valid access token in memory and persists it (so restarts don’t re-auth).
getAccessToken() returns a non-expired token, refreshing 60s before expiry.
Refresh strategy: try the refresh-token grant, fall back to the password grant.
Concurrency: a ReentrantLock ensures only one refresh happens at a time.

8.3 The “re-login on 401” pattern — `withRetryOn401`

Feature clients wrap calls like this:

tokenService.withRetryOn401(() -> apiRestClient.post().uri(base + PATH).body(x).retrieve().body(String.class));

If Unicommerce returns 401 (token revoked server-side), the helper invalidates the cached token and retries once — which transparently re-authenticates. You’ll see this in every Unicommerce client (UnicommerceItemTypeClient, UnicommerceCategoryClient, UnicommerceInventoryClient, etc.).

8.4 Test proxy (dev only)

UnicommerceTestProxyInterceptor can route calls through a whitelisted relay so local dev machines (without a whitelisted IP) can still reach Unicommerce. Controlled by app.unicommerce.test-proxy.enabled.

9. Subsystem tour

9.1 🚚 Manifests (`manifest/`)

Pulls shipping-manifest data from Unicommerce and stores it as ShippingManifest → ManifestPackage → ManifestLineItem.

ManifestFetchService — the workhorse. Resolves a time window using a gapless high-water-mark cursor (continues from the previous run’s windowEnd; default 30-minute window), then fetches for active outlets in a 3-phase concurrent pattern.
ManifestRunService — records each run (ManifestRun) for the admin history.
ManifestScheduler — an in-process scheduler (see §11).
ManifestController — ⚠️ a temporary public POST /api/manifest/run manual trigger.

9.2 🏷️ Catalog (`catalog/`)

Pushes products into Unicommerce with forensic logging + manual replay — a pattern worth learning because inventory copies it too.

Two near-identical stacks:

Feature	Client endpoint	Unicommerce path	Log table
Item types	`POST /api/client/catalog/itemType/createOrEdit`	`…/catalog/itemType/createOrEdit`	`catalog_item_type_log`
Categories	`POST /api/client/catalog/category/addOrEdit`	`…/product/category/addOrEdit`	`catalog_category_log`

The logging pattern (in ItemTypeService / CategoryService):

Save a PENDING log row before the HTTP call (so a crash mid-call still leaves a trace).
Call Unicommerce via the client (with withRetryOn401).
Update the row to SUCCESS / FAILED with the request payload + raw response.
Never auto-retry. An admin reviews failures and replays them (POST /api/catalog/logs/{id}/retry) — safe because these operations are idempotent upstream.

Admin views: GET /api/catalog/logs and GET /api/catalog/category/logs (PLATFORM_ADMIN only).

9.3 📊 Inventory (`inventory/`)

Two distinct things:

(a) Adjustments — POS pushes stock changes: POST /api/client/inventory/adjust → InventoryService (same PENDING→SUCCESS/PARTIAL/FAILED log pattern as catalog) → UnicommerceInventoryClient → …/inventory/adjust/bulk (sets a Facility header).

(b) Snapshot import pipeline (inventory/snapshot/) — admin batch job that imports the full inventory matrix using a clever 3-phase, memory-flat design:

Phase 1 CREATE-ALL : ask Unicommerce to create an export job per outlet  (export/job/create)
Phase 2 POLL-ALL   : poll all jobs concurrently until COMPLETE           (export/job/status)
Phase 3 INGEST-ALL : stream-download each CSV and bulk-load via Postgres COPY

StreamingCsvReader parses CSV one row at a time (constant memory, RFC-4180-ish).
InventorySnapshotCsvIngestor streams rows straight into a Postgres COPY, tags each with the run id, then atomically deletes the previous run’s rows for that facility (full-replace per facility).
InventorySnapshotRunService records the batch outcome.

Why COPY? Per-row INSERT of ~100k rows would be painfully slow; COPY streams them in one shot.

9.4 🏬 Mall (`mall/`)

Lets external partners pull a store’s sales transactions (sourced from BigQuery).

MallAuthFilter guards /api/mall/**: reads X-API-Key, resolves client IP, calls MallService.verify(ip, token) (token is SHA-256 hashed and looked up; per-mall IP allowlist checked unless app.mall-auth.ip-check-enabled=false), then applies a per-mall rate limit (MallRateLimiter, fixed 60s window; 429 + Retry-After when exceeded).
MallTransactionController → MallTransactionService: validates the requested store_code maps to the mall’s outlet, queries BigQuery, returns JSON or XML (MallTransactionXml). Supports incremental pulls via a stored cursor (lastBilledMicros, lastInvoiceId).
MallAdminController (/api/malls, admin): CRUD malls, manage per-mall IPs and tokens. A token is shown once at generation (only its hash is stored) — same one-time-secret idea as client tokens.

9.5 🤝 Client access (`client/`)

The token+IP system behind /api/client/**:

ClientAccessService.verify(ip, token) — token (SHA-256) must match an active credential; the IP allowlist applies unless app.client-auth.ip-check-enabled=false (token-only mode).
ClientAccessController (admin) manages allowlisted IPs and one-time secret tokens.
TokenSupport — generates high-entropy tokens and hashes them.

9.6 🔎 BigQuery (`bigquery/`)

Read-only analytics access. BigQueryCredentialService stores a service-account JSON (encrypted), BigQueryClientProvider builds the client, BigQueryService runs parameterized queries. The mall feeds are the main consumer.

9.7 🏪 Outlets & 🔘 Operations

outlet/ — master data for stores/facilities (facility codes used across manifests/inventory/malls).
operations/ — feature toggles stored in the DB (operation_toggle), e.g. MANIFEST_FETCH, INVOICE_PUSH. Lets an admin turn a job on/off without a redeploy.

10. The “two-gate” pattern (important!)

Background jobs are guarded by two independent switches:

Config flag (env / Secrets Manager) — is this capability deployed/enabled at all? e.g. MANIFEST_SCHEDULER_ENABLED=true.
DB operation toggle (operations/) — should it actually run right now? e.g. the MANIFEST_FETCH row being enabled=true.

Both must be on. The config flag is an ops/deploy decision; the DB toggle is a day-to-day admin switch.

11. The scheduler (`ManifestScheduler`)

An in-process scheduler (@Scheduled, enabled by @EnableScheduling on the main class):

@Component
@ConditionalOnProperty(prefix="app.manifest.scheduler", name="enabled", havingValue="true")
class ManifestScheduler {
  @Scheduled(fixedDelayString="${app.manifest.scheduler.interval-ms:1800000}",   // 30 min
             initialDelayString="${app.manifest.scheduler.initial-delay-ms:60000}")
  void tick() { /* skip if MANIFEST_FETCH toggle off, else fetch + record */ }
}

Runs only if the config flag is on (MANIFEST_SCHEDULER_ENABLED) and the MANIFEST_FETCH DB toggle is on (the two-gate pattern).
fixedDelay = waits for the previous run to finish, then waits the interval (no overlap).
⚠️ This only works on a long-lived JVM (e.g. EC2). On AWS Lambda the process freezes between invocations, so you’d use EventBridge to trigger a run instead.

12. Configuration & secrets

12.1 `application.yml` (defaults) + `application-prod.yml` (prod overrides)

Settings use ${ENV_VAR:default} so the same jar runs locally and in prod by changing env vars. Key knobs:

Property	Env var	What it does
`server.port`	`SERVER_PORT`	HTTP port (default 8080).
`app.jwt.secret`	`JWT_SECRET`	HS256 signing key (≥32 bytes).
`app.cors.allowed-origins`	`CORS_ALLOWED_ORIGINS`	lock down in prod.
`app.manifest.scheduler.enabled`	`MANIFEST_SCHEDULER_ENABLED`	turn the 30-min job on.
`app.client-auth.ip-check-enabled`	`CLIENT_IP_CHECK_ENABLED`	token-only client mode when `false`.
`app.mall-auth.ip-check-enabled`	`MALL_IP_CHECK_ENABLED`	token-only mall mode when `false`.
`app.ip-allowlist.trust-forwarded-headers`	`IP_TRUST_FORWARDED`	trust `X-Forwarded-For` behind a proxy.

12.2 AWS Secrets Manager (`SecretsManagerInitializer`)

On non-dev profiles, before the Spring context starts, this loads a JSON secret (ditos-middleware/<profile>/db) and injects DB credentials so application-prod.yml can resolve ${DATABASE_*}. On EC2 the instance-profile IAM role authenticates automatically.

The whitelist trap (read this!): the initializer only injects a hard-coded list of keys. If you add a new secret key (e.g. a new toggle) and forget to add it to this class, it is silently ignored. Whenever a “I set it in Secrets Manager but nothing changed” bug appears, check this file first.

12.3 Behind nginx — the client-IP gotcha

When the app runs behind nginx, request.getRemoteAddr() is nginx’s 127.0.0.1, not the real client IP. The real IP arrives in X-Forwarded-For / X-Real-IP, which the auth filters only trust when trust-forwarded-headers=true. Only enable that if port 8080 is not directly reachable (otherwise the header can be spoofed).

13. Running it locally

# 1. Start PostgreSQL and create the DB (default: ditos_manifest on localhost:5432)
# 2. From backend/:
mvn spring-boot:run            # Flyway auto-runs migrations on startup

# 3. Watch the logs for the first-boot admin banner:
#    FIRST-BOOT: initial admin created   username: admin   password: <random>

# 4. API is on http://localhost:8080 ; Swagger UI at /swagger-ui.html

Health check: GET http://localhost:8080/actuator/health → {"status":"UP"}.
The dev profile skips AWS Secrets Manager and uses the plaintext defaults in application.yml.

14. Cross-cutting conventions (the “house style”)

Controller → Service → Repository. Controllers are thin (HTTP shape + validation); services hold logic and @Transactional; repositories are Spring Data JPA interfaces.
DTOs are Java records, grouped in a …Dtos holder per area, with Bean Validation annotations (@NotBlank, @Size) and @JsonInclude(NON_NULL).
Every response is an ApiResponse; every error path goes through GlobalExceptionHandler.
Secrets never hard-coded — env vars or Secrets Manager only; sensitive columns use AesStringConverter.
External call logs (catalog/inventory): persist PENDING first, never auto-retry, allow manual replay.
Tokens (client & mall): high entropy, stored hashed, shown in plaintext exactly once.
Trace everything: each request has an X-Trace-Id in logs and the response header.

15. Mini-glossary (for newcomers)

Term	Meaning
Bean	An object Spring creates and manages for you (controllers, services, etc.).
Filter	Code that runs on every request before the controller — used here for auth gates.
JWT	A signed token proving who you are; sent as `Authorization: Bearer <token>`.
DTO	“Data Transfer Object” — the request/response shape, separate from DB entities.
Entity	A Java class mapped to a DB table (JPA).
Repository	An interface giving you DB queries without writing SQL.
Flyway migration	A versioned `.sql` file that evolves the schema in order.
Soft delete	Marking a row `is_deleted=true` instead of removing it.
Idempotent	Safe to run twice with the same effect (why manual replay is safe).
High-water-mark cursor	Remembering “where I stopped” so the next run continues gaplessly.

16. Suggested reading path for a new developer

DitosManifestApplication.java → config/SecurityConfig.java (how requests are gated).
common/ApiResponse, ApiException, GlobalExceptionHandler, BaseEntity (the conventions).
auth/ end-to-end (login → JWT → filter → controller).
One simple feature slice: outlet/ (clean Controller→Service→Repository example).
unicommerce/UnicommerceTokenService + one client (catalog/UnicommerceCategoryClient).
A “logged + replayable” feature: catalog/CategoryService.
The advanced pipeline: inventory/snapshot/ (concurrency + Postgres COPY).

Once these click, every other package follows the same shapes.

📦 Ditos Manifest Service — Backend Guide

On this page

1. What is this project? (the 30-second version)

2. The technology stack

3. The big picture

4. How the code is organized

5. The request lifecycle (this is the heart of it)

6. Shared building blocks (common/)

6.1 ApiResponse<T> — the uniform envelope

6.2 ApiException + GlobalExceptionHandler — one place for errors

6.3 BaseEntity — audit + soft-delete for free

6.4 AesStringConverter — encrypt sensitive columns at rest

7. Authentication & authorization deep-dive (auth/)

7.1 The players

7.2 Login flow

7.3 First-boot admin (no seeded password in SQL!)

7.4 Forced password change

8. The Unicommerce integration (unicommerce/)

8.1 Two HTTP clients, one host

8.2 Token lifecycle (UnicommerceTokenService)

8.3 The “re-login on 401” pattern — withRetryOn401

8.4 Test proxy (dev only)

9. Subsystem tour

9.1 🚚 Manifests (manifest/)

9.2 🏷️ Catalog (catalog/)

9.3 📊 Inventory (inventory/)

9.4 🏬 Mall (mall/)

9.5 🤝 Client access (client/)

9.6 🔎 BigQuery (bigquery/)

9.7 🏪 Outlets & 🔘 Operations

10. The “two-gate” pattern (important!)

11. The scheduler (ManifestScheduler)

12. Configuration & secrets

12.1 application.yml (defaults) + application-prod.yml (prod overrides)

12.2 AWS Secrets Manager (SecretsManagerInitializer)

12.3 Behind nginx — the client-IP gotcha

13. Running it locally

14. Cross-cutting conventions (the “house style”)

15. Mini-glossary (for newcomers)

16. Suggested reading path for a new developer

6. Shared building blocks (`common/`)

6.1 `ApiResponse<T>` — the uniform envelope

6.2 `ApiException` + `GlobalExceptionHandler` — one place for errors

6.3 `BaseEntity` — audit + soft-delete for free

6.4 `AesStringConverter` — encrypt sensitive columns at rest

7. Authentication & authorization deep-dive (`auth/`)

8. The Unicommerce integration (`unicommerce/`)

8.2 Token lifecycle (`UnicommerceTokenService`)

8.3 The “re-login on 401” pattern — `withRetryOn401`

9.1 🚚 Manifests (`manifest/`)

9.2 🏷️ Catalog (`catalog/`)

9.3 📊 Inventory (`inventory/`)

9.4 🏬 Mall (`mall/`)

9.5 🤝 Client access (`client/`)

9.6 🔎 BigQuery (`bigquery/`)

11. The scheduler (`ManifestScheduler`)

12.1 `application.yml` (defaults) + `application-prod.yml` (prod overrides)

12.2 AWS Secrets Manager (`SecretsManagerInitializer`)